Abnormal detection method, system and storage medium for examinee's answer information

Abstract

The embodiment of the present invention provides a method, system and storage medium for detecting anomalies in the answer information of examinees, which belongs to the field of computer algorithm application. The method includes: obtaining the answer process and answer information of examinees during the paperless examination, and extracting the key features in the answer process and answer information to generate corresponding matrix data; performing similarity calculation on the matrix data to determine the similarity of answer behaviors between different examinees; if the matrix data is unlabeled data, using a clustering algorithm to perform unsupervised learning on the matrix data to determine the behavioral characteristics of abnormal examinees; if the matrix data is labeled data, using supervised learning to determine the behavioral characteristics of abnormal examinees. By using the data of the answers in the test to screen the examinees for abnormalities and similarities, calculating the similarity values ​​of the behaviors and answer contents between the examinees, and using supervised learning or unsupervised learning to identify batches of examinees with similar behaviors, early warning of suspected abnormal situations of the examinees is issued.

Description

Technical Field

[0001] The present invention relates to the field of computer algorithm application technology, and in particular to a method, system and storage medium for detecting anomalies in examinee answer information. Background Art

[0002] In recent years, with the rapid development of information technology, paperless examinations have become an important part of the modern education examination system. Compared with traditional paper-based examinations, paperless examinations provide more flexibility and convenience for examination organizers and reduce the cost of examinations. As an important trend, paperless examinations are increasingly used in major examinations at home and abroad.

[0003] However, the popularity of this form of examination has also brought about a new problem, such as abnormal behaviors such as cheating. In order to maintain the fairness and impartiality of the examination and prevent the occurrence of cheating, it is particularly important to develop an efficient cheating detection algorithm. Summary of the invention

[0004] The purpose of the embodiments of the present invention is to provide a method, system and storage medium for detecting anomalies in candidate answer information, which are used to conduct in-depth analysis of candidate answer data in computer-based examinations, thereby identifying potential abnormal behaviors of candidates and better ensuring the completion of paperless examinations.

[0005] In order to achieve the above object, an embodiment of the present invention provides a method for detecting anomalies in examinee answer information, comprising:

[0006] Obtain the answering process and answering information of the examinee during the paperless examination, and extract key features from the answering process and answering information to generate corresponding matrix data;

[0007] Perform similarity calculations on the matrix data to determine the similarity of answering behaviors between different candidates;

[0008] If the matrix data is unlabeled data, a clustering algorithm is used to perform unsupervised learning on the matrix data, and the examinees are divided into normal examinees and abnormal examinees based on the similarity of the answering behaviors of different examinees, and anomaly detection is performed on the abnormal examinees to determine the behavioral characteristics of the abnormal examinees;

[0009] If the matrix data is labeled data, supervised learning is used to perform anomaly detection on the marked abnormal candidate samples to determine the behavioral characteristics of the abnormal candidates.

[0010] Optionally, the candidate's answering process and answering information include but are not limited to test center information, examination room information, examination area information, examination batch information, candidate information, examination subject information, examination paper number information, examination paper question number information, examination subject question sequence information, examination question structure information, candidate answering question time information, candidate browsing question time information, candidate modifying question time information and candidate answer content information.

[0011] Optionally, extracting key features from the answering process and the answering information to generate corresponding matrix data includes:

[0012] Taking answering time as the key factor, extract the answering time of each candidate for each question, and construct a matrix with the elements corresponding to the horizontal and vertical coordinates as time, using the candidate number and the question number as the horizontal and vertical coordinates respectively;

[0013] Taking the answer content as the key factor, extract the answer content of each candidate for each question, and use the candidate number and question number as the horizontal and vertical coordinates to construct a matrix in which the elements corresponding to the horizontal and vertical coordinates are the answer content;

[0014] Taking browsing time and revision times as key factors, the answering time of each candidate for each question is extracted, and the candidate number and question number are used as the horizontal and vertical axes, plus a column of revision times, to construct a matrix in which the horizontal and vertical axes correspond to browsing time and revision times.

[0015] Optionally, similarity calculations are performed on the matrix data to determine the similarity of answering behaviors between different candidates, including:

[0016] For similar calculations of the answer content matrix, the following formula is used:

[0017] ;

[0018] In the formula, is the XOR operation, , are different sample sets, and i is the label in the sample set.

[0019] Optionally, similarity calculations are performed on the matrix data to determine the similarity of answering behaviors between different candidates, including:

[0020] For similar calculation of the time trajectory matrix, the following formula is used:

[0021] ;

[0022] In the formula, Represents the Pearson correlation coefficient of two continuous variables (X, Y). represents the covariance between two continuous variables (X, Y), It represents the product of the standard deviations of two continuous variables (X, Y). Represents two variables , The expected value of the product of the differences with their respective expectations.

[0023] Optionally, if the matrix data is unlabeled data, a clustering algorithm is used to perform unsupervised learning on the matrix data, and the examinees are distinguished into normal examinees and abnormal examinees based on the similarity of answering behaviors between different examinees, and abnormality detection is performed on the abnormal examinees to determine the behavioral characteristics of the abnormal examinees, including:

[0024] For large-scale data across test areas and test sites, the K-Means clustering algorithm is used to identify data points far away from the main cluster in the large-scale candidate answer information data as abnormal points, and then identify abnormal candidates;

[0025] For the data of small-scale test sites, a single-class support vector machine is used to identify the data points whose distance from the small-scale test site data to the center point is less than or equal to the critical distance as abnormal points, and then identify the abnormal candidates.

[0026] Optionally, if the matrix data is labeled data, supervised learning is used to perform anomaly detection on the labeled abnormal candidate samples to determine the behavioral characteristics of the abnormal candidates, including:

[0027] If the matrix data is labeled data, the XGBoost ensemble learning method is used to perform anomaly detection on the marked abnormal candidate samples to determine the behavioral characteristics of the abnormal candidates;

[0028] Among them, the objective function of the XGBoost ensemble learning method is as follows:

[0029]

[0030] In the formula, As the loss function to measure the difference between the predicted result and the actual result, the loss function uses the mean square error. For the The true value of the samples, For the Sample prediction values, is the weak evaluator, i.e. the decision tree added, t is the number of prediction rounds, is the penalty term, which is the sum of the number of leaf nodes and the coefficient multiplied by the square of the weight L2 modulus. is a constant term.

[0031] Optionally, the method for detecting anomalies in the examinee's answer information further includes:

[0032] Treat the examinees' answer information as discretely distributed time series data and construct a time series model;

[0033] The obtained historical answering behavior data of candidates are input into the time series model to predict the candidates' future answering behavior.

[0034] On the other hand, the present invention also provides a system for detecting anomalies in examinee answer information, comprising:

[0035] An acquisition unit is used to acquire the answering process and answering information of the examinee during the paperless examination, and extract key features in the answering process and answering information to generate corresponding matrix data;

[0036] A calculation unit, used for performing similarity calculation on the matrix data to determine the similarity of answering behaviors between different examinees;

[0037] A first anomaly detection unit is used to perform unsupervised learning on the matrix data using a clustering algorithm if the matrix data is unlabeled data, and to distinguish the examinees into normal examinees and abnormal examinees based on the similarity of answering behaviors between different examinees, and to perform anomaly detection on the abnormal examinees to determine the behavioral characteristics of the abnormal examinees;

[0038] The second anomaly detection unit is used to perform anomaly detection on the marked abnormal candidate samples by using a supervised learning method if the matrix data is labeled data, so as to determine the behavioral characteristics of the abnormal candidates.

[0039] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method for detecting anomalies in examinee answer information as described in any one of the above items.

[0040] Through the above technical solution, the data of candidates' answers during the exam are used to screen for abnormalities and similarities among candidates, and the similarity values ​​of the behaviors and answer contents among candidates are accurately quantified and calculated, so as to identify candidates who are suspected of cheating in groups and identify batches of candidates with similar and identical behaviors, so as to warn the invigilators or exam organizers of suspected abnormal situations in the candidates' answering process.

[0041] Other features and advantages of the embodiments of the present invention will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the embodiments of the present invention, but do not constitute a limitation on the embodiments of the present invention. In the accompanying drawings:

[0043] Figure 1It is a flowchart of an implementation of a method for detecting abnormalities in examinee answer information provided by an embodiment of the present invention;

[0044] Figure 2 1 is a schematic diagram of a candidate's trajectory visualization provided by an embodiment of the present invention, wherein (a) is a schematic diagram of the test trajectory of candidate Zhang San, and (b) is a schematic diagram of the test trajectory of candidate Wang Si;

[0045] Figure 3 This is a schematic diagram for comparing different answering trajectories among examinees provided by an embodiment of the present invention;

[0046] Figure 4 This is a schematic diagram for comparing similar answer trajectories between examinees provided by an embodiment of the present invention;

[0047] Figure 5 It is a dimensionality reduction visualization example provided by an embodiment of the present invention;

[0048] Figure 6 is a precision-recall rate example provided by an embodiment of the present invention;

[0049] Figure 7 It is a specificity and sensitivity example provided by an embodiment of the present invention;

[0050] Figure 8 It is a structural diagram of a system for detecting anomalies in examinee answer information provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The specific implementation of the embodiment of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the embodiment of the present invention, and is not used to limit the embodiment of the present invention.

[0052] See also Figure 1 As shown, it is a flowchart of an implementation method of an abnormal detection method of examinee answer information provided by an embodiment of the present invention, including the following execution steps:

[0053] Step 100: Obtain the answering process and answering information of the examinee during the paperless examination, and extract key features in the answering process and answering information to generate corresponding matrix data.

[0054] Preferably, the candidate's answering process and answering information include but are not limited to test site information, examination room information, examination area information, examination batch information, candidate information, examination subject information, examination paper number information, examination paper question number information, examination subject question sequence information, examination question structure information, candidate answering question time information, candidate browsing question time information, candidate modifying question time information and candidate answer content information.

[0055] Specifically, extracting key features from the answering process and answering information to generate corresponding matrix data includes the following three cases:

[0056] Case 1: Taking answering time as the key factor, extract the answering time of each candidate for each question, and construct a matrix with the elements corresponding to the horizontal and vertical coordinates as time, using the candidate number and the question number as the horizontal and vertical coordinates respectively;

[0057] Case 2: Taking the answer content as the key factor, extract the answer content of each candidate for each question, and use the candidate number and question number as the horizontal and vertical coordinates to construct a matrix with the elements corresponding to the horizontal and vertical coordinates as the answer content;

[0058] Case 3: Taking browsing time and number of revisions as key factors, extract the answering time of each candidate for each question, and use the candidate number and question number as the horizontal and vertical axes plus a column of revision times to construct a matrix with the horizontal and vertical axes corresponding to the browsing time and the number of revisions.

[0059] In some embodiments, the table (or matrix) of time and answer content and the table generated by extracting information cannot be directly calculated, and some standardization processing needs to be performed on the generated table (or matrix). These data are cleaned and standardized to eliminate noise and erroneous data to ensure the accuracy and consistency of the data. Finally, a standard data set format is formulated to organize the screened and cleaned data into a structure that is easy to calculate and analyze. Therefore, after executing step 100, the following steps are also performed:

[0060] S1: Temporal dataset normalization:

[0061] For the matrix of answering time or browsing time, first determine the test batch, compare the time format such as "%Y-%m-%d%H:%M:%S" with the start time of the test batch, and convert it into the number of seconds after the test starts. Based on this, some samples and some features can be generated as shown in Table 1 below:

[0062] Table 1 Time matrix for some questions of some candidates

[0063]

[0064] Calculate the mean of each feature (title) and fill in the missing value with the mean. Put all features on the same scale. Rescale the features to zero mean and unit variance to standardize the temporal data.

[0065] S2: Standardization of answer content dataset:

[0066] For the matrix of answer content, the answer content needs to be encoded. Conventional single-choice questions can directly encode ABCD or true or false as 1, 2, 3, 4 and 1, 2. For multiple-choice questions, one encoding method is the same as single-choice questions, which is to convert the numbers corresponding to the options into strings, concatenate them, and then convert them back to digital types. However, the results of encoding using this encoding method may affect the effect of subsequent modeling because the values ​​of some answers are too large compared to the single-choice encoding, and the differences are no longer on the same scale. For this reason, an improved encoding method is proposed, using the bit mask method in bit operations:

[0067] The bit mask method is often used to represent a set of switches, each switch corresponds to a bit. In this example, we can assign a different bit to each option:

[0068] A: 1 (0001 in binary)

[0069] B: 2 (0010 in binary)

[0070] C: 4 (0100 in binary)

[0071] D: 8 (1000 in binary)

[0072] This way, if the candidate selects multiple options, the values ​​can simply be added together to create a unique coded value. For example:

[0073] Choose A and B: 1 + 2 = 3 (0011 in binary)

[0074] Choose A and C: 1 + 4 = 5 (0101 in binary)

[0075] Choose B and C: 2 + 4 = 6 (0110 in binary)

[0076] Choose A, B, and C: 1 + 2 + 4 = 7 (0111 in binary)

[0077] Select all ABCD: 1 + 2 + 4 + 8 = 15 (1111 in binary)

[0078] To infer the combination of options, you can use bitwise operations. For example, to check whether A is selected, you can use the bitwise AND (&) operation:

[0079] Code value & 1: If the result is 1, A is selected

[0080] Code value & 2: If the result is 2, B is selected

[0081] Code value & 4: If the result is 4, C is selected

[0082] Coded value & 8: If the result is 8, D is selected

[0083] Code:

[0084] def encode_options(options):

[0085] # Assume options is a list or set containing the selected option characters

[0086] # For example: ['A', 'B'] or {'A', 'B'}

[0087] encoded_value = 0

[0088] for option in options:

[0089] if option == 'A':

[0090] encoded_value |= 1

[0091] elif option == 'B':

[0092] encoded_value |= 2

[0093] elif option == 'C':

[0094] encoded_value |= 4

[0095] elif option == 'D':

[0096] encoded_value |= 8

[0097] else:

[0098] raise ValueError(f"Invalid option: {option}")

[0099] return encoded_value

[0100] def decode_options(encoded_value):

[0101] options = []

[0102] if encoded_value&1:

[0103] options.append('A')

[0104] if encoded_value&2:

[0105] options.append('B')

[0106] if encoded_value&4:

[0107] options.append('C')

[0108] if encoded_value&8:

[0109] options.append('D')

[0110] return option

[0111] Some examples and features of the matrix generation for the coded answer category are shown in Table 2 below:

[0112] Table 2 Matrix of answers to some questions by some candidates

[0113]

[0114] S3: Normalized vector visual display:

[0115] For one or more samples (candidates), you can specifically choose to visualize these samples, with the attribute (question number) as the vertical axis and the time corresponding to each attribute as the horizontal axis, and draw a scatter plot on the horizontal and vertical axes. You can display multiple different features, such as the answer time, reading time, and modification time of a question together, and display the trajectories of different candidates (taking Zhang San and Wang Si as examples) (see Figure 2 Middle (a), Figure 2 As shown in (b), Figure 2 (a) shows the trajectory of candidate Zhang San. Figure 2 (b) is the trajectory display of candidate Wang Si) Putting them together can make an intuitive comparison.

[0116] Step 101: Perform similarity calculation on the matrix data to determine the similarity of answering behaviors between different candidates.

[0117] Based on the generated matrix data, the elements in all the attributes of each sample (candidate) can form a vector. In order to obtain the similarity between samples, the distance and similarity between the vectors of samples are calculated. Different measurement methods should be used for time trajectory and answer content.

[0118] Specifically, when the candidate's trajectory vector is sparse relative to the time dimension, the number of characters in the corresponding positions between equal-length strings is similarly calculated for the answer content matrix using the following formula:

[0119] ;

[0120] In the formula, is the XOR operation, , are different sample sets, and i is the label in the sample set.

[0121] The answer trajectory matrix obtained in step S2 is first rounded to the integer of the vector, for example, rounded down at intervals of 5 minutes * 60 seconds, so as to obtain a new matrix, as shown in Table 3:

[0122] Table 3 The time rounding matrix of some questions for some candidates

[0123]

[0124] The elements in the table are statistics of the answer time falling within a certain period of time, and the Hamming distance is calculated for the vector rounded by the time period.

[0125] In some embodiments, the correlation calculation of the time trajectory uses the Pearson correlation coefficient, and the Pearson correlation coefficient of two continuous variables (X, Y) is equal to the covariance cov(X, Y) between them divided by the product of the standard deviation of the continuous variables ( ). The coefficient value is always between -1.0 and 1.0. Variables close to 0 are called uncorrelated, and those close to 1 or -1 are called strongly correlated. Similar calculations for the time trajectory matrix use the following formula:

[0126] ;

[0127] In the formula, Represents the Pearson correlation coefficient of two continuous variables (X, Y). represents the covariance between two continuous variables (X, Y), It represents the product of the standard deviations of two continuous variables (X, Y). Represents two variables , The expected value of the product of the differences with their respective expectations.

[0128] Each type of calculation has its own characteristics. You can choose the appropriate method based on the characteristics of the data. The distance / similarity calculated by the above methods can get the similarity between candidates x and y. If you sort candidate x, you can also find the candidate who is most similar to candidate x.

[0129] Take a certain exam as an example, where the vector correlation between candidates 1991 and 1063 is 0.315038634. The actual query answer trajectory is as follows Figure 3 The correlation coefficient between candidates 1125 and 1156 is 0.9991242, and the answering trajectory is as follows. Figure 4 shown.

[0130] In some implementations, the original time data and answer data are a two-dimensional matrix of candidate number and question information. After calculation, an autocorrelation table of candidates is constructed, as shown in Table 4. The horizontal and vertical axes are the candidate numbers, and the elements are the correlations. The autocorrelation matrix can be further used to screen candidates with large correlation coefficients.

[0131] Table 4 Autocorrelation matrix

[0132]

[0133] In addition to calculating the similarity of candidates who have already answered, you can also conduct a preliminary screening of candidates who are suspected to be similar during the exam stage. First, consider setting a similarity value for each candidate when receiving the answer information, and set a time period x. When a candidate submits question n at time t, and another candidate submits question n within time tx, the candidate's similarity value is +1. Select a threshold or ratio, and perform vector distance similarity calculations on the candidate trajectories that exceed the threshold or ratio to avoid vector calculations for all candidates and reduce the amount of calculations. Then, for some of the selected candidates, perform similarity calculations with other candidates to find candidates with similar trajectories.

[0134] In some implementations, for the generated standard data set, if there is no clear information on the examinees' disciplinary violations and a list of disciplinary violations, unsupervised learning can be used.

[0135] Unsupervised learning is a common technique in machine learning that can automatically identify abnormal or non-conforming samples from data without explicit labels or classification information. Unsupervised learning methods are particularly applicable in anomaly detection scenarios based on paperless test taker answer information, because abnormal behavior is sometimes difficult to define or fully label in advance. Unsupervised learning is used to calculate the standardized data set, divide the data set into two or more classes, and apply the selected clustering algorithm to perform cluster analysis on the feature data. During the clustering process, the clustering algorithm will divide similar samples into the same cluster and dissimilar samples into different clusters. In this process, some clusters may contain a large number of similar normal samples, while other clusters may contain a small number of abnormal samples or be composed entirely of abnormal samples. The image features of the data clusters are checked to determine what kind of candidate information of a certain class may belong to.

[0136] Clustering algorithms are the core of unsupervised learning algorithms. Commonly used clustering algorithms include K-means, DBSCAN, and Hierarchical Clustering. Each algorithm has its own characteristics and applicable scenarios. You need to choose a suitable algorithm based on the characteristics of the data and the needs of anomaly detection.

[0137] Step 102: If the matrix data is unlabeled data, a clustering algorithm is used to perform unsupervised learning on the matrix data, and the candidates are divided into normal candidates and abnormal candidates based on the similarity of the answering behaviors between different candidates, and anomaly detection is performed on the abnormal candidates to determine the behavioral characteristics of the abnormal candidates.

[0138] Specifically, there are two cases when executing step 102:

[0139] Case 1: For a large range of data across test areas and test sites, the K-Means clustering algorithm is used to identify data points far away from the main clusters in the large range of candidate answer information data as outliers, and then identify abnormal candidates.

[0140] Specifically, for paperless exams, one possible risk is that there may be collective cheating organized on a large scale and distributed in different examination halls and even regions. The K-Means clustering algorithm is simple and fast, suitable for large data sets, and can maintain scalability and efficiency. This kind of clustering algorithm is very suitable for aggregating all the information of the entire examination in all regions, examination halls, and batches into a large data set, specifically for clustering and identifying collective cheating. As a classic clustering algorithm, it iteratively assigns data points to predefined K clusters, so that the points in each cluster are as similar as possible, and the points in different clusters are as different as possible. Data points far away from the main cluster can be identified as anomalies in the answer information data of a large range of candidates.

[0141] Assuming the input sample is S=x1, x2, ......, xm, the algorithm steps are:

[0142] Select the initial k category centers, , , ......, .

[0143] For each sample , mark the category closest to the category center, that is:

[0144] ;

[0145] Update each class center to the mean of all samples belonging to that class.

[0146] ;

[0147] In the formula, For the Classification clusters, is the center of the cluster.

[0148] Repeat the next two steps until the category center change is less than a certain threshold.

[0149] Termination conditions: number of iterations, cluster center change rate, minimum square error MSE.

[0150] Case 2: For the data of a small range of test sites, a single-class support vector machine is used to take the data points whose distance from the small range of test sites to the center point is less than or equal to the critical distance as outliers, and then identify the abnormal candidates.

[0151] Specifically, small-scale cheating often occurs at a test site with the same geographical location, which has several examination rooms. To address this type of anomaly, we can calculate it separately for each test site.

[0152] One-Class Support Vector Machine (One-Class SVM) is a special type of Support Vector Machine (SVM). Its purpose is to separate data points of different categories. It is mainly used for anomaly detection or outlier detection. Its goal is to build a hyperplane (or hypersphere) by modeling only using normal data, so that normal data points are enclosed inside and abnormal data points are outside the hyperplane (or hypersphere), and identify outlier data points that are different from the normal pattern.

[0153] One-Class SVM only requires normal data (unlabeled) for training, no abnormal data is needed, which makes it particularly useful when abnormal data is difficult to obtain or label. Especially in high-dimensional data, or when the distribution of answers is not known in advance and no assumptions are made, One-Class SVM can be used as a good outlier calculation method.

[0154] One-Class SVM requires training sample data The distance to the center point is Within, and at the same time construct the penalty coefficient as The slack variables are optimized as follows:

[0155]

[0156]

[0157]

[0158] In the formula, is the cluster center point, is the slack variable.

[0159] After optimization, the distance from the sample point to the center is calculated to determine whether it is an outlier. After using this method for calculation, the anomaly is identified by clustering results. One method is to define anomalies based on the size or density of the cluster. For example, if the number of samples in a cluster is much smaller than that of other clusters, or the sample density in the cluster is much lower than the average level, then this cluster is likely to contain anomalies. In addition, the distance from each sample to the center of its cluster can also be calculated. Samples with a longer distance are more likely to be anomalies.

[0160] Unsupervised learning methods are generally based on density estimation. They are a good method for large data sets that are inconvenient to label. For paperless exam anomaly detection tasks, unsupervised learning can be used to pre-screen a large number of samples, verify and mark the pre-screened data, and then combine the information summarized after the exam to more accurately and easily label the data set. With a labeled data set, you can use a supervised machine learning model algorithm.

[0161] Step 103: If the matrix data is labeled data, supervised learning is used to perform anomaly detection on the marked abnormal candidate samples to determine the behavioral characteristics of the abnormal candidates.

[0162] Specifically, when executing step 103, the following steps may be specifically performed:

[0163] If the matrix data is labeled data, the XGBoost ensemble learning method is used to perform anomaly detection on the marked abnormal candidate samples to determine the behavioral characteristics of the abnormal candidates;

[0164] Xtreme Gradient Boosting is used to train the labeled data set. XGBoost is an ensemble learning method based on the gradient boosting framework. It optimizes the prediction performance of the model by iteratively training multiple decision trees. XGBoost is efficient and optimizes the execution speed and memory usage of the algorithm compared to other methods. It is also flexible and supports custom loss functions and evaluation criteria, which is suitable for a variety of tasks. By integrating multiple weak learners, the prediction accuracy of the model can be significantly improved, complex patterns can be learned in anomaly detection, and abnormal behaviors can be effectively identified.

[0165] Among them, the objective function of the XGBoost ensemble learning method is as follows:

[0166]

[0167] In the formula, As the loss function to measure the difference between the predicted result and the actual result, the loss function uses the mean square error. For the The true value of the samples, For the Sample prediction values, is the weak evaluator, i.e. the decision tree added, t is the number of prediction rounds, is the penalty term, which is the sum of the number of leaf nodes and the coefficient multiplied by the square of the weight L2 modulus. is a constant term.

[0168] In some implementations, the flowchart of the method for detecting anomalies in examinee answer information further includes the following execution steps:

[0169] The examinee's answer information is regarded as discretely distributed time series data, and a time series model is constructed; the examinee's historical answer behavior data is input into the time series model to predict the examinee's future answer behavior.

[0170] Specifically, firstly, the time trajectory data is further classified and normalized:

[0171] When training the time series model, the standard data is further divided into different data sets for training based on the same examination subjects, the same question types, and the same number of question types.

[0172] In the matrix of candidate behavior time, the information contained in each candidate vector is a time series, and the past behavior time points will have an impact on the future. By training the corresponding model with the candidate's behavior vector matrix as a time series, the time when the candidate's behavior may occur in the future can be predicted based on the candidate's past behavior.

[0173] When performing time series prediction, the time data needs to be further normalized to reduce the impact of the relative time difference when switching between different types of questions. Use zero mean normalization to normalize the time trajectory data to a data set with a variance of 1 and a mean of 0, according to the following formula:

[0174]

[0175] In the formula, and are the mean and variance, is the original data, is the normalized data.

[0176] The variance of the normalized time trajectory data is consistent, which can avoid errors caused by inconsistent dimensions of data in different dimensions in the subsequent calculation steps.

[0177] Next, the moving average model is used for prediction:

[0178] The Autoregressive Integrated Moving Average (ARIMA) model is used to predict the normalized data. The ARIMA model is one of the classic and widely used methods in time series prediction. It combines the three elements of autoregression (AR), difference (I) and moving average (MA), and is particularly suitable for processing time series data with trend and seasonal characteristics. In the prediction of candidate behavior, ARIMA can help capture the long-term trend and cyclical pattern of behavior changes over time, thereby predicting the time of future behavior.

[0179] Finally, set the anomaly rules to deploy the prediction model:

[0180] The moving average model is used to train the time series prediction model and deploy it on the test server. Further settings are required to predict the behavior of candidates. The specific settings are as follows:

[0181] Set the residual threshold. When the difference between the predicted value and the true value is greater than a certain value, it will be considered as an abnormal situation.

[0182] Set the anomaly count. When the residual between the predicted value and the true value exceeds the threshold, the count increases by 1.

[0183] Set the counting threshold and counting clearing rules. When the count is greater than the threshold, the system will issue an early warning to inform the invigilator or examination staff to check the situation. The counting value can be cleared manually after the examination staff confirms that it is correct. The other is to set the time or number of questions after which the count is cleared or the count value is reduced according to the specific examination situation.

[0184] At the same time, the principle of data set division is also followed when deploying the prediction model, and the models trained with the same division are correctly used in the same type of exams.

[0185] In some implementations, the implementation process of the anomaly detection method for examinee answer information also includes evaluating the performance of the anomaly detection model through evaluation indicators such as cross-validation, confusion matrix, ROC curve, etc. Based on the evaluation results, the model is optimized and adjusted to improve the accuracy and robustness of its anomaly detection. New examinee answer data is continuously collected, and the model is iteratively updated to adapt to the ever-changing examination environment and examinee behavior patterns.

[0186] 1. Similarity calculation model evaluation:

[0187] The model performance is evaluated by comparing the similarity predicted by the model with the actual annotations (human-judged similarity). Cross-validation can be used to ensure the stability and reliability of the evaluation results. During the evaluation process, attention should be paid to identifying overfitting or underfitting phenomena, as well as the model's sensitivity to outliers.

[0188] Commonly used evaluation indicators for similarity calculation models include accuracy (for similarity / dissimilarity judgment in classification tasks), mean average precision (MAP), etc. In addition, indicators such as recall rate and F1 score can also be examined based on specific application scenarios.

[0189] 2. Unsupervised learning model evaluation:

[0190] The results of unsupervised learning can be visualized by reducing the dimension, see Figure 5 As shown in the figure, internal evaluation indicators including Silhouette Score and Calinski-Harabasz index (i.e., variance ratio criterion) are used to measure the quality of clustering effect. If there is external labeled data (such as known cluster labels), external evaluation indicators such as accuracy and NMI (Normalized Mutual Information) can be used to evaluate the consistency between clustering results and true labels. In addition, k-fold crossover data can be used in calculations, and the model can be run multiple times to compare the consistency of the results to evaluate the stability of the model.

[0191] 3. Supervised learning model evaluation:

[0192] K-fold cross validation and other methods are used to ensure the generalization ability of the evaluation results. Confusion matrix, accuracy, precision, recall, sensitivity and specificity, F1 score, ROC curve and AUC value are used to evaluate the performance of the classification model. The evaluation indicators after training the model can be displayed intuitively. For the accuracy and recall figures, see Figure 6 As shown, specificity and sensitivity refer to Figure 7 shown.

[0193] 4. Time Series Model Evaluation:

[0194] For the prediction accuracy of the time series model, regression evaluation indicators (such as MSE, RMSE) are used to measure the prediction accuracy of the time series model. For the evaluation of model prediction lag, it is necessary to evaluate the response speed of the model to the changes in the time series to ensure that the model can capture and predict trend changes in a timely manner. Robustness is also an important indicator. By adding noise or outliers to the time series data, the robustness and robustness of the model are tested.

[0195] 5. Model optimization:

[0196] Further feature transformation is performed on the data set, and feature distribution is improved through standardization, normalization, encoding, etc., to improve the efficiency of model training. There are several methods for model parameter tuning. The grid search method is used to exhaustively search the key parameters of the model to find the optimal parameter combination. The random search method is used to randomly sample in the parameter space to find a better parameter combination, which is more efficient than the grid search. Bayesian optimization is used to use the Bayesian theorem to intelligently select the next search direction through historical search results to accelerate the search for the optimal solution. For subsequent algorithm optimization, a hybrid model can be used to combine the advantages of multiple algorithms to build a hybrid model to improve prediction performance. For example, the clustering results of unsupervised learning are combined with the classifiers of supervised learning. Or ensemble learning is used, using ensemble learning methods such as Bagging and Boosting to improve the overall performance by combining the prediction results of multiple base learners. Finally, the abnormal algorithm model also needs to be updated and maintained. The model needs to be continuously learned. As new data is generated, the model is regularly updated to adapt to changes in data distribution.

[0197] This application can achieve the following functions based on the anomaly detection method of paperless test taker answer information:

[0198] 1. Generate standardized candidate answer information data;

[0199] 2. Screening for abnormalities and similarities among candidates;

[0200] 3. Testing of candidates suspected of collective cheating;

[0201] 4. Prediction and early warning of abnormal behavior during the exam;

[0202] Among them, the standardized candidate answer information data uses the original data recorded in the paperless test, extracts useful information to form an appropriate data structure, and formulates a standard data set format. The data set has the characteristics of including key information, accuracy, and easy calculation and training. The candidate abnormality and similarity screening uses the real-time data of the answers in the test to accurately quantify the similarity values ​​of the behaviors and answer contents between the candidates. The identification of candidates suspected of collective cheating identifies batches of candidates with similar and identical behaviors. Multiple candidates with similar abnormalities can be identified and warned as suspected collective cheating. The prediction and warning of abnormal behavior in the test predicts the next behavior of the candidates based on the past paperless test information and the information that the candidates have answered, which can be used as an auxiliary judgment of abnormalities. According to the model learned in the past or the rules set by the test organizer, the suspected abnormal situation in the candidate's answering process is warned to notify the invigilator or the test organizer.

[0203] See also Figure 8 FIG. 1 is a schematic diagram of a structure of an abnormal detection system for examinee answer information provided by an embodiment of the present invention, comprising:

[0204] The acquisition unit 80 is used to acquire the answering process and answering information of the examinee during the paperless examination, and extract key features in the answering process and answering information to generate corresponding matrix data;

[0205] A calculation unit 81 is used to perform similarity calculation on the matrix data to determine the similarity of answering behaviors between different examinees;

[0206] The first anomaly detection unit 82 is used to use a clustering algorithm to perform unsupervised learning on the matrix data if the matrix data is unlabeled data, and to distinguish the examinees into normal examinees and abnormal examinees based on the similarity of answering behaviors between different examinees, and to perform anomaly detection on the abnormal examinees to determine the behavioral characteristics of the abnormal examinees;

[0207] The second anomaly detection unit 83 is used to perform anomaly detection on the marked abnormal examinee samples by using a supervised learning method if the matrix data is labeled data, so as to determine the behavioral characteristics of the abnormal examinees.

[0208] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method for detecting anomalies in examinee answer information as described in any of the above-mentioned embodiments.

[0209] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0210] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0211] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0212] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0213] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0214] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0215] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0216] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0217] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A method for detecting anomalies in examinee answer information, characterized in that: include: Obtain the answering process and answering information of the examinee during the paperless examination, and extract key features from the answering process and answering information to generate corresponding matrix data; Perform similarity calculations on the matrix data to determine the similarity of answering behaviors between different candidates; If the matrix data is unlabeled data, a clustering algorithm is used to perform unsupervised learning on the matrix data, and the examinees are divided into normal examinees and abnormal examinees based on the similarity of the answering behaviors of different examinees, and anomaly detection is performed on the abnormal examinees to determine the behavioral characteristics of the abnormal examinees; If the matrix data is labeled data, supervised learning is used to perform anomaly detection on the marked abnormal examinee samples to determine the behavioral characteristics of the abnormal examinees; The candidate's answering process and answering information include but are not limited to test center information, test room information, test area information, test batch information, candidate information, test subject information, test paper number information, test paper question number information, test subject question sequence information, test question structure information, candidate answering time information, candidate browsing time information, candidate modifying time information and candidate answer content information; The key features in the answering process and the answering information are extracted to generate corresponding matrix data, including: Taking answering time as the key factor, extract the answering time of each candidate for each question, and construct a matrix with the elements corresponding to the horizontal and vertical coordinates as time, using the candidate number and the question number as the horizontal and vertical coordinates respectively; Taking the answer content as the key factor, extract the answer content of each candidate for each question, and use the candidate number and question number as the horizontal and vertical coordinates to construct a matrix in which the elements corresponding to the horizontal and vertical coordinates are the answer content; Taking browsing time and revision times as key factors, the answering time of each candidate for each question is extracted, and the candidate number and question number are used as the horizontal and vertical axes, plus a column of revision times, to construct a matrix in which the horizontal and vertical axes correspond to browsing time and revision times.

2. The method for detecting anomalies in examinee answer information according to claim 1, characterized in that: Perform similarity calculations on the matrix data to determine the similarity of answering behaviors between different candidates, including: For similar calculations of the answer content matrix, the following formula is used: ； In the formula, is the XOR operation, , are different sample sets, and i is the label in the sample set.

3. The method for detecting abnormalities in examinee answer information according to claim 1, characterized in that: Perform similarity calculations on the matrix data to determine the similarity of answering behaviors between different candidates, including: For similar calculation of the time trajectory matrix, the following formula is used: ； In the formula, Represents the Pearson correlation coefficient of two continuous variables (X, Y). represents the covariance between two continuous variables (X, Y), It represents the product of the standard deviations of two continuous variables (X, Y). Represents two variables , The expected value of the product of the differences with their respective expectations.

4. The method for detecting anomalies in examinee answer information according to claim 1, characterized in that: If the matrix data is labeled data, supervised learning is used to perform anomaly detection on the marked abnormal candidate samples to determine the behavioral characteristics of the abnormal candidates, including: If the matrix data is labeled data, the XGBoost ensemble learning method is used to perform anomaly detection on the marked abnormal candidate samples to determine the behavioral characteristics of the abnormal candidates; Among them, the objective function of the XGBoost ensemble learning method is as follows: In the formula, As the loss function to measure the difference between the predicted result and the actual result, the loss function uses the mean square error. For the The true value of the samples, For the Sample prediction values, is the weak evaluator, i.e. the decision tree added, t is the number of prediction rounds, is the penalty term, which is the sum of the number of leaf nodes and the coefficient multiplied by the square of the weight L2 modulus. is a constant term.

5. The method for detecting anomalies in examinee answer information according to claim 1, characterized in that: The method for detecting abnormalities in the examinee's answer information further includes: Treat the examinees' answer information as discretely distributed time series data and construct a time series model; The obtained historical answering behavior data of candidates are input into the time series model to predict the candidates' future answering behavior.

6. A system for detecting anomalies in examinee's answer information applied to the method for detecting anomalies in examinee's answer information according to any one of claims 1 to 5, characterized in that: include: An acquisition unit is used to acquire the answering process and answering information of the examinee during the paperless examination, and extract key features in the answering process and answering information to generate corresponding matrix data; A calculation unit, used for performing similarity calculation on the matrix data to determine the similarity of answering behaviors between different examinees; A first anomaly detection unit is used to perform unsupervised learning on the matrix data using a clustering algorithm if the matrix data is unlabeled data, and to distinguish the examinees into normal examinees and abnormal examinees based on the similarity of answering behaviors between different examinees, and to perform anomaly detection on the abnormal examinees to determine the behavioral characteristics of the abnormal examinees; The second anomaly detection unit is used to perform anomaly detection on the marked abnormal candidate samples by using a supervised learning method if the matrix data is labeled data, so as to determine the behavioral characteristics of the abnormal candidates.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for detecting anomalies in examinee's answer information as described in any one of claims 1 to 5 are implemented.

Images