An intelligent evaluation method for college students' artificial intelligence literacy based on BP neural network evaluation algorithm

Abstract

This invention is applicable to the fields of educational assessment and intelligent data processing, providing a method for intelligent evaluation of college students' artificial intelligence literacy based on a BP neural network evaluation algorithm. The invention constructs an evaluation index system for college students' artificial intelligence literacy, designs a structured questionnaire, collects valid questionnaire data, calculates a percentage-based comprehensive score, classifies literacy levels, and constructs initial sample data; selects the optimal feature subset, divides it into training and testing sets, and generates a modeling dataset; determines the optimal number of hidden layer nodes and learning rate, uses the modeling dataset for model training, and generates an optimal BP neural network evaluation model; inputs the data to be evaluated into the optimal BP neural network evaluation model, outputs an intelligent evaluation report, and regularly updates the optimal BP neural network evaluation model. This invention enables the quantitative assessment and level determination of college students' artificial intelligence literacy, outputs an intelligent evaluation report, and provides reusable methodological support for similar literacy evaluations.

Description

Technical Field

[0001] This invention belongs to the field of educational assessment and intelligent data processing, and in particular relates to an intelligent evaluation method for college students' artificial intelligence literacy based on a BP neural network evaluation algorithm. Background Technology

[0002] In an era of rapid popularization of artificial intelligence technology, college students' AI literacy has become a core indicator for measuring the quality of talent cultivation, and is directly related to students' employment competitiveness and social adaptability.

[0003] Currently, universities mostly use traditional methods such as questionnaire scoring, expert review, and simple weighted statistics to evaluate college students' artificial intelligence literacy, which have obvious limitations: First, the evaluation is highly subjective, with expert scores easily influenced by personal cognitive preferences, and the weighting of indicators lacks data support; second, the indicator system is fragmented, with existing evaluations focusing on a single dimension of AI knowledge mastery or tool use, lacking a systematic integration of knowledge, skills, ethics, and innovation; third, data utilization is low, failing to effectively integrate multi-source evaluation data and making it difficult to quantify the impact of each indicator on literacy levels; and fourth, generalization ability is weak, with traditional models unable to adapt to the differentiated evaluation needs of students from different majors and levels, resulting in insufficient stability of evaluation results.

[0004] Backpropagation (BP) neural networks, as classic multilayer feedforward neural networks, possess powerful nonlinear fitting, adaptive learning, and complex pattern recognition capabilities. They can predict comprehensive scores and determine competency levels by learning the mapping relationship between indicator features and comprehensive scores, and have been widely used in fields such as education evaluation and competency prediction.

[0005] While existing research attempts to use BP neural networks for student literacy assessment, it still has shortcomings in terms of the scientific nature of the indicator system, data quality control, model structure optimization, and the interpretability of evaluation results: First, there is a lack of standardized AI literacy assessment indicators suitable for university students, with both indicator redundancy and dimension missing; second, the data preprocessing process is imperfect, failing to adequately remove abnormal samples and invalid data, resulting in impaired model training accuracy; third, data preprocessing and feature selection are not strictly limited to the training set, which may lead to test data leakage; and fourth, network parameters rely on empirical settings, and the evaluation results lack dimensional diagnosis and improvement suggestions, failing to form a closed loop of "evaluation-feedback-improvement". Summary of the Invention

[0006] The purpose of this invention is to provide an intelligent evaluation method for college students' artificial intelligence literacy based on a BP neural network evaluation algorithm, aiming to solve the technical problems existing in the prior art mentioned in the background.

[0007] The embodiments of the present invention are implemented as follows:

[0008] A method for intelligently evaluating college students' artificial intelligence literacy based on a BP neural network evaluation algorithm, the method specifically includes the following steps:

[0009] Construct an evaluation index system for college students' artificial intelligence literacy and design a structured questionnaire;

[0010] University students were selected as the research subjects. Valid questionnaire data were collected through the structured questionnaire, and a comprehensive score on a percentage scale was calculated to classify the literacy levels and construct the initial sample data.

[0011] The initial sample data were subjected to questionnaire quality control and reliability and validity testing. Stratified sampling was carried out according to literacy levels and the training set and test set were divided. Based on the training set, missing value handling parameters, normalization parameters and optimal feature subsets were determined and applied to the test set to generate the modeling dataset.

[0012] A three-layer BP neural network model is constructed, the optimal number of hidden layer nodes and learning rate are determined, the model is trained using the training set, and the model is evaluated using the test set to determine the optimal BP neural network evaluation model.

[0013] After the model evaluation is passed, the data to be evaluated is input into the optimal BP neural network evaluation model, the predicted comprehensive score is output and mapped to the literacy level, an intelligent evaluation report is generated, and the optimal BP neural network evaluation model is updated regularly.

[0014] As a further limitation of the technical solution of this invention, the evaluation index system for college students' artificial intelligence literacy includes four primary indicators: theoretical cognition, application skills, ethical literacy, and innovation ability. The four primary indicators are composed of sixteen secondary indicators. The theoretical cognition includes basic concept cognition, machine learning principles, deep learning fundamentals, and data algorithm fundamentals; the application skills include tool operation skills, data processing capabilities, model deployment and cross-scenario application; the ethical literacy includes privacy protection awareness, recognition of algorithmic bias, adherence to responsibility norms, and ethical decision-making ability; and the innovation ability includes problem transformation ability, model optimization thinking, interdisciplinary integration innovation, and critical reflection ability.

[0015] As a further limitation of the technical solution of this invention embodiment, the step of selecting college students as the survey subjects, collecting valid questionnaire data through the structured questionnaire, calculating a comprehensive score on a percentage basis, classifying literacy levels, and constructing initial sample data specifically includes the following steps:

[0016] The survey subjects were college students.

[0017] The structured questionnaire was used to collect survey data from multiple respondents.

[0018] Invalid questionnaires and abnormal samples were removed from the questionnaire survey data to obtain valid questionnaire data;

[0019] Based on the sixteen secondary indicators, the valid questionnaire data were analyzed, and the composite score of multiple survey participants was calculated on a percentage basis.

[0020] Based on the comprehensive score of multiple percentage points, the corresponding literacy levels are divided;

[0021] Construct initial sample data.

[0022] As a further limitation of the technical solution of this embodiment of the invention, the questionnaire survey data consists of basic information and indicator data of multiple survey subjects. The basic information includes the gender, education level, grade, school type, major, and use of smart tools of the survey subjects. The classification standard of the literacy level is as follows: [90,100] is level A, [80,90) is level B, [70,80) is level C, [60,70) is level D, and <60 is level E.

[0023] As a further limitation of the technical solution of this invention, the steps of performing questionnaire quality control and reliability and validity testing on the initial sample data, performing stratified sampling according to literacy levels and dividing the data into training and test sets, determining missing value handling parameters, normalization parameters and optimal feature subsets based on the training set, and applying them to the test set to generate the modeling dataset specifically include the following steps:

[0024] The initial sample data were subjected to questionnaire quality control and reliability and validity testing.

[0025] Stratified sampling was performed according to literacy levels to divide the initial sample data into training and test sets;

[0026] Based on the training set, determine the missing value handling parameters, normalization parameters, and optimal feature subset;

[0027] Using the sixteen secondary indicators in the training set as candidate features, the mutual information method is used to calculate the correlation between each candidate feature and the literacy level, and the Pearson correlation coefficient is used to remove highly redundant features, and the optimal feature subset consisting of K indicators is selected, where K is a positive integer not greater than 16.

[0028] The missing value handling parameters, the normalization parameters, and the optimal feature subset are used to process the training set and the test set respectively to generate a modeling dataset.

[0029] As a further limitation of the technical solution of this embodiment of the invention, the reliability and validity test adopts Cronbach's α coefficient, KMO test and Bartlett's test of sphericity, wherein the Cronbach's α coefficient is not less than 0.8, the KMO value is not less than 0.7, and the significance level of Bartlett's test of sphericity is p < 0.05; the outlier correction consists of outlier handling and missing value handling, wherein the outlier handling adopts the 3σ criterion, and the missing value handling adopts the mean imputation method based on the training set statistics; the normalization processing adopts the min-max standardization method, using only the minimum and maximum values ​​in the training set to determine the normalization parameters, and applying the normalization parameters to the training set, test set and data to be evaluated; in the removal of highly redundant features, the threshold for removing highly redundant indicators is the absolute value of the Pearson correlation coefficient ≥ 0.8; the ratio of the training set to the test set is 7:3.

[0030] As a further limitation of the technical solution of this embodiment of the invention, the steps of constructing a three-layer BP neural network model, determining the optimal number of hidden layer nodes and learning rate, training the model using the training set, and evaluating the model using the test set to determine the optimal BP neural network evaluation model specifically include the following steps:

[0031] Construct a three-layer BP neural network model;

[0032] The optimal number of hidden layer nodes and learning rate are determined by using 5-fold hierarchical cross-validation and grid search.

[0033] The model is trained using the training set, and the optimal parameter combination is selected in 5-fold hierarchical cross-validation. The candidate evaluation model is then trained on the complete training set using the optimal parameter combination.

[0034] The test set evaluation model is evaluated based on accuracy, precision, recall, F1 score, MSE, and Kappa coefficient, and it is determined whether the model evaluation passes.

[0035] After the model evaluation is passed, the candidate evaluation model is determined as the optimal BP neural network evaluation model.

[0036] As a further limitation of the technical solution of the embodiment of the present invention, in the three-layer BP neural network model, the number of neurons in the input layer is equal to K, the hidden layer adopts the Sigmoid activation function, and the output layer adopts the linear activation function; the candidate value of the optimal number of hidden layer nodes is 10 to 100; the candidate value of the learning rate is 0.01 to 0.1; the maximum number of training iterations of the model is 1000, and the error convergence threshold is 0.001.

[0037] As a further limitation of the technical solution of this invention embodiment, after the model evaluation is passed, the data to be evaluated is input into the optimal BP neural network evaluation model, the predicted comprehensive score is output and mapped to the literacy level, an intelligent evaluation report is generated, and the optimal BP neural network evaluation model is updated regularly, specifically including the following steps:

[0038] After the model evaluation is passed, the data to be evaluated is input into the optimal BP neural network evaluation model;

[0039] The optimal BP neural network evaluation model outputs a comprehensive score, which is then mapped to a competency level, and weak dimensions are identified.

[0040] Based on the overall score, the competency level, and the weak dimensions, an intelligent evaluation report is generated;

[0041] Collect new sample data regularly;

[0042] When the input feature structure of the newly added sample data is consistent with the existing model, the newly added sample data is processed using the existing data processing parameters and the optimal feature subset, and incremental training and parameter updates are performed on the optimal BP neural network evaluation model.

[0043] As a further limitation of the technical solution of this embodiment of the invention, the method further includes the following steps:

[0044] The evaluation index system for college students' artificial intelligence literacy is adaptively updated and adjusted; when the adjustment causes changes in the input feature structure of the model, the BP neural network evaluation model is retrained and evaluated based on the updated sample data.

[0045] Compared with the prior art, the beneficial effects of the present invention are:

[0046] (1) This invention constructs a standardized evaluation index system covering four dimensions: AI theoretical cognition, AI application skills, AI ethical literacy, and AI innovation ability. This system makes up for the shortcomings of existing evaluation dimensions being fragmented and emphasizing knowledge over ethical innovation. It comprehensively covers the core connotation of college students' AI literacy, with more complete evaluation dimensions and indicators that are more in line with the characteristics of college students.

[0047] (2) This invention effectively removes invalid data and redundant features through multiple stages of data preprocessing, such as reliability and validity testing, outlier data removal, normalization processing, and feature optimization, thereby reducing the interference of noisy data on the model, improving the reliability and effectiveness of input data, and providing high-quality data support for model training.

[0048] (3) This invention adopts a three-layer BP neural network structure, determines the model parameters through grid optimization and 5-fold cross-validation, learns the mapping relationship between the optimal feature subset and the comprehensive score, realizes comprehensive score prediction and literacy level determination, and overcomes the defects of strong subjectivity in traditional evaluation. In specific implementation, the model can be comprehensively verified by test set accuracy, F1 score, MSE and Kappa coefficient, and the model prediction performance can be continuously optimized according to sample data.

[0049] (4) This invention not only outputs the comprehensive score and level of AI literacy, but also accurately identifies the weak dimensions and provides personalized improvement suggestions, breaking the limitation of traditional evaluation that "emphasizes results and neglects diagnosis", realizing the closed loop of "evaluation-diagnosis-improvement", and providing accurate data support for the reform of AI teaching in colleges and universities, the optimization of curriculum settings and the cultivation of students' personalized abilities;

[0050] (5) This invention supports incremental training and parameter updates based on new samples when the input feature structure remains unchanged; when the evaluation index system is adjusted, the input feature structure changes, the model is retrained and evaluated, thus taking into account both the efficiency of model updates and the effectiveness of evaluation results. Attached Figure Description

[0051] Figure 1 The flowchart of the intelligent evaluation method for college students' artificial intelligence literacy based on the BP neural network evaluation algorithm provided in the embodiment of the present invention is shown.

[0052] Figure 2 The hierarchical structure diagram of the evaluation index system for college students' artificial intelligence literacy in the method provided by the embodiments of the present invention is shown.

[0053] Figure 3 This diagram illustrates the three-layer structure of the BP neural network model in the method provided by an embodiment of the present invention.

[0054] Figure 4 This diagram illustrates the model training and parameter optimization process in the method provided by an embodiment of the present invention.

[0055] Figure 5 A bar chart showing the model performance evaluation results and comparative model metrics in the method provided by the embodiments of the present invention is shown.

[0056] Figure 6 This diagram illustrates the framework for intelligent evaluation report output in the method provided by an embodiment of the present invention. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0058] Understandably, while existing technologies attempt to use BP neural networks for evaluating students' AI literacy, they still have shortcomings in areas such as the scientific nature of the indicator system, data quality control, model structure optimization, and the interpretability of evaluation results. First, there is a lack of standardized AI literacy evaluation indicators suitable for university students, with both redundant indicators and missing dimensions. Second, the data preprocessing process is incomplete, failing to adequately remove abnormal samples and invalid data, leading to impaired model training accuracy. Third, the boundaries between data preprocessing and feature selection are unclear, and using test data to determine processing parameters may cause data leakage. Fourth, network parameters rely on experience-based settings, and the evaluation results lack dimensional diagnosis and improvement suggestions, making it difficult to form a closed loop of "evaluation-feedback-improvement."

[0059] To address the aforementioned issues, this invention discloses an intelligent evaluation method for college students' artificial intelligence literacy based on a BP neural network evaluation algorithm. This method involves constructing an evaluation index system for college students' artificial intelligence literacy and designing a structured questionnaire. College students are selected as the survey subjects, and valid questionnaire data is collected through the structured questionnaire. A percentage-based comprehensive score is calculated, and literacy levels are categorized to construct initial sample data. The initial sample data undergoes questionnaire quality control and reliability / validity testing. Stratified sampling is performed according to literacy levels, and training and test sets are divided. Missing value handling parameters, normalization parameters, and the optimal feature subset are determined based on the training set and applied to the test set to generate a modeling dataset. A three-layer BP neural network model is constructed, and the optimal number of hidden layer nodes and learning rate are determined. The model is trained using the training set and evaluated using the test set to determine the optimal BP neural network evaluation model. After the model evaluation is passed, the data to be evaluated is input into the optimal BP neural network evaluation model, which outputs a predicted comprehensive score and maps it to the literacy level, generating an intelligent evaluation report. The optimal BP neural network evaluation model is updated periodically. It can quantitatively assess and grade college students' artificial intelligence literacy, output intelligent evaluation reports, and provide reusable methodological support for similar literacy assessments.

[0060] It can quantitatively assess and grade college students' artificial intelligence literacy, output intelligent evaluation reports, and provide reusable methodological support for similar literacy assessments.

[0061] Specifically, Figure 1 The flowchart illustrates a method for intelligent evaluation of college students' artificial intelligence literacy based on a BP neural network evaluation algorithm, as provided in an embodiment of the present invention.

[0062] In a preferred embodiment of the present invention, a method for intelligently evaluating the artificial intelligence literacy of college students based on a BP neural network evaluation algorithm is provided, the method specifically including the following steps:

[0063] Step 1: Construct an evaluation index system for college students' artificial intelligence literacy and design a structured questionnaire.

[0064] In embodiments of the present invention, such as Figure 2 The diagram illustrates the hierarchical structure of the evaluation index system for college students' artificial intelligence literacy in the method provided by this embodiment of the invention. This evaluation index system is constructed around the knowledge base, application ability, ethical awareness, and innovative development ability of college students' artificial intelligence literacy, and includes four categories of factors: theoretical cognition, application skills, ethical literacy, and innovative ability. Specific evaluation indicators are shown in the table below:

[0065] Step 2: Select college students as the research subjects, collect valid questionnaire data through the structured questionnaire, calculate the comprehensive score on a percentage scale, classify the literacy levels, and construct the initial sample data.

[0066] In this embodiment of the invention, university students are selected as the research subjects. A structured questionnaire is used to collect questionnaire data from multiple subjects. Invalid questionnaires and abnormal samples are removed from the questionnaire data to obtain valid questionnaire data. Then, based on sixteen secondary indicators, the valid questionnaire data is analyzed to calculate the comprehensive percentage score of multiple subjects. Based on the comprehensive percentage score, corresponding competency levels are assigned to construct initial sample data. The questionnaire data consists of basic information and indicator data from multiple subjects. The basic information includes the subjects' gender, education level, grade, school type, major, and use of smart tools. The competency level assignment criteria are: [90, 100] is level A, [80, 90) is level B, [70, 80) is level C, [60, 70) is level D, and <60 is level E.

[0067] Understandably, to facilitate stratified data statistics and consistency verification, this embodiment simultaneously collects basic information of the survey subjects, and the relevant data is shown in the table below:

[0068]

[0069] In this embodiment, a total of 2,131 questionnaires were distributed and 1,591 were returned, of which 1,019 were valid.

[0070] In this embodiment of the invention, the comprehensive score of artificial intelligence literacy for each sample is calculated based on the scores of each indicator. All sixteen secondary indicators use a Likert 5-level scoring method, and the original score for each indicator ranges from 1 to 5. Let the original score of the j-th indicator be x. j Then the comprehensive score of artificial intelligence literacy for each sample is S=[Σxj / (16×5)]×100, where S is the overall score on a percentage basis. Further, the overall score is divided into five competency levels: A, B, C, D, and E, according to [90,100], [80,90), [70,80), [60,70), and <60, respectively.

[0071] It is understood that, in this embodiment of the invention, the basic information of the survey subjects, the data of sixteen secondary indicators, the comprehensive score on a percentage scale, and the literacy level labels are matched at the sample level to construct the initial sample data of college students' artificial intelligence literacy.

[0072] It is understood that, in the embodiments of the present invention, invalid questionnaires, abnormal samples and invalid items in the questionnaire survey data are processed. Invalid questionnaires include those with a missed answer rate exceeding a preset threshold, options with a regular distribution, and answer time that is obviously abnormal. Abnormal samples include those with indicator scores that exceed a reasonable range, obvious contradictions in answers to different items within the same questionnaire, and serious missing key fields.

[0073] Step 3: Perform questionnaire quality control and reliability and validity testing on the initial sample data. Perform stratified sampling according to literacy levels and divide the data into training set and test set. Based on the training set, determine the missing value handling parameters, normalization parameters and optimal feature subset, and apply them to the test set to generate the modeling dataset.

[0074] In this embodiment of the invention, the initial sample data is first subjected to questionnaire quality control and reliability and validity testing. The training set and test set are then stratified according to literacy levels in a 7:3 ratio. Subsequently, outlier processing and normalization parameter determination are performed only on the training set. Using sixteen secondary indicators from the training set as candidate features, the mutual information method is employed to calculate the correlation between each candidate feature and the literacy level. Highly redundant features are then eliminated using the Pearson correlation coefficient, and an optimal feature subset consisting of K indicators is selected, where K is a positive integer not greater than 16. Afterward, the data processing parameters and the optimal feature subset determined based on the training set are applied to the training set and test set respectively to generate the modeling dataset. Reliability and validity testing is performed using Cronbach's α coefficient, KMO test, and Bartlett's test of sphericity. The α coefficient is no less than 0.8, the KMO value is no less than 0.7, and the significance level of Bartlett's test of sphericity is p < 0.05. Outlier correction consists of outlier handling and missing value handling. Outlier handling adopts the 3σ criterion, and missing value handling adopts the mean imputation method based on the training set statistics. Normalization adopts the min-max standardization method, using only the minimum and maximum values ​​in the training set to determine the normalization parameters. In the removal of highly redundant features, the threshold for removing highly redundant indicators is the absolute value of the Pearson correlation coefficient ≥ 0.8.

[0075] It is understood that, in this embodiment of the invention, the overall Cronbach's α coefficient of the questionnaire is 0.894, indicating high internal consistency of the scale; at the same time, Bartlett's test of sphericity is significant (p<0.05), indicating that the data is suitable for factor analysis and can be used for subsequent statistical analysis and modeling.

[0076] It is understood that, in the embodiments of the present invention, for correctable missing data, mean filling or mean filling of similar samples can be used for processing; for samples that cannot be corrected or are obviously distorted, they are removed.

[0077] It is understood that in this embodiment of the invention, the 3σ criterion is used to identify extreme outliers in the index scores, and the outliers are selected for removal or correction based on their causes. To eliminate the influence of different index dimensions and value ranges on the training results of the BP neural network, the min-max standardization method is used to normalize the data of each index, mapping the scores of each index to the [0,1] interval. The normalization formula is as follows:

[0078] ;

[0079] In the formula, The original indicator score, This is the minimum value of the metric in the training set. This represents the maximum value of the metric in the training set. These are the normalized index values; both the test set and the data to be evaluated are normalized using the minimum and maximum values ​​determined in the training set, and the normalization parameters are not recalculated.

[0080] Understandably, feature optimization is performed on the candidate indicator features in the training set. First, the mutual information method is used to calculate the degree of correlation between each candidate indicator feature and the literacy level label. The larger the mutual information value, the higher the information contribution of the indicator to the literacy level classification, and the more suitable it is as an input feature for the BP neural network model.

[0081] Furthermore, the Pearson correlation coefficient between candidate indicator features is calculated using the training set data to identify highly redundant indicators. For any two candidate indicator features, if the absolute value of their Pearson correlation coefficient is not less than a preset threshold, it is determined that there is a high degree of redundancy between them; indicators with higher mutual information values ​​and stronger correlation with literacy level labels are preferentially retained, while indicators with high redundancy and low information contribution are eliminated. The preset threshold can be set to 0.8, or it can be adjusted according to the sample size and indicator distribution.

[0082] Furthermore, the optimal feature subset is obtained through mutual information ranking and relevance threshold constraints. The number of features in the optimal feature subset is denoted as K, where K is a positive integer not greater than 16, which can be determined by preset rules or cross-validation. When all evaluation indicators meet the requirements of low redundancy and high correlation, all 16 secondary indicators in Table 1 can be used as input features of the model; when some indicators have significant redundancy, the top K indicators in the comprehensive ranking can be selected as input features of the BP neural network model to improve model training efficiency and generalization stability.

[0083] Furthermore, the data processing parameters and optimal feature subset determined based on the training set are applied to both the training and test sets to generate the final modeling dataset. The training set is used for model training and parameter optimization, while the test set is not involved in determining the data processing parameters, optimal feature subset, or model hyperparameters; it is only used for final model performance validation. To reduce the impact of uneven sample distribution across different literacy levels on the model training and testing results, stratified sampling is used when dividing the training and test sets to ensure that the proportions of Level A, B, C, D, and E samples in both sets remain consistent with or substantially consistent with the original sample data.

[0084] Step 4: Construct a three-layer BP neural network model, determine the optimal number of hidden layer nodes and learning rate, train the model using the training set, and evaluate the model using the test set to determine the optimal BP neural network evaluation model.

[0085] In this embodiment of the invention, after constructing a three-layer BP neural network model, the optimal number of hidden layer nodes and learning rate are determined through 5-fold hierarchical cross-validation and grid search. The model is then trained using the model dataset. Subsequently, the optimal parameter combination is used to train a candidate evaluation model on the complete training set. The model is evaluated on the test set according to accuracy, precision, recall, F1 score, MSE, and Kappa coefficient. The model evaluation is then determined to be satisfactory. If the model passes the evaluation, the optimal BP neural network evaluation model is obtained. Specifically, as shown... Figure 3 The diagram shows a three-layer structure of the BP neural network model in the method provided by the embodiment of the present invention. In the three-layer BP neural network model, the number of neurons in the input layer is equal to K, the hidden layer uses the Sigmoid activation function, and the output layer uses the linear activation function. The candidate value for the optimal number of hidden layer nodes is 10 to 100. The candidate value for the learning rate is 0.01 to 0.1. The maximum number of training iterations for the model is 1000, and the error convergence threshold is 0.001.

[0086] It is understandable that a BP neural network includes an input layer, a hidden layer, and an output layer. The number of neurons in the input layer is equal to the number of features K in the optimal feature subset, and the input data is normalized artificial intelligence literacy index data. The hidden layer adopts a single hidden layer structure, and the activation function is preferably the Sigmoid function. The output layer adopts a linear activation function, outputs the comprehensive score of college students' artificial intelligence literacy, and maps it to the corresponding literacy level according to the preset level range.

[0087] Furthermore, the training process of the BP neural network model includes forward propagation and backpropagation. During forward propagation, the input layer passes the normalized index data to the hidden layer. The hidden layer, after processing by the activation function, passes the result to the output layer, which generates the predicted composite score. During backpropagation, the error between the predicted composite score and the true composite score is used as the loss. Based on the gradient descent principle, the connection weights and thresholds between the input layer and the hidden layer, and between the hidden layer and the output layer, are updated in reverse, gradually reducing the model's prediction error.

[0088] Furthermore, let the model input vector be X=(x1,x2,…,x…). K ), where K is the number of input features; H is the number of hidden layer nodes; and y' is the model output. The BP neural network establishes a non-linear mapping relationship between input features and the overall score through a weight matrix, threshold, and activation function. The model training objective is to minimize the error between the predicted output and the actual overall score. The error function can be the mean squared error (MSE), expressed as:

[0089]

[0090] In the formula, n represents the sample size. This represents the actual composite score of the i-th sample. This represents the overall prediction score for the i-th sample.

[0091] Furthermore, before model training, the connection weights and thresholds between the input layer and hidden layers, and between the hidden layers and the output layer, are initialized. The initial weight values ​​can be set to random numbers within the range [-1, 1], and the thresholds can be set to 0 or randomly initialized. Model training parameters include the learning rate, maximum number of iterations, error convergence threshold, and number of hidden layer nodes. The candidate values ​​for the learning rate can be set from 0.01 to 0.1, the candidate values ​​for the number of hidden layer nodes can be set from 10 to 100, the maximum number of iterations can be set to 1000, and the error convergence threshold can be set to 0.001.

[0092] Furthermore, such as Figure 4This diagram illustrates the model training and parameter optimization process in the method provided by this invention. A 5-fold hierarchical cross-validation combined with a grid search method is used to optimize the parameters of the BP neural network model. Specifically, the training set is divided into five subsets according to the proportion of skill levels. One subset is selected as the validation set at each step, and the remaining four subsets are used as the training subsets. Candidate combinations of hidden layer nodes and learning rates are iterated sequentially, and the average prediction error and skill level prediction performance of each parameter combination under cross-validation are calculated. The parameter combination with the smallest average cross-validation error and the best skill level prediction performance is selected as the optimal parameter combination. It should be noted that the test set does not participate in model parameter optimization; it is only used for final verification of the model's generalization ability after training.

[0093] Understandably, the test set is input into the trained BP neural network evaluation model to obtain a comprehensive prediction score for each test sample, and a corresponding prediction literacy level is generated based on a preset score range. The prediction literacy level is then compared with the true literacy level to evaluate the model's predictive performance and generalization ability.

[0094] Further, accuracy, precision, recall, F1 score, mean squared error, confusion matrix, and Kappa coefficient are used to comprehensively evaluate model performance. Accuracy reflects the proportion of the model's overall correct grade predictions; precision reflects the proportion of samples predicted as a certain grade by the model that actually belong to that grade; recall reflects the proportion of samples of a certain true grade that are correctly identified by the model; F1 score is used to comprehensively evaluate precision and recall; MSE is used to evaluate the deviation between the predicted overall score and the actual overall score; and the Kappa coefficient is used to measure the consistency between the predicted grade and the true grade, which is particularly suitable for multi-class classification scenarios.

[0095] Furthermore, to verify the effectiveness of the method of this invention, a comparative experiment can be conducted with machine learning models such as random forest, support vector machine, XGBoost, and KNN. All models use the same training set, test set, and evaluation metrics, and are trained and tested under the same feature input conditions. By comparing the accuracy, F1 score, MSE, and Kappa coefficient of different models, the predictive performance and stability of the BP neural network evaluation model in the college student artificial intelligence literacy assessment task are determined. The classification results of different models on the test set are compared in the table below:

[0096] Furthermore, such as Figure 5The diagram shows a bar chart comparing the model performance evaluation results with the comparative model metrics in the method provided by this invention. Bar charts or line charts can be used to display the evaluation results of different models on the test set. When the BP neural network model outperforms the comparative model overall in terms of accuracy, F1 score, MSE, and Kappa coefficient, it can effectively predict and determine the comprehensive score of college students' artificial intelligence literacy, demonstrating good prediction accuracy and generalization ability.

[0097] Step 5: After the model evaluation is passed, the data to be evaluated is input into the optimal BP neural network evaluation model, the predicted comprehensive score is output and mapped to the literacy level, an intelligent evaluation report is generated, and the optimal BP neural network evaluation model is updated regularly.

[0098] In this embodiment of the invention, after the model evaluation is passed, the data to be evaluated is input into the optimal BP neural network evaluation model; then, the comprehensive score, competence level and weak dimension are output, and an intelligent evaluation report is generated based on the comprehensive score, competence level and weak dimension; and new sample data is collected periodically. When the input feature structure of the new sample remains unchanged, the model is incrementally trained and the parameters are updated; when the evaluation index system is adjusted, resulting in a change in the input feature structure, the model is retrained and evaluated.

[0099] It is understandable that, such as Figure 6 This diagram illustrates the framework for intelligent evaluation report output in the method provided by this invention. A BP neural network evaluation model is used to output a predicted comprehensive score, with each dimension's score calculated based on the secondary indicator data under the corresponding primary indicator. After inputting the indicator data of the university student to be evaluated into the trained BP neural network model, the system outputs a comprehensive score for artificial intelligence literacy and maps it to a literacy level. Furthermore, it calculates the dimensional scores for four dimensions: theoretical cognition, application skills, ethical literacy, and innovation ability. For dimensions whose scores are significantly lower than a set threshold or lower than the individual's overall level, the system identifies them as weak dimensions. Preferably, when a dimension's score is below 60 points, or more than 10 points lower than the individual's average score across the four dimensions, that dimension is identified as a weak dimension, and a corresponding diagnostic result is generated.

[0100] Furthermore, the intelligent evaluation report includes a comprehensive evaluation result, competency level, scores for each dimension, diagnosis of weak dimensions, and personalized improvement suggestions. Personalized improvement suggestions are generated based on the type of weak dimension: when theoretical cognition is weak, it is recommended to strengthen the learning of basic AI concepts, machine learning principles, and data algorithm fundamentals; when applied skills are weak, it is recommended to increase AI tool operation, data processing, and model deployment training; when ethical literacy is weak, it is recommended to strengthen privacy protection, algorithmic bias identification, and adherence to accountability standards; when innovation capability is weak, it is recommended to strengthen training in problem transformation, model optimization, interdisciplinary innovation, and critical reflection.

[0101] Furthermore, this embodiment can periodically collect new sample data to perform incremental training and parameter fine-tuning on the already trained BP neural network model. After quality control, the new sample data is processed using the missing value handling parameters, normalization parameters, and optimal feature subset determined during the existing model training phase, and then input into the original BP neural network model for incremental training and parameter fine-tuning. When the evaluation index system is adjusted, causing changes in the model's input feature structure, feature selection, model training, and model evaluation are performed again.

[0102] At the same time, the evaluation index system can be dynamically adjusted according to the development of artificial intelligence technology and the talent training needs of universities. For example, indicators such as generative artificial intelligence applications and intelligent agent collaborative applications can be added to maintain the timeliness and applicability of the evaluation method.

[0103] It should be noted that those skilled in the art can make various modifications and improvements without departing from the inventive concept, and these all fall within the scope of protection of this invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A method for intelligent evaluation of college students' artificial intelligence literacy based on a BP neural network evaluation algorithm, characterized in that, The method specifically includes the following steps: Construct an evaluation index system for college students' artificial intelligence literacy and design a structured questionnaire; University students were selected as the research subjects. Valid questionnaire data were collected through the structured questionnaire, and the comprehensive scores were converted into percentages to classify the literacy levels and construct the initial sample data. The initial sample data were subjected to questionnaire quality control and reliability and validity testing. Stratified sampling was carried out according to literacy levels and the training set and test set were divided. Based on the training set, missing value handling parameters, normalization parameters and optimal feature subsets were determined and applied to the test set to generate the modeling dataset. A three-layer BP neural network model is constructed, the optimal number of hidden layer nodes and learning rate are determined, the model is trained using the training set, and the model is evaluated using the test set to determine the optimal BP neural network evaluation model. After the model evaluation is passed, the data to be evaluated is input into the optimal BP neural network evaluation model, the predicted comprehensive score is output and mapped to the literacy level, an intelligent evaluation report is generated, and the optimal BP neural network evaluation model is updated regularly.

2. The intelligent evaluation method for college students' artificial intelligence literacy based on the BP neural network evaluation algorithm according to claim 1, characterized in that, The evaluation index system for college students' artificial intelligence literacy includes four primary indicators: theoretical cognition, application skills, ethical literacy, and innovation ability. These four primary indicators consist of sixteen secondary indicators. Theoretical cognition includes basic concept cognition, machine learning principles, deep learning fundamentals, and data algorithm fundamentals. Application skills include tool operation skills, data processing capabilities, model deployment and application across scenarios. Ethical literacy includes privacy awareness, awareness of algorithmic bias, adherence to responsibility norms, and ethical decision-making ability. Innovation ability includes problem transformation ability, model optimization thinking, interdisciplinary innovation, and critical reflection ability.

3. The intelligent evaluation method for college students' artificial intelligence literacy based on the BP neural network evaluation algorithm according to claim 2, characterized in that, The process of selecting university students as the research subjects, collecting valid questionnaire data through the structured questionnaire, calculating a comprehensive score on a percentage basis, classifying literacy levels, and constructing initial sample data specifically includes the following steps: The survey subjects were college students. The structured questionnaire was used to collect survey data from multiple respondents. Invalid questionnaires and abnormal samples were removed from the questionnaire survey data to obtain valid questionnaire data; Based on the sixteen secondary indicators, the valid questionnaire data were analyzed, and the composite score of multiple survey participants was calculated on a percentage basis. Based on the comprehensive score of multiple percentage points, the corresponding literacy levels are divided; Construct initial sample data.

4. The intelligent evaluation method for college students' artificial intelligence literacy based on the BP neural network evaluation algorithm according to claim 3, characterized in that, The questionnaire survey data consists of basic information and indicator data of multiple survey subjects. The basic information includes the survey subjects' gender, education level, grade, school type, major, and use of smart tools. The classification criteria for the literacy level are as follows: [90,100] is level A, [80,90) is level B, [70,80) is level C, [60,70) is level D, and <60 is level E.

5. The intelligent evaluation method for college students' artificial intelligence literacy based on the BP neural network evaluation algorithm according to claim 2, characterized in that, The process of performing questionnaire quality control and reliability and validity testing on the initial sample data, stratified sampling according to literacy levels to divide the data into training and test sets, determining missing value handling parameters, normalization parameters, and optimal feature subsets based on the training set, and applying these parameters to the test set to generate the modeling dataset specifically includes the following steps: The initial sample data were subjected to questionnaire quality control and reliability and validity testing. Stratified sampling was performed according to literacy levels to divide the initial sample data into training and test sets; Based on the training set, determine the missing value handling parameters, normalization parameters, and optimal feature subset; Using the sixteen secondary indicators in the training set as candidate features, the mutual information method is used to calculate the correlation between each candidate feature and the literacy level, and the Pearson correlation coefficient is used to remove highly redundant features, and the optimal feature subset consisting of K indicators is selected, where K is a positive integer not greater than 16. The missing value handling parameters, the normalization parameters, and the optimal feature subset are used to process the training set and the test set respectively to generate a modeling dataset.

6. The intelligent evaluation method for college students' artificial intelligence literacy based on the BP neural network evaluation algorithm according to claim 5, characterized in that, The reliability and validity tests employ Cronbach's α coefficient, KMO test, and Bartlett's test of sphericity, wherein the Cronbach's α coefficient is not less than 0.8, the KMO value is not less than 0.7, and the significance level of the Bartlett's test of sphericity is p < 0.05; the outlier correction consists of outlier handling and missing value handling, wherein outlier handling adopts the 3σ criterion, and missing value handling adopts the mean imputation method based on the training set statistics; the normalization process adopts the min-max standardization method, using only the minimum and maximum values ​​in the training set to determine the normalization parameters, and applying the normalization parameters to the training set, test set, and data to be evaluated; in the removal of highly redundant features, the threshold for removing highly redundant indicators is the absolute value of the Pearson correlation coefficient ≥ 0.8; the ratio of the training set to the test set is 7:

3.

7. The intelligent evaluation method for college students' artificial intelligence literacy based on the BP neural network evaluation algorithm according to claim 5, characterized in that, The process of constructing a three-layer backpropagation (BP) neural network model, determining the optimal number of hidden layer nodes and learning rate, training the model using the training set, and evaluating the model using the test set to determine the optimal BP neural network evaluation model specifically includes the following steps: Construct a three-layer BP neural network model; The optimal number of hidden layer nodes and learning rate are determined by using 5-fold hierarchical cross-validation and grid search. The model is trained using the training set, and the optimal parameter combination is selected in 5-fold hierarchical cross-validation. The candidate evaluation model is then trained on the complete training set using the optimal parameter combination. The test set evaluation model is evaluated based on accuracy, precision, recall, F1 score, MSE, and Kappa coefficient, and it is determined whether the model evaluation passes. After the model evaluation is passed, the candidate evaluation model is determined as the optimal BP neural network evaluation model.

8. The intelligent evaluation method for college students' artificial intelligence literacy based on the BP neural network evaluation algorithm according to claim 5, characterized in that, In the three-layer BP neural network model, the number of neurons in the input layer is equal to K, the hidden layer uses the Sigmoid activation function, and the output layer uses the linear activation function; the candidate value for the optimal number of hidden layer nodes is 10 to 100; the candidate value for the learning rate is 0.01 to 0.1; the maximum number of training iterations for the model is 1000, and the error convergence threshold is 0.

001.

9. The intelligent evaluation method for college students' artificial intelligence literacy based on the BP neural network evaluation algorithm according to claim 1, characterized in that, After the model evaluation is passed, the data to be evaluated is input into the optimal BP neural network evaluation model, which outputs a predicted comprehensive score and maps it to the literacy level, generating an intelligent evaluation report. The optimal BP neural network evaluation model is updated regularly, specifically including the following steps: After the model evaluation is passed, the data to be evaluated is input into the optimal BP neural network evaluation model; The optimal BP neural network evaluation model outputs a comprehensive score, which is then mapped to a competency level, and weak dimensions are identified. Based on the overall score, the competency level, and the weak dimensions, an intelligent evaluation report is generated; Collect new sample data regularly; When the input feature structure of the newly added sample data is consistent with the existing model, the newly added sample data is processed using the existing data processing parameters and the optimal feature subset, and incremental training and parameter updates are performed on the optimal BP neural network evaluation model.

10. The intelligent evaluation method for college students' artificial intelligence literacy based on the BP neural network evaluation algorithm according to claim 1, characterized in that, The method further includes the following steps: The evaluation index system for college students' artificial intelligence literacy is adaptively updated and adjusted; when the adjustment causes changes in the input feature structure of the model, the BP neural network evaluation model is retrained and evaluated based on the updated sample data.

Images