Career data acquisition and analysis system and method based on big data

Abstract

The invention relates to the technical field of data analysis, in particular to a career data acquisition and analysis system and method based on big data, and the system comprises a matrix construction unit, a model training unit, a model construction unit and an exercise recommendation unit, and further comprises a data acquisition unit, the data acquisition unit is used for acquiring career data of an online learning website of the target user through a web crawler technology, the career data comprises knowledge points of exercises done by the target user, and the career data is transmitted to the probability prediction unit and the matrix construction unit. According to the method, targeted exercise recommendation is provided for the user by analyzing the career data of the user, establishing the cognitive level matrix of the user and identifying the strength of the user on the knowledge points, the exercises most suitable for the user can be recommended through the exercise recommendation model, the user is helped to concentrate on the knowledge points needing to be improved, and therefore the learning efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data analysis, and in particular to a career data collection and analysis system and method based on big data. Background Art

[0002] Career data includes the exercises that the target users have done and the knowledge points involved in the learning process. These data can be used to track students' learning progress, evaluate learning outcomes, identify learning difficulties, personalize learning paths, etc. By analyzing career data, educational institutions and educators can better understand students' learning needs and provide targeted support and guidance, thereby promoting students' learning and development.

[0003] Traditional systems are usually based on fixed course designs and cannot be personalized to suit the individual differences of each student. This may result in students being unable to obtain exercises and learning content tailored to their specific needs. Traditional systems usually rely on the experience of teachers or textbooks rather than data-based analysis, which results in the system being unable to adjust in a timely manner to meet students' changing needs. Due to the lack of personalization and intelligent recommendations in traditional systems, students may need to purchase additional materials or spend extra time on tutoring, increasing the cost of learning. Traditional systems usually rely on paper assignments, exams, and classroom activities, which collects a small amount of data and cannot fully reflect students' learning behaviors, knowledge mastery, and learning habits. Summary of the invention

[0004] The purpose of the present invention is to address the problems existing in the background technology and to propose a career data collection and analysis system and method based on big data.

[0005] The technical solution of the present invention: a career data collection and analysis system based on big data, including a matrix construction unit, a model training unit, a model construction unit and an exercise recommendation unit, and also includes:

[0006] A data collection unit, the data collection unit is used to obtain the career data of the target user's online learning website through a web crawler technology, the career data including the knowledge points of the exercises the target user has done, and transmit the career data to the probability prediction unit and the matrix construction unit;

[0007] A probability prediction unit, wherein the probability prediction unit receives the career data transmitted by the data acquisition unit, constructs an exercise prediction model based on the career data, and adds an unexpected coefficient to the exercise prediction model to obtain a modified exercise prediction model, wherein the unexpected coefficient includes a carelessness coefficient and a guessing coefficient, and transmits the modified exercise prediction model to the model solving unit;

[0008] A model solving unit, wherein the model solving unit receives the modified exercise prediction model transmitted by the probability prediction unit, and constructs a conditional distribution and a marginal distribution of answer results of the exercises done by the target user based on the modified exercise prediction model, and uses the Newton-Raphson iteration method based on the conditional distribution and the marginal distribution to solve each unknown parameter in the modified exercise prediction model, so as to obtain a modified probability that the target user is correct on each exercise, and transmits the probability that the target user is correct on each exercise to the cognitive matrix construction unit.

[0009] Preferably, the matrix construction unit receives the probability of the target user being correct on each exercise transmitted by the model solving unit and the career data transmitted by the data collection unit, and constructs a cognitive level matrix of the target user based on the probability of the target user being correct on each exercise, and matches the cognitive level matrix with the career data to obtain a knowledge point matrix corresponding to the cognitive level matrix, and transmits the cognitive level matrix and the knowledge point matrix to the model training unit.

[0010] Preferably, the model building unit is used to build an exercise recommendation model based on a deep neural network, the exercise recommendation model includes an input layer, an embedding layer, a fusion layer, a hidden layer and a prediction layer, and transmit the exercise recommendation model to the model training unit.

[0011] Preferably, the model training unit receives the cognitive level matrix and the knowledge point matrix transmitted by the matrix construction unit and the exercise recommendation model transmitted by the model construction unit, and trains the exercise recommendation model through the cognitive level matrix and the knowledge point matrix to obtain a trained exercise recommendation model, and transmits the trained exercise recommendation model to the exercise recommendation unit.

[0012] Preferably, the exercise recommendation unit receives the trained exercise recommendation model transmitted by the model training unit, and inputs the cognitive level matrix and the knowledge point matrix of the target user who needs exercise recommendation into the trained exercise recommendation model, outputs the exercise recommendation probability through the trained exercise recommendation model, and compares the exercise recommendation probability with a preset threshold to obtain the recommended exercises.

[0013] Preferably, the step of obtaining the career data of the target user on the online learning website by using a web crawler technology comprises the following steps:

[0014] A1. Locate the URL of the online learning website, and filter target content from the online learning website based on the filter conditions provided by the target user;

[0015] A2. Obtain the response information of the online learning website server through the Python requests module. The server lists the target content in JSON format and returns it in the form of JavaScript.

[0016] A3. Use Python regular expression module to match the response information content, extract the response information content to form the required data list;

[0017] A4. Building an access link list based on the required data list, acquiring the target content based on the access link list, and building the target content in a Python dictionary format to obtain career data, and building a career data list with all the career data.

[0018] Preferably, the exercise prediction model is as follows:

[0019]

[0020] Among them, δ jk represents the estimated correct probability of the target user j on exercise k based on the knowledge point mastery, B kn Indicates the examination of exercise k on knowledge point n, C jn represents the mastery of knowledge point n by the target user j, and N represents the total number of knowledge points related to exercise k;

[0021] The modified exercise prediction model is as follows:

[0022] P jk =β k (1-γ k );

[0023] Among them, P jk represents the correct probability of the target user j on exercise k, β k represents the probability that the target user j makes a careless mistake on exercise k, γ k represents the probability that the target user j guesses correctly on question k.

[0024] Preferably, the conditional distribution of the answer results of the exercises that the target user has done is as follows:

[0025]

[0026] Among them, P jk represents the correct probability that the target user j is correct on exercise k, L1 represents the conditional distribution of the answer results of the exercises that the target user has done, and m represents the total number of exercises that the target user has done;

[0027] The marginal distribution of the answer results of the exercises done by the target user is as follows:

[0028]

[0029] Among them, P jk represents the corrected probability that the target user j is correct on exercise k, L2 represents the marginal distribution of the answer results of the exercises that the target user has done, J represents the total number of the target users, and m represents the total number of exercises that the target user has done.

[0030] Preferably, the method of solving each unknown parameter in the modified exercise prediction model based on the conditional distribution and the marginal distribution using the Newton-Raphson iteration method comprises the following steps:

[0031] B1. Taking the logarithm of the marginal distribution of the answer results of the exercises that the target user has done, the marginal distribution of the answer results of the exercises that the target user has done after taking the logarithm is as follows:

[0032]

[0033] B2. Based on the marginal distribution of the answer results of the target user who has done the exercises after taking the logarithm, the probability of the target user j making a careless mistake on exercise k and the probability of the target user j guessing correctly on exercise k are derived. The marginal distribution of the answer results of the target user who has done the exercises after taking the logarithm is as follows:

[0034]

[0035] B3. Calculate the marginal distribution of the answer results of the target user who has done the exercises after derivation and logarithmization by the Newton-Raphson iteration method, so as to obtain the estimated values ​​of the probability that the target user j makes a careless mistake on exercise k and the probability that the target user j guesses correctly on exercise k.

[0036] Preferably, the step of inputting the cognitive level matrix and the knowledge point matrix of the target user who needs exercise recommendation into the trained exercise recommendation model, and outputting the exercise recommendation probability through the trained exercise recommendation model comprises the following steps:

[0037] C1. Input the cognitive level matrix and the knowledge point matrix of the target user who needs exercise recommendation into the input layer, and the input layer inputs the cognitive level matrix and the knowledge point matrix into the embedding layer. The input layer performs one-hot encoding on the cognitive level matrix and the knowledge point matrix, and inputs the one-hot encoded cognitive level matrix and the knowledge point matrix into the embedding layer;

[0038] C2, the embedding layer uses a fully connected layer to embed the cognitive level matrix and the knowledge point matrix, thereby obtaining a cognitive level preference vector and a knowledge point preference vector, and the embedding layer embeds the cognitive level matrix and the knowledge point matrix after unique hot encoding, thereby obtaining a cognitive level latent vector and a knowledge point latent vector, and the cognitive level preference vector and the knowledge point preference vector are as follows:

[0039]

[0040] Among them, e u represents the cognitive level preference vector, s u Represents the cognitive level matrix, W u represents the weight matrix of the embedding layer with respect to the cognitive level matrix, d i represents the knowledge point preference vector, s i Represents the knowledge point matrix, W i Representing a weight matrix in the embedding layer with respect to the knowledge point matrix;

[0041] The cognitive level latent vector and the knowledge point latent vector are as follows:

[0042]

[0043] Among them, p u represents the cognitive level latent vector, P T represents the parameter matrix of the cognitive level matrix after one-hot encoding in the embedding layer, represents the cognitive level matrix after one-hot encoding, q i represents the potential vector of the knowledge point, Q T Represents the parameter matrix of the knowledge point matrix after one-hot encoding in the embedding layer, V i I represents the knowledge point matrix after one-hot encoding;

[0044] C3, the fusion layer performs feature fusion on the cognitive level preference vector and the knowledge point preference vector with the corresponding cognitive level latent vector and the knowledge point latent vector, thereby obtaining a target user preference vector, and the target user preference vector is as follows:

[0045]

[0046] Among them, f represents the target user preference vector, Indicates the multiplication of two vectors according to corresponding elements;

[0047] C4, the hidden layer is a tower neural network structure composed of multiple fully connected layers, through which multiple target user preference vectors are jointly encoded, and the nonlinear relationship between the multiple target user preference vectors is captured, and the ReLU activation function is used as the nonlinear factor of the hidden layer;

[0048] C5. The prediction layer maps the output of the hidden layer to the exercise recommendation probability, and the exercise recommendation probability expression is as follows:

[0049]

[0050] in, represents the probability of exercise recommendation, sigmoid() represents the activation function of the prediction layer, and x represents the specific value output by the hidden layer.

[0051] The technical solution of the present invention is a career data collection and analysis method based on big data, which is applicable to the career data collection and analysis system based on big data, and comprises the following steps:

[0052] S1. Collect career data from the target user's online learning website through web crawler technology. The career data includes exercises done by the target user and related knowledge points.

[0053] S2, analyzing the career data and constructing an exercise prediction model, adding a carelessness coefficient and a guessing coefficient to the exercise prediction model to correct the exercise prediction model;

[0054] S3, use the Newton-Raphson iteration method to solve the unknown parameters in the prediction model of the modified exercise;

[0055] S4, based on solving the unknown parameters in the prediction model of the revised exercises, the probability of the target user being correct on each exercise is obtained, thereby constructing the user's cognitive level matrix, and matching the cognitive level matrix with the career data to generate a knowledge point matrix corresponding to the cognitive level matrix;

[0056] S5. Use the cognitive level matrix and knowledge point matrix to train the exercise recommendation model;

[0057] S6. Input the target user's cognitive level matrix and knowledge point matrix into the exercise recommendation model, and provide the final recommended exercises based on the comparison between the exercise recommendation probability output by the exercise recommendation model and the preset threshold.

[0058] Compared with the prior art, the above technical solution of the present invention has the following beneficial technical effects:

[0059] 1. The present invention analyzes the user's career data, establishes the user's cognitive level matrix, identifies the user's strengths and weaknesses in knowledge points, and thus provides users with targeted exercise recommendations. The exercise recommendation model can recommend the most suitable exercises for users, helping users to focus on the knowledge points that need to be improved, thereby improving learning efficiency. In addition, by adopting big data technology and using methods such as web crawlers to collect and analyze users' online learning data, the system makes decisions based on real data, thereby improving the accuracy of exercise recommendations.

[0060] 2. The accuracy of exercise recommendations is continuously improved through exercise prediction models and deep neural networks to better meet users' learning needs. Through intelligent recommendations, users can reduce the time and energy spent on blindly doing exercises and focus on the most important knowledge points, thereby reducing the overall learning cost. Personalized exercise recommendations can enhance users' learning experience and increase the fun and effectiveness of learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 A schematic diagram of the overall system flow in an embodiment of the present invention;

[0062] Figure 2 A schematic diagram of a process for obtaining career data in an embodiment of the present invention;

[0063] Figure 3 A schematic diagram of a flow chart of solving various unknown parameters in a prediction model for a modified exercise in an embodiment of the present invention;

[0064] Figure 4 A schematic diagram of a flow chart of outputting exercise recommendation probabilities by an exercise recommendation model in an embodiment of the present invention;

[0065] Figure 5 The figure is a flow chart of an overall method in one embodiment of the present invention.

[0066] Figure numerals: 1. Data acquisition unit; 2. Probability prediction unit; 3. Model solving unit; 4. Matrix construction unit; 5. Model training unit; 6. Model construction unit; 7. Exercise recommendation unit. DETAILED DESCRIPTION

[0067] Embodiment 1, as Figure 1 As shown, the career data collection and analysis system based on big data proposed by the present invention includes a matrix construction unit 4, a model training unit 5, a model construction unit 6 and an exercise recommendation unit 7, and also includes:

[0068] Data collection unit 1, which is used to obtain career data of the target user's online learning website through web crawler technology, the career data including the knowledge points of the exercises done by the target user, and transmit the career data to probability prediction unit 2 and matrix construction unit 4;

[0069] The probability prediction unit 2 receives the career data transmitted by the data acquisition unit 1, and constructs an exercise prediction model based on the career data, and adds an unexpected coefficient to the exercise prediction model to obtain a modified exercise prediction model, wherein the unexpected coefficient includes a carelessness coefficient and a guessing coefficient, and transmits the modified exercise prediction model to the model solving unit 3;

[0070] Model solving unit 3, model solving unit 3 receives the modified exercise prediction model transmitted by probability prediction unit 2, and constructs the conditional distribution and marginal distribution of the answer results of the exercises done by the target user based on the modified exercise prediction model, and uses the Newton-Raphson iteration method based on the conditional distribution and marginal distribution to solve each unknown parameter in the modified exercise prediction model, so as to obtain the probability that the target user is correct on each exercise after correction, and transmits the probability that the target user is correct on each exercise to cognitive matrix construction unit 4.

[0071] In the present invention, the Newton-Raphson iteration method is a numerical method for solving equations and optimization problems. Its basic principle is to iteratively approximate the root or extreme value of the objective function through linear approximation. Specifically, it uses the function and its derivatives to calculate the iteration step size, and then approximate the solution. This method is widely used in numerical calculations to solve nonlinear equations and optimization problems.

[0072] In an optional embodiment, the matrix construction unit 4 receives the probability of the target user being correct on each exercise transmitted by the model solving unit 3 and the career data transmitted by the data collection unit 1, and constructs a cognitive level matrix of the target user based on the probability of the target user being correct on each exercise, and matches the cognitive level matrix with the career data to obtain a knowledge point matrix corresponding to the cognitive level matrix, and transmits the cognitive level matrix and the knowledge point matrix to the model training unit 5.

[0073] In an optional embodiment, the model building unit 6 is used to build an exercise recommendation model based on a deep neural network, the exercise recommendation model includes an input layer, an embedding layer, a fusion layer, a hidden layer and a prediction layer, and transmit the exercise recommendation model to the model training unit 5.

[0074] It should be noted that a deep neural network (DNN) is an artificial neural network whose main feature is that it has multiple hidden layers. Compared with traditional shallow neural networks, DNN can learn complex patterns and features in data through more complex structures and more computing power.

[0075] In an optional embodiment, the model training unit 5 receives the cognitive level matrix and the knowledge point matrix transmitted by the matrix construction unit 4 and the exercise recommendation model transmitted by the model construction unit 6, and trains the exercise recommendation model through the cognitive level matrix and the knowledge point matrix to obtain the trained exercise recommendation model, and transmits the trained exercise recommendation model to the exercise recommendation unit 7.

[0076] In an optional embodiment, the exercise recommendation unit 7 receives the trained exercise recommendation model transmitted by the model training unit 5, and inputs the cognitive level matrix and knowledge point matrix of the target user who needs exercise recommendation into the trained exercise recommendation model, outputs the exercise recommendation probability through the trained exercise recommendation model, and compares the exercise recommendation probability with a preset threshold to obtain the recommended exercises.

[0077] Embodiment 2, as Figure 2-4 As shown, the career data collection and analysis system based on big data proposed by the present invention, compared with the first embodiment, this embodiment also includes:

[0078] The career data of target users on online learning websites is obtained through web crawler technology, including the following steps:

[0079] A1. Locate the URL of the online learning website and filter the target content from the online learning website based on the filter criteria provided by the target user;

[0080] A2. Obtain the response information of the online learning website server through Python's requests module. The server lists the target content in JSON format and returns it in the form of javascript.

[0081] A3. Use Python regular expression module to match the response information content, extract the response information content to form the required data list;

[0082] A4. Build an access link list based on the required data list, obtain the target content based on the access link list, and build the target content in Python dictionary format to obtain career data, and build a career data list with all career data.

[0083] In this embodiment, web crawler technology refers to the use of automated scripts or programs to browse and extract information on websites. This process usually involves simulating the behavior of human users on the Internet, retrieving web page content, and extracting useful data according to predetermined rules. The application scope of web crawler technology is very wide, including search engine indexing, data collection, market research, competition analysis, price tracking, content monitoring, academic research, etc.

[0084] In an optional embodiment, the exercise prediction model is as follows:

[0085]

[0086] Among them, δ jk represents the estimated correct probability of target user j on exercise k based on the knowledge point mastered, B kn Indicates the examination of exercise k on knowledge point n, C jn represents the mastery of knowledge point n by target user j, and N represents the total number of knowledge points related to exercise k;

[0087] The revised exercise prediction model is as follows:

[0088] P jk =β k (1-γ k );

[0089] Among them, P jk represents the probability that the correct target user j is correct on exercise k, β k represents the probability that target user j makes a careless mistake on exercise k, γ k represents the probability that target user j guesses correctly on question k.

[0090] In an optional embodiment, the conditional distribution of the answer results of the exercises that the target user has done is as follows:

[0091]

[0092] Among them, P jk represents the probability that the target user j is correct on exercise k after correction, L1 represents the conditional distribution of the answer results of the exercises that the target user has done, and m represents the total number of exercises that the target user has done;

[0093] The marginal distribution of the answer results of the exercises that the target user has done is as follows:

[0094]

[0095] Among them, P jkrepresents the probability that the target user j is correct on exercise k after correction, L2 represents the marginal distribution of the answer results of the exercises that the target user has done, J represents the total number of target users, and m represents the total number of exercises that the target user has done.

[0096] In an optional embodiment, the Newton-Raphson iteration method is used based on the conditional distribution and the marginal distribution to solve the unknown parameters in the modified exercise prediction model, including the following steps:

[0097] B1. Take the logarithm of the marginal distribution of the answer results of the exercises that the target user has done. The marginal distribution of the answer results of the exercises that the target user has done after taking the logarithm is as follows:

[0098]

[0099] B2. Based on the marginal distribution of the answer results of the target user who has done the exercises after taking the logarithm, the probability of the target user j making a careless mistake on exercise k and the probability of the target user j guessing correctly on exercise k are derived. The marginal distribution of the answer results of the target user who has done the exercises after taking the logarithm after the derivation is as follows:

[0100]

[0101] B3. The marginal distribution of the answer results of the exercises that the target user has done is calculated by the Newton-Raphson iteration method after differentiation and logarithm taking, so as to obtain the estimated values ​​of the probability that the target user j makes a careless mistake on exercise k and the probability that the target user j guesses correctly on exercise k.

[0102] In an optional embodiment, the cognitive level matrix and knowledge point matrix of the target user who needs exercise recommendation are input into the trained exercise recommendation model, and the exercise recommendation probability is output through the trained exercise recommendation model, including the following steps:

[0103] C1. Input the cognitive level matrix and knowledge point matrix of the target user who needs exercise recommendation into the input layer. The input layer inputs the cognitive level matrix and knowledge point matrix into the embedding layer. The input layer performs one-hot encoding on the cognitive level matrix and knowledge point matrix, and inputs the one-hot encoded cognitive level matrix and knowledge point matrix into the embedding layer.

[0104] C2, the embedding layer uses the fully connected layer to embed the cognitive level matrix and the knowledge point matrix to obtain the cognitive level preference vector and the knowledge point preference vector, and the embedding layer embeds the cognitive level matrix and the knowledge point matrix after the unique hot encoding to obtain the cognitive level latent vector and the knowledge point latent vector. The cognitive level preference vector and the knowledge point preference vector are as follows:

[0105]

[0106] Among them, e u represents the cognitive level preference vector, s u represents the cognitive level matrix, W u represents the weight matrix of the cognitive level matrix in the embedding layer, d i represents the knowledge point preference vector, s i represents the knowledge point matrix, W i Represents the weight matrix of the knowledge point matrix in the embedding layer;

[0107] The cognitive level latent vector and knowledge point latent vector are as follows:

[0108]

[0109] Among them, p u represents the cognitive level latent vector, P T represents the parameter matrix of the one-hot-encoded cognitive level matrix in the embedding layer, represents the cognitive level matrix after one-hot encoding, q i represents the potential vector of the knowledge point, Q T Represents the parameter matrix of the knowledge point matrix after one-hot encoding in the embedding layer, V i I Represents the knowledge point matrix after one-hot encoding;

[0110] C3, the fusion layer performs feature fusion on the cognitive level preference vector and the knowledge point preference vector with the corresponding cognitive level latent vector and the knowledge point latent vector to obtain the target user preference vector. The target user preference vector is as follows:

[0111]

[0112] Among them, f represents the target user preference vector, Indicates the multiplication of two vectors according to corresponding elements;

[0113] C4, the hidden layer is a tower neural network structure composed of multiple fully connected layers. The hidden layer jointly encodes multiple target user preference vectors and captures the nonlinear relationship between multiple target user preference vectors. The ReLU activation function is used as the nonlinear factor of the hidden layer.

[0114] C5, the prediction layer maps the output of the hidden layer to the probability of exercise recommendation. The expression of exercise recommendation probability is as follows:

[0115]

[0116] in, represents the probability of exercise recommendation, sigmoid() represents the activation function of the prediction layer, and x represents the specific value of the hidden layer output.

[0117] It should be noted that the tower neural network structure is a deep learning network structure, which is usually used in application scenarios such as recommendation systems, sorting tasks, and click-through rate prediction. Its characteristic is that the input passes through a series of gradually shrinking fully connected layers (from wide to narrow) to form a tower-like structure, hence the name. The design concept of the tower structure is to gradually capture more complex nonlinear relationships between data in deep neural networks.

[0118] Embodiment three, as Figure 5 As shown, the career data collection and analysis method based on big data proposed by the present invention is applicable to the career data collection and analysis system based on big data, and includes the following steps:

[0119] S1. Collect career data from the target user's online learning website through web crawler technology. The career data includes exercises done by the target user and related knowledge points.

[0120] S2, analyzing the career data and constructing an exercise prediction model, adding a carelessness coefficient and a guessing coefficient to the exercise prediction model to correct the exercise prediction model;

[0121] S3, use the Newton-Raphson iteration method to solve the unknown parameters in the prediction model of the modified exercise;

[0122] S4, based on solving the unknown parameters in the prediction model of the revised exercises, the probability of the target user being correct on each exercise is obtained, thereby constructing the user's cognitive level matrix, and matching the cognitive level matrix with the career data to generate a knowledge point matrix corresponding to the cognitive level matrix;

[0123] S5. Use the cognitive level matrix and knowledge point matrix to train the exercise recommendation model;

[0124] S6. Input the target user's cognitive level matrix and knowledge point matrix into the exercise recommendation model, and provide the final recommended exercises based on the comparison between the exercise recommendation probability output by the exercise recommendation model and the preset threshold.

[0125] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited thereto, and various changes can be made within the knowledge scope of technicians in the relevant technical field without departing from the purpose of the present invention.

Claims

1. A career data collection and analysis system based on big data, comprising a matrix construction unit (4), a model training unit (5), a model construction unit (6) and an exercise recommendation unit (7), characterized in that: A data collection unit (1), the data collection unit (1) is used to obtain the career data of the target user on the online learning website through a web crawler technology, the career data including the knowledge points of the exercises done by the target user, and transmit the career data to the probability prediction unit (2) and the matrix construction unit (4); A probability prediction unit (2), wherein the probability prediction unit (2) receives the career data transmitted by the data acquisition unit (1), constructs an exercise prediction model based on the career data, and adds an unexpected coefficient to the exercise prediction model to obtain a modified exercise prediction model, wherein the unexpected coefficient includes a carelessness coefficient and a guessing coefficient, and transmits the modified exercise prediction model to the model solving unit (3); A model solving unit (3), wherein the model solving unit (3) receives the modified exercise prediction model transmitted by the probability prediction unit (2), and constructs a conditional distribution and a marginal distribution of answer results of the exercises completed by the target user based on the modified exercise prediction model, and solves each unknown parameter in the modified exercise prediction model using a Newton-Raphson iteration method based on the conditional distribution and the marginal distribution, thereby obtaining a modified probability that the target user is correct on each exercise, and transmits the probability that the target user is correct on each exercise to a cognitive matrix construction unit (4).

2. The career data collection and analysis system based on big data according to claim 1 is characterized in that: The matrix construction unit (4) receives the probability of the target user being correct on each exercise transmitted by the model solving unit (3) and the career data transmitted by the data collection unit (1), and constructs a cognitive level matrix of the target user based on the probability of the target user being correct on each exercise, matches the cognitive level matrix with the career data, thereby obtaining a knowledge point matrix corresponding to the cognitive level matrix, and transmits the cognitive level matrix and the knowledge point matrix to the model training unit (5); The model construction unit (6) is used to construct an exercise recommendation model based on a deep neural network, the exercise recommendation model comprising an input layer, an embedding layer, a fusion layer, a hidden layer and a prediction layer, and transmit the exercise recommendation model to the model training unit (5).

3. The career data collection and analysis system based on big data according to claim 2 is characterized in that: The model training unit (5) receives the cognitive level matrix and the knowledge point matrix transmitted by the matrix construction unit (4) and the exercise recommendation model transmitted by the model construction unit (6), and trains the exercise recommendation model through the cognitive level matrix and the knowledge point matrix to obtain a trained exercise recommendation model, and transmits the trained exercise recommendation model to the exercise recommendation unit (7); The exercise recommendation unit (7) receives the trained exercise recommendation model transmitted by the model training unit (5), and inputs the cognitive level matrix and the knowledge point matrix of the target user who needs exercise recommendation into the trained exercise recommendation model, outputs the exercise recommendation probability through the trained exercise recommendation model, and compares the exercise recommendation probability with a preset threshold to obtain the recommended exercises.

4. The career data collection and analysis system based on big data according to claim 1 is characterized in that: The method of obtaining the career data of the target user on the online learning website by using the web crawler technology includes the following steps: A1. Locate the URL of the online learning website, and filter target content from the online learning website based on the filter conditions provided by the target user; A2. Obtain the response information of the online learning website server through the Python requests module. The server lists the target content in JSON format and returns it in the form of JavaScript. A3. Use Python regular expression module to match the response information content, extract the response information content to form the required data list; A4. Building an access link list based on the required data list, acquiring the target content based on the access link list, and building the target content in a Python dictionary format to obtain career data, and building a career data list with all the career data.

5. The career data collection and analysis system based on big data according to claim 1 is characterized in that: The exercise prediction model is as follows: Among them, δ jk represents the estimated correct probability of the target user j on exercise k based on the knowledge point mastery, B kn Indicates the examination of exercise k on knowledge point n, C jn represents the mastery of knowledge point n by the target user j, and N represents the total number of knowledge points related to exercise k; The modified exercise prediction model is as follows: P jk =b k (1-c k ); Among them, P jk represents the correct probability of the target user j on exercise k, β k represents the probability that the target user j makes a careless mistake on exercise k, γ k represents the probability that the target user j guesses correctly on question k.

6. The career data collection and analysis system based on big data according to claim 5 is characterized in that: The conditional distribution of the answer results of the exercises that the target user has done is as follows: Among them, P jk represents the correct probability that the target user j is correct on exercise k, L1 represents the conditional distribution of the answer results of the exercises that the target user has done, and m represents the total number of exercises that the target user has done; The marginal distribution of the answer results of the exercises done by the target user is as follows: Among them, P jk represents the corrected probability that the target user j is correct on exercise k, L2 represents the marginal distribution of the answer results of the exercises that the target user has done, J represents the total number of the target users, and m represents the total number of exercises that the target user has done.

7. The career data collection and analysis system based on big data according to claim 6 is characterized in that: The method of using Newton-Raphson iteration based on the conditional distribution and marginal distribution to solve various unknown parameters in the modified exercise prediction model includes the following steps: B1. Taking the logarithm of the marginal distribution of the answer results of the exercises that the target user has done, the marginal distribution of the answer results of the exercises that the target user has done after taking the logarithm is as follows: B2. Based on the marginal distribution of the answer results of the target user who has done the exercises after taking the logarithm, the probability of the target user j making a careless mistake on exercise k and the probability of the target user j guessing correctly on exercise k are derived. The marginal distribution of the answer results of the target user who has done the exercises after taking the logarithm is as follows: B3. Calculate the marginal distribution of the answer results of the target user who has done the exercises after derivation and logarithmization by the Newton-Raphson iteration method, so as to obtain the estimated values ​​of the probability that the target user j makes a careless mistake on exercise k and the probability that the target user j guesses correctly on exercise k.

8. The career data collection and analysis system based on big data according to claim 3 is characterized in that: The step of inputting the cognitive level matrix and the knowledge point matrix of the target user who needs exercise recommendation into the trained exercise recommendation model, and outputting the exercise recommendation probability through the trained exercise recommendation model, comprises the following steps: C1. Input the cognitive level matrix and the knowledge point matrix of the target user who needs exercise recommendation into the input layer, and the input layer inputs the cognitive level matrix and the knowledge point matrix into the embedding layer. The input layer performs one-hot encoding on the cognitive level matrix and the knowledge point matrix, and inputs the one-hot encoded cognitive level matrix and the knowledge point matrix into the embedding layer; C2, the embedding layer uses a fully connected layer to embed the cognitive level matrix and the knowledge point matrix, thereby obtaining a cognitive level preference vector and a knowledge point preference vector, and the embedding layer embeds the cognitive level matrix and the knowledge point matrix after unique hot encoding, thereby obtaining a cognitive level latent vector and a knowledge point latent vector, and the cognitive level preference vector and the knowledge point preference vector are as follows: Among them, e u represents the cognitive level preference vector, s u Represents the cognitive level matrix, W u represents the weight matrix of the embedding layer with respect to the cognitive level matrix, d i represents the knowledge point preference vector, s i Represents the knowledge point matrix, W i Representing a weight matrix in the embedding layer with respect to the knowledge point matrix; The cognitive level latent vector and the knowledge point latent vector are as follows: Among them, p u represents the cognitive level latent vector, P T represents the parameter matrix of the cognitive level matrix after one-hot encoding in the embedding layer, represents the cognitive level matrix after one-hot encoding, q i represents the potential vector of the knowledge point, Q T represents the parameter matrix of the knowledge point matrix after one-hot encoding in the embedding layer, V i I represents the knowledge point matrix after one-hot encoding; C3, the fusion layer performs feature fusion on the cognitive level preference vector and the knowledge point preference vector with the corresponding cognitive level latent vector and the knowledge point latent vector, thereby obtaining a target user preference vector, and the target user preference vector is as follows: Among them, f represents the target user preference vector, Indicates the multiplication of two vectors according to corresponding elements; C4, the hidden layer is a tower neural network structure composed of multiple fully connected layers, through which multiple target user preference vectors are jointly encoded, and the nonlinear relationship between the multiple target user preference vectors is captured, and the ReLU activation function is used as the nonlinear factor of the hidden layer; C5. The prediction layer maps the output of the hidden layer to the exercise recommendation probability, and the exercise recommendation probability expression is as follows: in, represents the probability of exercise recommendation, sigmoid() represents the activation function of the prediction layer, and x represents the specific value output by the hidden layer.

9. A career data collection and analysis method based on big data, which is applicable to a career data collection and analysis system based on big data as claimed in any one of claims 1 to 8, characterized in that: The following steps are involved: S1. Collect career data from the target user's online learning website through web crawler technology. The career data includes exercises done by the target user and related knowledge points. S2, analyzing the career data and constructing an exercise prediction model, adding a carelessness coefficient and a guessing coefficient to the exercise prediction model to correct the exercise prediction model; S3, use the Newton-Raphson iteration method to solve the unknown parameters in the prediction model of the modified exercise; S4, based on solving the unknown parameters in the prediction model of the revised exercises, the probability of the target user being correct on each exercise is obtained, thereby constructing the user's cognitive level matrix, and matching the cognitive level matrix with the career data to generate a knowledge point matrix corresponding to the cognitive level matrix; S5. Use the cognitive level matrix and knowledge point matrix to train the exercise recommendation model; S6. Input the target user's cognitive level matrix and knowledge point matrix into the exercise recommendation model, and provide the final recommended exercises based on the comparison between the exercise recommendation probability output by the exercise recommendation model and the preset threshold.

Images