A method for visualizing learning process sequence data based on convolutional neural networks
By converting the action sequences of students during the learning process into images and using convolutional neural networks for feature extraction and classification to generate a feature matrix, the problem of existing technologies being unable to effectively process complex action sequence data is solved, thereby improving the teaching effectiveness and personalized support of online education.
Patent Information
- Application Number
- CN202310843139.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-11
- Publication Date
- 2025-12-02
- Estimated Expiration
- 2043-07-11
AI Technical Summary
Existing temporal feature prediction and classification algorithms cannot effectively handle complex action sequence data in the student learning process, making it difficult to provide personalized and precise teaching support.
The sequence of actions during the student's learning process is converted into images, and features are extracted and classified using a convolutional neural network to generate a feature matrix. Finally, it is converted into single-channel or three-channel images to reflect the learning situation.
It improves the teaching effectiveness of online education, provides personalized learning support, enhances teachers' feedback on students' learning progress, preserves the temporal information of sequence data, and intuitively presents patterns and trends.
Smart Images
Figure CN117079178B_ABST
Abstract
Description
Technical Field
[0001] This invention pertains to data visualization methods based on convolutional neural networks, specifically relating to a method for visualizing sequence data of the learning process based on convolutional neural networks. Background Technology
[0002] With the popularization of online education, how to better utilize technologies such as big data and artificial intelligence to provide personalized and precise teaching support for online education has become one of the hot research topics in the industry; how to monitor students' learning status has become one of the problems that educational institutions and edtech companies need to solve. Among these, analyzing the sequence of students' actions during the learning process can reveal their behavioral patterns, thereby providing targeted feedback and guidance. Traditional analysis methods are mostly based on manually formulated rules or expert experience, which are difficult to handle large-scale and complex student behavior data. Therefore, machine learning algorithms are needed to automate the analysis of student behavior.
[0003] However, existing temporal feature prediction and classification algorithms cannot effectively solve this problem and cannot handle complex temporal data, such as data where the intervals between actions do not follow a clear pattern. Against this backdrop, research on image processing methods for the learning process based on convolutional neural networks has emerged, providing a new solution for online education. The innovation of this method lies in converting the sequence of actions during the student's learning process into images, and then using convolutional neural networks to extract and classify features from these images, ultimately generating a feature matrix that reflects the student's learning progress. Summary of the Invention
[0004] To address the aforementioned problems in existing technologies, this invention provides a method for visualizing learning process sequence data based on convolutional neural networks. This method collects action sequence data from student learning videos, cleans it, and then uses a convolutional neural network to convert the action sequences into feature matrices for individual videos. The weights of feature blocks are calculated based on video length and the importance of knowledge points, and these weights are then concatenated to form a complete feature matrix. Finally, the feature matrix is converted into single-channel or three-channel images for output, depending on the requirements.
[0005] The specific technical solution is as follows:
[0006] A method for visualizing learning process sequence data based on convolutional neural networks, characterized by the following steps:
[0007] 1) Collect data on a series of action sequences generated by students while watching instructional videos;
[0008] 2) The collected action sequence data generated by students while watching the instructional videos is cleaned and processed to obtain a data list containing the timestamp of each action, i.e., [(t1,a1),(t2,a2),...,(t...]. n ,a n )], where t i Let a represent the timestamp of the i-th action. i This indicates the type of the i-th action; the purpose of data cleaning is to remove useless and abnormal data. During the data cleaning process, data preprocessing and correction are required according to the actual situation to ensure the accuracy and reliability of the data.
[0009] 3) The collected action sequence data generated by students while watching instructional videos is used to generate corresponding feature matrices according to certain rules to represent the students' learning status while watching the videos. Specifically, key actions can be extracted from the videos, and features can be extracted from each key action to generate corresponding feature matrices. In this way, the video learning data is converted into an image-based feature matrix, providing input data for subsequent convolutional neural networks.
[0010] 4) Concatenate the above components according to certain rules to generate all corresponding feature matrices;
[0011] 5) Convert the above feature matrix into single-channel or three-channel image output according to the requirements to express the learning situation of the entire course; during the image output process, set different colors and sizes to reflect different learning states and data change trends. In this way, the image classification network can be used to understand the students' learning situation and provide a more scientific and effective reference for teaching.
[0012] Furthermore, in step 3), the series of action sequences generated by students while watching the instructional videos is collected and processed according to certain rules to generate a corresponding feature matrix, including the following steps:
[0013] (1) Calculate the size of the feature matrix:
[0014] First, the size of the feature matrix needs to be calculated. The size of the feature matrix can be calculated based on the number of course videos and the size of the output image. The number of course videos refers to the fact that there are multiple videos for each course online. The final generated image is obtained by concatenating the feature matrices of all the videos. The size of the feature matrix of each video can be calculated based on the number of course videos and the size of the output image.
[0015] The size of the output image is defined from the beginning, either 128x128, 256x256, or something else.
[0016] (2) Input the student's action sequence and count the number of all actions that appear in the video;
[0017] The system requires inputting the student's action sequence and counting the total number of actions performed by the student while watching the video; the student's action sequence is the action recorded by the system backend.
[0018] (3) Calculate the number of feature blocks required:
[0019] Based on the number of all actions appearing in the video and the size of the feature matrix, the number of feature matrices to be divided is calculated. Based on the number of feature matrices, the student's action sequence is divided into multiple subsequences, and each subsequence is processed.
[0020] (4) Zero-filling of the action sequence:
[0021] To ensure that the length of each action sequence is sufficient to fill the feature matrix, each subsequence needs to be zero-padded. Zero-padded means adding zero elements to the end of the action sequence so that the length of all action sequences meets the requirements. The specific padding length can be calculated based on the size and number of feature matrices.
[0022] (5) Convert the action sequence into a two-dimensional feature matrix:
[0023] Next, the action sequence is converted into multiple two-dimensional feature matrices. According to the size of the feature matrix, the corresponding number of data are taken out from the action sequence and stored into the feature matrix in turn. The storage rule is row-major. The length of the action sequence is the product of the size and number of feature matrices. In this way, the entire action sequence can be converted into two-dimensional feature matrices.
[0024] (6) Convolve all feature matrices:
[0025] Finally, all the two-dimensional feature matrices are input into a convolutional neural network for processing. The convolutional neural network can stack and merge all the feature matrices to obtain the feature matrix of a single video for a student's learning. The convolutional neural network is usually composed of multiple convolutional layers, pooling layers and fully connected layers, which can stack and process each feature block to obtain a single feature matrix.
[0026] Furthermore, step 4) involves concatenating the feature matrices of all the videos for this course, including the following steps:
[0027] (1) For each feature matrix, its corresponding weight needs to be calculated based on the length of the video and the importance of the knowledge point; the video length refers to the number of feature matrices contained in the video, and the importance of the knowledge point is evaluated by experts based on the difficulty and importance of the course.
[0028] (2) The weighted average method is adopted, that is, the weight of each feature matrix is multiplied by the weight of the video in which it is located, and then the weighted sum of all feature matrices is divided by the total weight to obtain the weighted average.
[0029] (3) Concatenate all feature matrices in the order they appear in the video. First, multiply all feature matrices by their corresponding weight values. Then, calculate the number of feature matrices needed for each row. Concatenate all feature matrices in sequence. Finally, determine whether the image size is fully utilized. If not, pad with zeros to the specified image size.
[0030] Furthermore, in step 5), the above feature matrix is converted into a three-channel image output according to the requirements, including the following conversion methods: using the matplotlib library for conversion. The matplotlib library provides the plt.imshow function to convert a single-channel grayscale image into a color RGB image. This function will automatically map the grayscale values to the corresponding RGB values according to the color mapping table cmap, thereby obtaining a color image.
[0031] Furthermore, in step 5), the above feature matrix is converted into a three-channel image output according to the requirements, including the following conversion methods: a custom method is used for channel conversion, which adopts a simple linear mapping method, that is, a piecewise function is set to calculate the corresponding RGB value based on the gray value, the gray value range is divided into several intervals, a corresponding RGB value is set for each interval, and then the gray value is mapped to the corresponding interval according to the gray value, and the corresponding RGB value is calculated to obtain a color image.
[0032] The present invention also provides an electronic device, characterized in that it includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor runs the computer program, it performs the steps of a method for image representation of learning process sequence data based on a convolutional neural network as described above.
[0033] The present invention also provides a computer-readable storage medium storing a computer program, characterized in that the computer program causes a computer to execute the steps of the above-described method for visualizing learning process sequence data based on a convolutional neural network.
[0034] Beneficial technical effects of the present invention:
[0035] 1. The present invention provides a method for visualizing learning process sequence data based on convolutional neural networks. This method can improve students' learning efficiency, effectively enhance the teaching effect of online education, and provide personalized learning support.
[0036] 2. The method for visualizing learning process sequence data based on convolutional neural networks described in this invention can provide teachers with more accurate feedback on students' learning progress, so as to better formulate teaching plans and improve teaching methods.
[0037] 3. The method for visualizing learning process sequence data based on convolutional neural networks described in this invention, compared to traditional sequence data processing methods, not only improves the accuracy of training and prediction but also better preserves the temporal information of the sequence data, presenting the patterns and trends of the sequence data more intuitively. Therefore, this invention can be applied to fields such as sequence data analysis, student learning evaluation, and time series analysis, and has broad application prospects. Attached Figure Description
[0038] Figure 1 This is a flowchart of a method for visualizing learning process sequence data based on a convolutional neural network, as described in this invention.
[0039] Figure 2 A flowchart for generating a single video feature matrix provided by the present invention;
[0040] Figure 3 This is a flowchart illustrating the stitching of feature matrices for all videos in a single course, as provided by the present invention. Detailed Implementation
[0041] To make the objectives, technical solutions, and advantages of this invention clearer, the following description is provided in conjunction with the appendix. Figure 1-3 The present invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
[0042] from Figure 1 As can be seen, the present invention provides a method for visualizing learning process sequence data based on convolutional neural networks, characterized by the following steps:
[0043] 1) Collect data on a series of action sequences generated by students while watching instructional videos; these actions include starting to watch, playing, pausing, and finishing.
[0044] 2) The collected action sequence data generated by students while watching the instructional videos is cleaned and processed to obtain a data list containing the timestamp of each action, i.e., [(t1,a1),(t2,a2),...,(t...]. n ,a n )], where t i Let a represent the timestamp of the i-th action. iThis indicates the type of the i-th action; the purpose of data cleaning is to remove useless and abnormal data. During the data cleaning process, data preprocessing and correction are required according to the actual situation to ensure the accuracy and reliability of the data.
[0045] 3) The collected action sequence data generated by students while watching instructional videos is used to generate corresponding feature matrices according to certain rules to represent the students' learning status while watching the videos. Specifically, key actions can be extracted from the videos, and features can be extracted from each key action to generate corresponding feature matrices. In this way, the video learning data is converted into an image-based feature matrix, providing input data for subsequent convolutional neural networks.
[0046] 4) Concatenate the above components according to certain rules to generate all corresponding feature matrices;
[0047] 5) Convert the above feature matrix into single-channel or three-channel image output according to the requirements to express the learning situation of the entire course; during the image output process, set different colors and sizes to reflect different learning states and data change trends. In this way, the image classification network can be used to understand the students' learning situation and provide a more scientific and effective reference for teaching.
[0048] Furthermore, in step 3), the series of action sequences generated by students while watching the instructional videos is collected and processed according to certain rules to generate a corresponding feature matrix, including the following steps:
[0049] (1) Calculate the size of the feature matrix:
[0050] First, the size of the feature matrix needs to be calculated. The size of the feature matrix can be calculated based on the number of course videos and the size of the output image. The number of course videos refers to the fact that there are multiple videos for each course online. The final generated image is obtained by concatenating the feature matrices of all the videos. The size of the feature matrix of each video can be calculated based on the number of course videos and the size of the output image.
[0051] The size of the output image is defined from the beginning, such as 128x128, 256x256, or other sizes. Here, we only know the size of the output image, not its specific content.
[0052] (2) Input the student's action sequence and count the number of all actions that appear in the video;
[0053] The system requires inputting the student's action sequence and counting the total number of actions performed by the student while watching the video; the student's action sequence is the action recorded by the system backend.
[0054] (3) Calculate the number of feature blocks required:
[0055] Based on the number of all actions appearing in the video and the size of the feature matrix, the number of feature matrices to be divided is calculated. Based on the number of feature matrices, the student's action sequence is divided into multiple subsequences, and each subsequence is processed.
[0056] (4) Zero-filling of the action sequence:
[0057] To ensure that the length of each action sequence is sufficient to fill the feature matrix, each subsequence needs to be zero-padded. Zero-padded means adding zero elements to the end of the action sequence to ensure that the length of all action sequences meets the requirements. The specific padding length can be calculated based on the size and number of feature matrices. Specifically, it refers to whether the length of the action sequence in a single video meets the requirements. For example, if the size of the feature matrix for each video is calculated to be (3x3), and the action sequence length is 16, then two zeros need to be padded to obtain two 3x3 matrices. These two matrices can then be convolved to obtain a single 3x3 matrix.
[0058] (5) Convert the action sequence into a two-dimensional feature matrix:
[0059] Next, the action sequence is converted into multiple two-dimensional feature matrices. According to the size of the feature matrix, the corresponding number of data are taken out from the action sequence and stored into the feature matrix in turn. The storage rule is row-major. The length of the action sequence is the product of the size and number of feature matrices. In this way, the entire action sequence can be converted into two-dimensional feature matrices.
[0060] (6) Convolve all feature matrices:
[0061] Finally, all the two-dimensional feature matrices are input into a convolutional neural network for processing. The convolutional neural network can stack and merge all the feature matrices to obtain the feature matrix of a single video for a student's learning. The convolutional neural network is usually composed of multiple convolutional layers, pooling layers and fully connected layers, which can stack and process each feature block to obtain a single feature matrix.
[0062] Furthermore, step 4) involves concatenating the feature matrices of all the videos for this course, including the following steps:
[0063] (1) For each feature matrix, its corresponding weight needs to be calculated based on the length of the video and the importance of the knowledge point; the video length refers to the number of feature matrices contained in the video, and the importance of the knowledge point is evaluated by experts based on the difficulty and importance of the course.
[0064] (2) The weighted average method is adopted, that is, the weight of each feature matrix is multiplied by the weight of the video in which it is located, and then the weighted sum of all feature matrices is divided by the total weight to obtain the weighted average.
[0065] (3) Concatenate all feature matrices according to their order in the video. First, multiply each feature matrix by its corresponding weight value. Then, calculate the number of feature matrices needed for each row. Concatenate all feature matrices sequentially. Finally, check if the image size is fully utilized. If not, pad with zeros to the specified image size. In other words, check if the image size after concatenating the feature matrices from multiple videos meets the specified size. For example, if the concatenated feature matrix size is 126x126, but a 128x128 image size is required, padding is necessary. This applies to the feature matrix of the entire course.
[0066] Furthermore, in step 5), the above feature matrix is converted into a three-channel image output according to the requirements, including the following conversion methods: using the matplotlib library for conversion. The matplotlib library provides the plt.imshow function to convert a single-channel grayscale image into a color RGB image. This function will automatically map the grayscale values to the corresponding RGB values according to the color mapping table cmap, thereby obtaining a color image.
[0067] Furthermore, in step 5), the above feature matrix is converted into a three-channel image output according to the requirements, including the following conversion methods: a custom method is used for channel conversion, which adopts a simple linear mapping method, that is, a piecewise function is set to calculate the corresponding RGB value based on the gray value, the gray value range is divided into several intervals, a corresponding RGB value is set for each interval, and then the gray value is mapped to the corresponding interval according to the gray value, and the corresponding RGB value is calculated to obtain a color image.
[0068] The present invention also provides an electronic device, characterized in that it includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor runs the computer program, it performs the steps of the above-described method for image representation of learning process sequence data based on a convolutional neural network.
[0069] The present invention also provides a computer-readable storage medium storing a computer program, characterized in that the computer program causes a computer to execute the steps of the above-described method for visualizing learning process sequence data based on a convolutional neural network.
[0070] Specific embodiments are shown below:
[0071] First, the input data is processed. The input data for this method is a series of action sequences generated by students while watching the teaching video, including actions such as starting to watch, playing, pausing, and finishing.
[0072] Secondly, these action sequences need to be cleaned and processed to obtain a data list containing the timestamps of each action, that is, [(t1,a1),(t2,a2),...,(t n ,a n )], where t i represents the timestamp of the i-th action, and a i represents the type of the i-th action.
[0073] Next, the time series is divided into several feature matrices of a fixed length. Assuming the feature block size is w×h, the time series is divided into feature matrices. For the convenience of convolution operations, the time series needs to be padded with 0s, that is, the length of the time series is padded to mwh, and the padded part is all filled with 0s. A video has multiple feature matrices, m represents the number of feature matrices of a single video, w represents the width of each feature matrix in a single video, and h represents the height of each feature matrix in a single video.
[0074] Then, the mwh actions are arranged in order into a two-dimensional matrix X m×wh , where the i-th row represents the action sequence of the i-th feature block. Then, X is subjected to a convolution operation to obtain a w×h feature matrix F.
[0075] Define a k×k convolution kernel K, where K(a,b) represents the element in the a-th row and b-th column of the convolution kernel. For each position (x,y) in the feature matrix F, where 1≤i≤m, 0≤x<w, 0≤y<h, the specific process of the convolution operation can be expressed as:
[0076]
[0077] where i represents the index of the feature block, with a value range of 1 to m, and a and b respectively represent the row and column indices of the convolution kernel K, with a value range of 0 to k-1. The above formula means that each position (x,y) in the feature matrix F is obtained by multiplying each element in the matrix X by the corresponding element in the convolution kernel K and summing up the products. <00001I is the length of video i, n is the total number of videos, and I k This represents the importance of knowledge point k, where k is the total number of knowledge points. In this formula, the first fraction calculates the proportion of each video in the entire course, and the second fraction calculates the proportion of each knowledge point in the entire course.
[0081] After calculating the weights of each feature matrix, these feature matrices can be concatenated. The concatenation process can be performed using the following formula:
[0082]
[0083] Where F is the feature matrix of the entire course, n is the total number of videos, and M is the feature matrix of the entire course. i w is the feature matrix of the i-th video. i This represents the weight of the i-th video. In this formula, each feature matrix is multiplied by its corresponding weight, and then the results are summed to obtain the feature matrix of the entire course.
[0084] Finally, the generated feature matrix is converted into an image for output. Since the feature matrix is single-channel, a single-channel grayscale image can be directly output. Alternatively, Matplotlib can be used to convert the single-channel grayscale image into a three-channel RGB image, or a custom method can be used for channel conversion. The rule for custom methods is to set a piecewise function to calculate the corresponding RGB values based on the grayscale values.
[0085] The specific formula for custom channel conversion can be expressed as:
[0086]
[0087] f refers to a custom channel conversion function, r is the brightness of red, g is the brightness of green, and b is the brightness of blue.
[0088] The above description is merely a preferred embodiment of the present invention and is not intended to limit this application. For those skilled in the art, various modifications and variations of the embodiments of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A method for visualizing learning process sequence data based on convolutional neural networks, characterized in that, Includes the following steps: 1) Collect data on a series of action sequences generated by students while watching instructional videos; 2) The collected action sequence data generated by students while watching the instructional videos is cleaned and processed to obtain a data list containing the timestamp of each action, i.e., [(t1,a1),(t2,a2),...,(t...]. n ,a n )], where t i Let a represent the timestamp of the i-th action. i This indicates the type of the i-th action; the purpose of data cleaning is to remove useless and abnormal data. During the data cleaning process, data preprocessing and correction are required according to the actual situation to ensure the accuracy and reliability of the data. 3) Generate a corresponding feature matrix from the series of action sequences generated by students while watching the teaching videos, according to certain rules, so as to express the students' learning status while watching the teaching videos; 4) Concatenate the above components according to certain rules to generate all corresponding feature matrices; 5) Convert the above feature matrix into single-channel or three-channel image output according to the requirements to express the learning situation of the entire course; Step 3) involves generating a corresponding feature matrix from the series of action sequences generated by students while watching instructional videos, according to certain rules. This includes the following steps: (1) Calculate the size of the feature matrix: First, the size of the feature matrix needs to be calculated, which is based on the number of course videos and the size of the output images. (2) Input the student's action sequence and count the number of all actions that appear in the video; The system requires inputting the student's action sequence and counting the total number of actions performed by the student while watching the video; the student's action sequence is the action recorded by the system backend. (3) Calculate the number of feature blocks required: Based on the number of all actions appearing in the video and the size of the feature matrix, the number of feature matrices to be divided is calculated. Based on the number of feature matrices, the student's action sequence is divided into multiple subsequences, and each subsequence is processed. (4) Zero-filling of the action sequence: To ensure that the length of each action sequence is sufficient to fill the feature matrix, each subsequence needs to be zero-padded. Zero-padded means adding zero elements to the end of the action sequence so that the length of all action sequences meets the requirements. The specific padding length can be calculated based on the size and number of feature matrices. (5) Convert the action sequence into a two-dimensional feature matrix: Next, the action sequence is converted into multiple two-dimensional feature matrices; Based on the size of the feature matrix, the corresponding number of data are sequentially extracted from the action sequence and stored into the feature matrix. The storage rule is row-major. The length of the action sequence is the product of the size and number of feature matrices. In this way, the entire action sequence can be converted into a two-dimensional feature matrix. (6) Convolve all feature matrices: Finally, all the two-dimensional feature matrices are input into a convolutional neural network for processing. The convolutional neural network stacks and merges all the feature matrices to obtain the feature matrix of a single video in which the student learns. The convolutional neural network consists of multiple convolutional layers, pooling layers, and fully connected layers, which stack and process each feature block to obtain a single feature matrix.
2. The method for visualizing learning process sequence data based on convolutional neural networks according to claim 1, characterized in that: Step 4) involves concatenating the feature matrices of all videos for this course, including the following steps: (1) For each feature matrix, its corresponding weight needs to be calculated based on the length of the video and the importance of the knowledge point; the video length refers to the number of feature matrices contained in the video, and the importance of the knowledge point is evaluated by experts based on the difficulty and importance of the course. (2) The weighted average method is adopted, that is, the weight of each feature matrix is multiplied by the weight of the video in which it is located, and then the weighted sum of all feature matrices is divided by the total weight to obtain the weighted average. (3) Concatenate all feature matrices in the order they appear in the video. First, multiply all feature matrices by their corresponding weight values. Then, calculate the number of feature matrices needed for each row. Concatenate all feature matrices in sequence. Finally, determine whether the image size is fully utilized. If not, pad with zeros to the specified image size.
3. The method for visualizing learning process sequence data based on convolutional neural networks according to claim 1, characterized in that: In step 5), the above feature matrix is converted into a three-channel image output according to the requirements, including the following conversion methods: using the matplotlib library for conversion. The matplotlib library provides the plt.imshow function to convert a single-channel grayscale image into a color RGB image. This function will automatically map the grayscale values to the corresponding RGB values according to the color mapping table cmap, thereby obtaining a color image.
4. The method for visualizing learning process sequence data based on convolutional neural networks according to claim 1, characterized in that: In step 5), the above feature matrix is converted into a three-channel image output according to the requirements, including the following conversion methods: using a custom method for channel conversion, adopting a linear mapping method, that is, setting a piecewise function to calculate the corresponding RGB value based on the gray value, dividing the gray value range into several intervals, setting a corresponding RGB value for each interval, and then mapping it to the corresponding interval according to the gray value, and calculating the corresponding RGB value to obtain a color image.
5. An electronic device, characterized in that: It includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor runs the computer program, it performs the steps of a method for visualizing learning process sequence data based on a convolutional neural network as described in any one of claims 1-4.
6. A computer-readable storage medium storing a computer program, characterized in that, The computer program causes the computer to perform the steps of the method for visualizing learning process sequence data based on a convolutional neural network as described in any one of claims 1-4.