Data mining method and system based on big data

Abstract

The invention relates to the technical field of information retrieval, and discloses a data mining method and system based on big data. The method comprises the steps of collecting multi-source heterogeneous data through a distributed computing framework and storing the multi-source heterogeneous data into an HDFS file storage; constructing a feature matrix; performing clustering analysis on the data after dimension reduction by adopting a Spark-based clustering algorithm; visual display is carried out through a UniApp platform; compared with a common data analysis method in the prior art, the technical problem that the learning mode and demand of a student group cannot be accurately found due to the fact that the heterogeneous data is difficult to process efficiently and accurately in the traditional technology especially in the face of large-scale and diversified education data such as learning behaviors and course feedback of students is solved. Due to the fact that the distributed computing framework and the clustering algorithm are introduced, the efficiency and accuracy of large-scale data processing are improved, the quality and value of data mining are improved, and data support is provided for education decision making.

Description

Technical Field

[0001] The present invention belongs to the technical field of information retrieval technology, and in particular to a data mining method and system based on big data. Background Art

[0002] At present, with the continuous development of information technology, the amount of data in the field of education has shown an exponential growth, especially in online education and intelligent education systems, where the amount of data such as students' learning behavior, course feedback, and teaching content is increasing. However, existing traditional data analysis methods, such as rule-based statistical analysis methods, can often only process relatively simple and small amounts of data, and have great limitations in the processing and analysis of multi-source heterogeneous data. For example, when processing students' online learning behavior data, course feedback data, and teaching content published by teachers, existing technologies are difficult to efficiently integrate these heterogeneous data sources, and are unable to fully explore potential student learning patterns and course optimization opportunities. Especially when the amount of data is extremely large, existing systems often cannot efficiently complete a series of tasks such as data cleaning, feature extraction, dimensionality reduction, and model training. Therefore, there is an urgent need for an efficient and accurate data mining method and system that can still achieve high-efficiency and high-precision data mining in the face of large-scale, multi-source heterogeneous data, so as to improve the accuracy and efficiency of education data analysis. Summary of the invention

[0003] In view of the above-mentioned technical deficiencies, the purpose of the present invention is to propose a data mining method based on big data, aiming to solve the common data analysis methods in the prior art, especially when faced with large-scale and diversified educational data such as students' learning behavior, course feedback, etc., traditional technologies are difficult to efficiently and accurately process such heterogeneous data, and thus cannot accurately discover the learning patterns and needs of student groups.

[0004] In order to solve the above technical problems, the present invention adopts the following technical solutions: The present invention provides a data mining method based on big data.

[0005] The data mining method based on big data includes:

[0006] Step S10: Collect multi-source heterogeneous data through a distributed computing framework and store it in HDFS file storage. The multi-source heterogeneous data includes:

[0007] Student learning behavior data, including student ID, course ID, online learning duration field, and homework completion rate field;

[0008] Course feedback data, including students’ course satisfaction ratings and comment content fields;

[0009] Teaching support data, including course content outlines composed of structured fields published by teachers;

[0010] Multi-source heterogeneous data is transmitted through the Kafka streaming processing framework and stored in a distributed JSON file indexed by timestamps;

[0011] Step S20: performing cleaning processing on multi-source heterogeneous data;

[0012] For the online learning duration field, the time series interpolation method is used to fill in missing values, and data with a value greater than two times the standard deviation are removed;

[0013] The job completion rate field is filled with the historical average value, and abnormal records exceeding 100% or below 0% are removed;

[0014] Perform word segmentation on the comment content fields in the course feedback data, remove stop words, and calculate the positive and negative sentiment tendency scores;

[0015] Generate and output the cleaned standardized data set D2;

[0016] Step S30: extracting numerical features from the standardized data set D2, including numerical features of learning time, numerical features of homework completion rate, and numerical features of satisfaction score, and constructing a feature matrix X;

[0017] Construct a covariance matrix C based on the feature matrix X, calculate the n eigenvalues ​​and n eigenvectors of the covariance matrix C, sort the n eigenvalues ​​from large to small, calculate the contribution rate of the n eigenvectors, select k eigenvectors with a contribution rate greater than 95% as the principal components, construct a k×k principal component vector matrix P, and apply weight adjustment to the principal component vector matrix P in combination with the course satisfaction score to output the reduced dimension feature matrix P′;

[0018] Step S40: using the K-means clustering algorithm in the machine learning library MLlib of the distributed computing framework Spark to perform cluster analysis on the feature matrix P′ after dimension reduction to obtain cluster analysis results, the cluster analysis results include cluster labels, cluster centers, statistical information of cluster groups, and distances between clusters; the statistical information of cluster groups includes the number of students in the cluster group, the average learning time in the cluster group, the homework completion rate, and the satisfaction score;

[0019] Step S50: Display the statistical information of each cluster group across platforms through UniApp, and dynamically visualize the cluster statistical information using data binding and chart components in UniApp.

[0020] Preferably, in step S20, the standardized data set D2 includes:

[0021] Student behavior characteristics, including student ID, course ID, study time field, and homework completion rate field;

[0022] Course feedback features include satisfaction rating field, keyword field, and sentiment tendency field.

[0023] Preferably, in step S30, the formula for constructing the covariance matrix C is:

[0024]

[0025] Among them, X is the feature matrix, X T is the transpose of the feature matrix.

[0026] Preferably, in step S30, the contribution rate calculation formula of the feature vector is:

[0027]

[0028] Among them, i is the index, λ i is the eigenvalue of the i-th eigenvector, is the sum of all eigenvalues.

[0029] Preferably, in step S40, the K-means clustering algorithm adopts the formula:

[0030]

[0031] Among them, J is the objective function, K is the preset number of clusters, b and v are indexes, C b is the bth cluster, which contains all the sample points belonging to this cluster, x v is the sample point belonging to the vth cluster, μ b is the average value of all sample points in cluster b.

[0032] Preferably, in step S20, the positive and negative sentiment tendency scores are obtained by statistics and weighting based on a trained sentiment dictionary.

[0033] Preferably, in step S50, the chart component is visualized using the echars component.

[0034] The present invention also provides a data mining system based on big data, comprising:

[0035] The data collection module is used to collect multi-source heterogeneous data through a distributed computing framework and store it in HDFS file storage. The multi-source heterogeneous data includes:

[0036] Student learning behavior data, including student ID, course ID, online learning duration field, and homework completion rate field;

[0037] Course feedback data, including students’ course satisfaction ratings and comment content fields;

[0038] Teaching support data, including course content outlines composed of structured fields published by teachers;

[0039] Multi-source heterogeneous data is transmitted through the Kafka streaming processing framework and stored in a distributed JSON file indexed by timestamps;

[0040] Data cleaning module, used to perform cleaning processing on multi-source heterogeneous data;

[0041] For the online learning duration field, the time series interpolation method is used to fill in missing values, and data with a value greater than two times the standard deviation are removed;

[0042] The job completion rate field is filled with the historical average value, and abnormal records exceeding 100% or below 0% are removed;

[0043] Perform word segmentation on the comment content fields in the course feedback data, remove stop words, and calculate the positive and negative sentiment tendency scores;

[0044] Generate and output the cleaned standardized data set D2;

[0045] The dimension reduction processing module is used to extract numerical features from the standardized data set D2, including numerical features of learning time, numerical features of homework completion rate, and numerical features of satisfaction score, and construct a feature matrix X;

[0046] Construct a covariance matrix C based on the feature matrix X, calculate the n eigenvalues ​​and n eigenvectors of the covariance matrix C, sort the n eigenvalues ​​from large to small, calculate the contribution rate of the n eigenvectors, select k eigenvectors with a cumulative contribution rate greater than 95% as the principal components, construct a k×k principal component vector matrix P, and apply weight adjustment to the principal component vector matrix P in combination with the course satisfaction score to output the reduced dimension feature matrix P′;

[0047] The cluster analysis module is used to perform cluster analysis on the feature matrix P′ after dimension reduction using the K-means clustering algorithm in the machine learning library MLlib of the distributed computing framework Spark to obtain cluster analysis results. The cluster analysis results include cluster labels, cluster centers, statistical information of cluster groups, and distances between clusters. The statistical information of cluster groups includes the number of students in the cluster group, the average learning time in the cluster group, the homework completion rate, and the satisfaction score.

[0048] The result display module is used to display the statistical information of each cluster group across the UniApp platform, and dynamically visualize the cluster statistical information using the data binding and chart components in the UniApp.

[0049] The present invention also provides a data mining device based on big data, comprising a memory, a processor, and a data mining program based on big data stored in the memory and executable on the processor, wherein the data mining program based on big data implements the data mining method based on big data when executed by the processor.

[0050] The present invention also provides a computer program product, including a data mining program based on big data, and when the data mining program based on big data is executed by a processor, the data mining method based on big data is implemented.

[0051] The beneficial effect of the present invention is that compared with the common data analysis methods in the prior art, especially when faced with large-scale and diversified educational data such as students' learning behavior, course feedback, etc., traditional technologies are difficult to efficiently and accurately process such heterogeneous data, and thus cannot accurately discover the learning patterns and needs of student groups. Technical problems, because this application introduces a distributed computing framework and a clustering algorithm, improves the efficiency and accuracy of large-scale data processing, improves the quality and value of data mining, and provides data support for educational decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0053] Figure 1 It is a flowchart diagram of a first embodiment of a data mining method based on big data of the present invention.

[0054] Figure 2 The present invention is a schematic diagram of a device for a data mining method based on big data. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] Embodiment 1: Figure 1 , which is a flow chart of the first embodiment of the data mining method based on big data of the present invention, proposes the first embodiment of the data mining method based on big data of the present invention.

[0057] In a first embodiment, the data mining method based on big data includes:

[0058] Step S10: Collect multi-source heterogeneous data through a distributed computing framework and store it in HDFS file storage. The multi-source heterogeneous data includes:

[0059] Student learning behavior data, including student ID, course ID, online learning duration field, and homework completion rate field;

[0060] Course feedback data, including students’ course satisfaction ratings and comment content fields;

[0061] Teaching support data, including course content outlines composed of structured fields published by teachers;

[0062] Multi-source heterogeneous data is transmitted through the Kafka streaming processing framework and stored in a distributed JSON file indexed by timestamps;

[0063] It should be noted that HDFS is a distributed file system that supports the storage and processing of large-scale data sets. It has the characteristics of high reliability, high fault tolerance and high throughput, and is suitable for the distributed storage needs of big data. Kafka is a distributed message queue system that supports high-performance data transmission and real-time stream processing.

[0064] It is understandable that the combination of HDFS and Kafka fully utilizes the distributed storage capabilities of HDFS and the efficient streaming data transmission capabilities of Kafka, ensuring the efficiency and reliability of multi-source heterogeneous data during collection, transmission and storage; and the choice of JSON file format not only facilitates data storage and management, but also provides a good foundation for subsequent data cleaning and mining.

[0065] Step S20: performing cleaning processing on multi-source heterogeneous data;

[0066] For the online learning duration field, the time series interpolation method is used to fill in missing values, and data with a value greater than two times the standard deviation are removed;

[0067] The job completion rate field is filled with the historical average value, and abnormal records exceeding 100% or below 0% are removed;

[0068] Perform word segmentation on the comment content fields in the course feedback data, remove stop words, and calculate the positive and negative sentiment tendency scores;

[0069] Generate and output the cleaned standardized data set D2;

[0070] It should be noted that the time series interpolation method can effectively fill in the missing values ​​in the time series data due to insufficient sampling or transmission loss to ensure the continuity of the data; eliminating data beyond the range of two times the standard deviation can avoid the influence of extreme values ​​on subsequent analysis results, thereby improving the accuracy of the analysis; calculating the positive and negative sentiment scores of the comment text based on the sentiment analysis method provides quantitative indicators for student feedback data.

[0071] For example, during the learning process of a student, the online learning time field of some courses was missing. The system used time series interpolation method to complete it to ensure data integrity; the homework completion rate of a student was recorded as 105%, which exceeded the normal range (0%-100%), and the system automatically deleted this abnormal record; in the feedback of a course, the student commented that "this course is very good and I gained a lot." The system extracted "very good" and "greatly gained" through word segmentation and calculated the positive sentiment tendency score as 0.9.

[0072] Step S30: extracting numerical features from the standardized data set D2, including numerical features of learning time, numerical features of homework completion rate, and numerical features of satisfaction score, and constructing a feature matrix X;

[0073] Construct a covariance matrix C based on the feature matrix X, calculate the n eigenvalues ​​and n eigenvectors of the covariance matrix C, sort the n eigenvalues ​​from large to small, calculate the contribution rate of the n eigenvectors, select k eigenvectors with a contribution rate greater than 95% as the principal components, construct a k×k principal component vector matrix P, and apply weight adjustment to the principal component vector matrix P in combination with the course satisfaction score to output the reduced dimension feature matrix P′;

[0074] It should be noted that the covariance matrix is ​​a basic tool in statistics that can reflect the correlation between multiple variables. In this step, the eigenvalues ​​and eigenvectors of the covariance matrix C are used to measure the importance of the variables and determine the main components retained after dimensionality reduction; the principal component selection criterion with a contribution rate greater than 95% ensures that the data after dimensionality reduction can still fully reflect the main information of the original features; the weight adjustment of the principal components makes the feature matrix after dimensionality reduction more in line with the actual needs of educational data, such as paying more attention to students' satisfaction feedback information, thereby improving the decision-making support capabilities of data mining results.

[0075] Step S40: using the K-means clustering algorithm in the machine learning library MLlib of the distributed computing framework Spark to perform cluster analysis on the feature matrix P′ after dimension reduction to obtain cluster analysis results, the cluster analysis results include cluster labels, cluster centers, statistical information of cluster groups, and distances between clusters; the statistical information of cluster groups includes the number of students in the cluster group, the average learning time in the cluster group, the homework completion rate, and the satisfaction score;

[0076] It should be noted that the K-means algorithm is a widely used unsupervised learning method that finds the clustering pattern of data by minimizing the distance between samples in a cluster group and the cluster center; Spark MLlib is a distributed machine learning library that can use distributed computing power to process large-scale data, especially suitable for processing data with high dimensionality and large-scale characteristics such as educational data;

[0077] It is understandable that MLlib using Spark can effectively utilize distributed computing resources and quickly complete large-scale data clustering tasks. Even when faced with millions of educational records, clustering results can be output within a reasonable time. The cluster labels and cluster centers of the K-means algorithm can clearly describe the learning patterns of student groups and provide guidance for subsequent data analysis and decision-making.

[0078] For example, the K-means algorithm can be used to divide student data into three cluster groups:

[0079] Cluster 1: contains 100 students, whose average study time is 30 hours, homework completion rate is 80%, and satisfaction score is 4.5.

[0080] Cluster 2: contains 50 students who have a shorter study time (average 10 hours), a lower homework completion rate (50%), and a satisfaction score of 3.0.

[0081] Cluster 3: contains 150 students who have a longer study time (50 hours on average), a higher homework completion rate (95%), and a satisfaction score of 4.8.

[0082] Step S50: Display the statistical information of each cluster group across platforms through UniApp, and dynamically visualize the cluster statistical information using data binding and chart components in UniApp.

[0083] It should be noted that the cross-platform capability of the UniApp framework enables the developed applications to present statistical information uniformly on different platforms without the need for separate development for each platform, which greatly improves development efficiency and system maintainability; the dynamic data binding function ensures that the display page can update the clustering analysis results in real time, and users can see the latest results without manually refreshing the page; chart components such as ECharts provide a wealth of graphical display methods, including bar charts, pie charts, line charts, etc., which help to intuitively present clustering statistical information and facilitate user understanding.

[0084] Understandably, dynamic visualization can help users quickly identify data patterns, such as the distribution of study time and differences in homework completion rates among different cluster groups, providing data support for educational decision-making. Through UniApp's chart component, complex clustering results can be presented in a visual way, converting the technical details of the data into easy-to-understand graphical content.

[0085] For example, the clustering results of a course are divided into three groups: Group A: 100 students, with an average study time of 30 hours, an assignment completion rate of 80%, and a satisfaction score of 4.5; Group B: 50 students, with an average study time of 10 hours, an assignment completion rate of 50%, and a satisfaction score of 3.0; Group C: 150 students, with an average study time of 50 hours, an assignment completion rate of 95%, and a satisfaction score of 4.8; the visualization effect is a bar chart showing the comparison of the study time, assignment completion rate, and satisfaction score of each group; the pie chart shows the distribution of the proportion of students in each group; users can view the detailed statistical information of the group by clicking on a bar chart.

[0086] In addition, the data mining system based on big data provided by the present invention adopts a data mining method based on big data in the above embodiment, which can solve a technical problem of data mining based on big data. Compared with the prior art, the beneficial effect of the data mining system based on big data provided by the present invention is the same as the beneficial effect of the data mining method based on big data provided by the above embodiment, and the other technical features of the data mining system based on big data are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0087] The present invention provides a data mining device based on big data, please refer to Figure 2, a data mining device based on big data includes: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a data mining method based on big data in the above-mentioned embodiment 1. A data mining device based on big data in the embodiment of the present invention may include but is not limited to mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions: tablet computers), PMPs (Portable Media Players: portable multimedia players), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. A data mining device based on big data is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention. A data mining device based on big data may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of a data mining device based on big data are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow a data mining device based on big data to communicate with other devices wirelessly or wired to exchange data. Although a data mining device based on big data with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have alternatively.

[0088] The present invention also provides a computer program product, including a computer program, which implements the steps of a data mining method based on big data as described above when the computer program is executed by a processor. The computer program product provided by the present invention can solve a technical problem of data mining based on big data. Compared with the prior art, the beneficial effects of the computer program product provided by the present invention are the same as the beneficial effects of a data mining method based on big data provided by the above embodiment, which will not be described in detail here.

[0089] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through a communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present invention are executed.

[0090] It should be understood that the various parts disclosed in the present invention can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0091] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A data mining method based on big data, characterized in that: Methods include: Step S10: Collect multi-source heterogeneous data through a distributed computing framework and store it in HDFS file storage. The multi-source heterogeneous data includes: Student learning behavior data, including student ID, course ID, online learning duration field, and homework completion rate field; Course feedback data, including students’ course satisfaction ratings and comment content fields; Teaching support data, including course content outlines composed of structured fields published by teachers; Multi-source heterogeneous data is transmitted through the Kafka streaming processing framework and stored in a distributed JSON file indexed by timestamps; Step S20: performing cleaning processing on multi-source heterogeneous data; For the online learning duration field, the time series interpolation method is used to fill in missing values, and data with a value greater than two times the standard deviation are removed; The job completion rate field is filled with the historical average value, and abnormal records exceeding 100% or below 0% are removed; Perform word segmentation on the comment content fields in the course feedback data, remove stop words, and calculate the positive and negative sentiment tendency scores; Generate and output the cleaned standardized data set D2; Step S30: extracting numerical features from the standardized data set D2, including numerical features of learning time, numerical features of homework completion rate, and numerical features of satisfaction score, and constructing a feature matrix X; Construct a covariance matrix C based on the feature matrix X, calculate the n eigenvalues ​​and n eigenvectors of the covariance matrix C, sort the n eigenvalues ​​from large to small, calculate the contribution rate of the n eigenvectors, select k eigenvectors with a contribution rate greater than 95% as the principal components, construct a k×k principal component vector matrix P, and apply weight adjustment to the principal component vector matrix P in combination with the course satisfaction score to output the reduced dimension feature matrix P′; Step S40: using the K-means clustering algorithm in the machine learning library MLlib of the distributed computing framework Spark to perform cluster analysis on the feature matrix P′ after dimension reduction to obtain cluster analysis results, the cluster analysis results include cluster labels, cluster centers, statistical information of cluster groups, and distances between clusters; the statistical information of cluster groups includes the number of students in the cluster group, the average learning time in the cluster group, the homework completion rate, and the satisfaction score; Step S50: Display the statistical information of each cluster group across platforms through UniApp, and dynamically visualize the cluster statistical information using data binding and chart components in UniApp.

2. A data mining method based on big data as claimed in claim 1, characterized in that: In step S20, the standardized data set D2 includes: Student behavior characteristics, including student ID, course ID, study time field, and homework completion rate field; Course feedback features include satisfaction rating field, keyword field, and sentiment tendency field.

3. A data mining method based on big data as claimed in claim 1, characterized in that: In step S30, the formula for constructing the covariance matrix C is: Among them, X is the feature matrix, X T is the transpose of the feature matrix.

4. A data mining method based on big data as claimed in claim 1, characterized in that: In step S30, the contribution rate of the feature vector is calculated as follows: Among them, i is the index, λ i is the eigenvalue of the i-th eigenvector, is the sum of all eigenvalues.

5. The data mining method based on big data according to claim 1, characterized in that: In step S40, the K-means clustering algorithm uses the formula: Among them, J is the objective function, K is the preset number of clusters, b and v are indexes, C b is the bth cluster, which contains all the sample points belonging to this cluster, x v is the sample point belonging to the vth cluster, μ b is the average value of all sample points in cluster b.

6. A data mining method based on big data as claimed in claim 1, characterized in that: In step S20, the positive and negative sentiment tendency scores are obtained by counting and weighting according to the trained sentiment dictionary.

7. A data mining method based on big data as claimed in claim 1, characterized in that: In step S50, the chart component is visualized using the echars component.

8. A data mining system based on big data, characterized in that: The data mining system based on big data includes: The data collection module is used to collect multi-source heterogeneous data through a distributed computing framework and store it in HDFS file storage. The multi-source heterogeneous data includes: Student learning behavior data, including student ID, course ID, online learning duration field, and homework completion rate field; Course feedback data, including students’ course satisfaction ratings and comment content fields; Teaching support data, including course content outlines composed of structured fields published by teachers; Multi-source heterogeneous data is transmitted through the Kafka streaming processing framework and stored in a distributed JSON file indexed by timestamps; Data cleaning module, used to perform cleaning processing on multi-source heterogeneous data; For the online learning duration field, the time series interpolation method is used to fill in missing values, and data with a value greater than two times the standard deviation are removed; The job completion rate field is filled with the historical average value, and abnormal records exceeding 100% or below 0% are removed; Perform word segmentation on the comment content fields in the course feedback data, remove stop words, and calculate the positive and negative sentiment tendency scores; Generate and output the cleaned standardized data set D2; The dimension reduction processing module is used to extract numerical features from the standardized data set D2, including numerical features of learning time, numerical features of homework completion rate, and numerical features of satisfaction score, and construct a feature matrix X; Construct a covariance matrix C based on the feature matrix X, calculate the n eigenvalues ​​and n eigenvectors of the covariance matrix C, sort the n eigenvalues ​​from large to small, calculate the contribution rate of the n eigenvectors, select k eigenvectors with a cumulative contribution rate greater than 95% as the principal components, construct a k×k principal component vector matrix P, and apply weight adjustment to the principal component vector matrix P in combination with the course satisfaction score to output the reduced dimension feature matrix P′; The cluster analysis module is used to perform cluster analysis on the feature matrix P′ after dimension reduction using the K-means clustering algorithm in the machine learning library MLlib of the distributed computing framework Spark to obtain cluster analysis results. The cluster analysis results include cluster labels, cluster centers, statistical information of cluster groups, and distances between clusters. The statistical information of cluster groups includes the number of students in the cluster group, the average learning time in the cluster group, the homework completion rate, and the satisfaction score. The result display module is used to display the statistical information of each cluster group across the UniApp platform, and dynamically visualize the cluster statistical information using the data binding and chart components in the UniApp.

9. A data mining device based on big data, characterized in that: The data mining device based on big data comprises: a memory, a processor and a data mining program based on big data stored in the memory and executable on the processor. When the data mining program based on big data is executed by the processor, the data mining method based on big data described in any one of claims 1 to 7 is implemented.

10. A computer program product, characterized in that The computer program product includes a data mining program based on big data, and when the data mining program based on big data is executed by a processor, it implements the data mining method based on big data described in any one of Claims 1 to 7.

Images