Full-data resource educational information platform deep analysis system and method based on structured and unstructured data fusion

Abstract

The invention discloses a full-data resource education information platform deep analysis system and method based on structured and unstructured data fusion. The system comprises a data acquisition and integration module; a data processing and analyzing module; a personalized recommendation and learning path planning module; a visual display and customized report generation module; and an authority management and security audit module. Education data are collected through the data acquisition and integration module, the data are cleaned, converted and the like through the data processing and analysis module, the accuracy and availability of the data are ensured, valuable information and modes are extracted through the personalized recommendation and learning path planning module, and the learning path planning module is used for planning the learning path. According to the system, personalized educational resource recommendation, learning path planning, teaching decision support and the like are provided for different user groups, analysis result checking, personalized recommendation receiving, interaction and feedback participation are realized conveniently through a visual display and customized report generation module, and a closed-loop educational resource optimization and service improvement mechanism is formed.

Description

Technical Field

[0001] The present invention belongs to the field of information technology, and specifically relates to a full-data resource education information platform in-depth analysis system and method based on the fusion of structured and unstructured data. Background Art

[0002] At present, educational informatization has been widely promoted and applied around the world. However, there are still many challenges in practical application. On the one hand, educational data is diverse, complex and dynamic, and traditional data processing and analysis methods are difficult to meet current needs. On the other hand, the data island phenomenon between different educational systems is serious, and data sharing and integration are difficult. In addition, data security and privacy protection issues are also one of the important challenges facing educational informatization. Summary of the invention

[0003] In order to solve the technical problems existing in the prior art, the purpose of the present invention is to provide a deep analysis system for the education information platform of all data resources based on the fusion of structured and unstructured data.

[0004] In order to achieve the above purpose and the above technical effect, the technical solution adopted by the present invention is:

[0005] A deep analysis system for the full data resource education information platform based on the fusion of structured and unstructured data, including:

[0006] Data collection and integration module, responsible for collecting data from multiple data sources and integrating them;

[0007] Data processing and analysis module, responsible for in-depth processing and analysis of the integrated data;

[0008] Personalized recommendation and learning path planning module;

[0009] Visual display and customized report generation module;

[0010] The authority management and security audit modules ensure the security and integrity of data.

[0011] Furthermore, the data collection and integration module includes:

[0012] Data source identification and configuration module, responsible for completing data source identification and data source configuration;

[0013] The data capture module is responsible for the scheduling and management of data collection tasks, data request and data reception;

[0014] The data source dynamic addition and deletion module is responsible for completing the dynamic addition and deletion of data sources;

[0015] The data cleaning module is responsible for completing data deduplication, error handling, and standardization processing;

[0016] Data conversion module, responsible for completing the structured conversion and mapping of data;

[0017] The data loading module is responsible for completing data storage, loading, indexing and optimization.

[0018] Furthermore, the data processing and analysis module includes:

[0019] Data preprocessing module, responsible for deep data cleaning and complex data conversion;

[0020] The data analysis and mining module is responsible for implementing in-depth descriptive statistical analysis of data, refined exploratory data analysis, advanced prediction and classification, and text and image analysis;

[0021] Customized data processing and analysis modules;

[0022] Intelligent recommendation module, responsible for intelligently recommending appropriate data processing methods and analysis models based on user historical behavior and data analysis results;

[0023] The monitoring module is responsible for automating and monitoring the data processing and analysis processes;

[0024] The optimization module is responsible for using machine learning technology to continuously optimize and improve data processing and analysis processes to improve overall efficiency and effectiveness.

[0025] Furthermore, the personalized recommendation and learning path planning module includes:

[0026] The personalized learning model building module is responsible for building a personalized learning model based on students’ learning behavior data and performance data;

[0027] The learning path planning module is responsible for recommending learning resources and paths that meet students' interests and needs through collaborative filtering algorithms and content-based recommendation algorithms. It also supports the learning path planning function to plan reasonable learning paths for students based on their learning goals and current learning status, and dynamically adjust them to ensure that students achieve their learning goals in the shortest time.

[0028] Furthermore, the visualization and customized report generation module includes:

[0029] Real-time data dashboard, which supports real-time data updates, allows users to freely configure the displayed information according to their needs, and presents it in an intuitive chart form;

[0030] Chart display module;

[0031] The geospatial visualization module is responsible for visualizing data with geographic location attributes;

[0032] Reporting module;

[0033] Report generation module;

[0034] Collaboration module.

[0035] Furthermore, the authority management and security audit module includes:

[0036] Permission management module;

[0037] Dynamic rights management module;

[0038] Security audit module;

[0039] Regular review and analysis of modules.

[0040] The present invention also discloses a method for deep analysis of a full data resource education information platform based on the fusion of structured and unstructured data, which uses the deep analysis system for a full data resource education information platform based on the fusion of structured and unstructured data as described above for analysis, and the method comprises the following steps:

[0041] S1: Data source identification and configuration:

[0042] 1.1、Data source identification;

[0043] 1.2、Data source configuration;

[0044] S2: Data capture:

[0045] 2.1、Task scheduling;

[0046] 2.2 Data request;

[0047] 2.3、Data reception;

[0048] S3: Dynamic addition and deletion of data sources:

[0049] 3.1、Dynamic addition;

[0050] 3.2, Dynamic deletion;

[0051] S4: Data cleaning:

[0052] 4.1. Deduplication processing;

[0053] 4.2. Error handling;

[0054] 4.3, Standardization processing;

[0055] S5: Data conversion:

[0056] 5.1, Structural transformation;

[0057] 5.2 Data mapping;

[0058] S6: Data loading:

[0059] 6.1、Data storage;

[0060] 6.2、Data loading;

[0061] 6.3, Indexing and optimization;

[0062] S7: Data preprocessing:

[0063] 7.1. Deep data cleaning;

[0064] 7.2. Complex data conversion;

[0065] S8: Data Analysis and Mining:

[0066] 8.1. In-depth descriptive statistical analysis;

[0067] 8.2. Refined exploratory data analysis;

[0068] 8.3. Advanced prediction and classification;

[0069] 8.4, text and image analysis;

[0070] S9: Customized data processing and analysis;

[0071] S10: Automation and intelligence;

[0072] S11: Construction of personalized learning model:

[0073] 11.1. Data collection and preprocessing;

[0074] 11.2. Feature Engineering;

[0075] 11.3. Model selection and training;

[0076] S12: Learning Path Planning:

[0077] 12.1. Learning goal setting;

[0078] 12.2. Assessment of current learning status;

[0079] 12.3. Plan your learning path;

[0080] 12.4 Dynamic adjustment and optimization;

[0081] S13: Visualization display and customized report generation;

[0082] S14: Permission management and security audit.

[0083] The present invention also discloses an electronic device, comprising:

[0084] at least one processor;

[0085] a memory communicatively coupled to the at least one processor;

[0086] The memory stores a computer program that can be executed by at least one processor, and the computer program is executed by at least one processor so that the at least one processor can execute the above-mentioned method for in-depth analysis of the full-data resource education information platform based on the fusion of structured and unstructured data.

[0087] The present invention also discloses a computer-readable storage medium storing a computer program, characterized in that when the computer program is executed by a processor, it implements the above-mentioned method for deep analysis of a full-data resource education information platform based on the fusion of structured and unstructured data.

[0088] Compared with the prior art, the present invention has the following beneficial effects:

[0089] The present invention discloses a system and method for deep analysis of a full-data resource education information platform based on the fusion of structured and unstructured data. Various types of education data are widely collected through a data acquisition and integration module. The original data are cleaned and converted by a data processing and analysis module to ensure the accuracy and availability of the data. The structured data and the unstructured data are seamlessly integrated through a personalized recommendation and learning path planning module to form a full-data resource pool. Advanced technologies such as machine learning and natural language processing are used to deeply mine and analyze the full-data resource pool to extract valuable information and patterns, such as students' learning habits, interest preferences, knowledge mastery, etc. Based on the results of intelligent analysis, the system can provide personalized education resource recommendations, learning path planning, teaching decision support and other services for different user groups (students, teachers, education managers). An intuitive and easy-to-use visualization interface is realized through a visualization display and customized report generation module. Users can easily view analysis results, receive personalized recommendations, and participate in the interaction and feedback of the platform to form a closed-loop education resource optimization and service improvement mechanism. DETAILED DESCRIPTION

[0090] The present invention is described in detail below so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more definite definition of the protection scope of the present invention.

[0091] A brief summary of one or more aspects is given below to provide a basic understanding of these aspects. This summary is not an exhaustive overview of all conceived aspects, and is neither intended to identify the key or critical elements of all aspects nor to define the scope of any or all aspects. Its only purpose is to give some concepts of one or more aspects in a simplified form as a prelude to a more detailed description that will be given later.

[0092] The present invention discloses a full data resource education information platform in-depth analysis system based on the fusion of structured and unstructured data, comprising:

[0093] The data collection and integration module is responsible for collecting data from multiple data sources and integrating them. By configuring data collection tasks and data integration rules, the module can automatically complete operations such as data capture, cleaning, conversion and loading. It also supports the dynamic addition and deletion of data sources, making it convenient for users to adjust the scope of data collection and integration according to actual needs;

[0094] The data processing and analysis module is responsible for in-depth processing and analysis of the integrated data. By providing a variety of data processing and analysis tools and technical support, this module can help users quickly extract valuable information and knowledge. It also supports the development and deployment of custom data processing rules and algorithms to meet the personalized needs of different users.

[0095] Personalized recommendation and learning path planning module, which builds a personalized learning model based on students' learning behavior data and performance data, and recommends learning resources and paths that meet students' interests and needs through collaborative filtering algorithms and content-based recommendation algorithms. This module also supports the learning path planning function to plan a reasonable learning path for students based on their learning goals and current learning status, and dynamically adjusts it to ensure that students achieve their learning goals in the shortest time;

[0096] Visualization display and customized report generation module. This module is an important bridge for the interaction between the platform and users. It integrates a variety of visualization tools, such as data dashboards, charts, maps, etc., so that users can intuitively understand complex data analysis results. Users can customize report templates according to their needs and quickly generate reports containing key indicators, trend analysis, and comparative analysis. These reports not only help education managers make more informed decisions, but also facilitate teachers to evaluate teaching effectiveness and adjust teaching strategies;

[0097] Permission management and security audit module. Given the sensitivity and privacy of educational data, the system pays special attention to data security and privacy protection. This module ensures that only authorized users can access corresponding data resources through refined role division and permission allocation. This module also records user operation behaviors, warns and tracks abnormal operations, and ensures data security and integrity.

[0098] In the present invention, the data collection and integration module includes:

[0099] Data source identification and configuration module, responsible for completing data source identification and data source configuration;

[0100] The data capture module is responsible for the scheduling and management of data collection tasks, data request and data reception;

[0101] The data source dynamic addition and deletion module is responsible for completing the dynamic addition and deletion of data sources;

[0102] The data cleaning module is responsible for completing data deduplication, error handling, and standardization processing;

[0103] Data conversion module, responsible for completing the structured conversion and mapping of data;

[0104] The data loading module is responsible for completing data storage, loading, indexing and optimization.

[0105] Data processing and analysis modules include:

[0106] Data preprocessing module, responsible for deep data cleaning and complex data conversion;

[0107] The data analysis and mining module is responsible for implementing in-depth descriptive statistical analysis of data, refined exploratory data analysis, advanced prediction and classification, and text and image analysis;

[0108] Customized data processing and analysis modules are highly configurable and provide a wealth of parameter settings and configuration options, allowing users to adjust data processing and analysis processes according to specific needs;

[0109] Intelligent recommendation module, responsible for intelligently recommending appropriate data processing methods and analysis models based on user historical behavior and data analysis results;

[0110] The monitoring module is responsible for automating and monitoring the data processing and analysis processes;

[0111] The optimization module is responsible for using machine learning technology to continuously optimize and improve the data processing and analysis processes to improve overall efficiency and effectiveness.

[0112] The personalized recommendation and learning path planning modules include:

[0113] The personalized learning model building module is responsible for building a personalized learning model based on students’ learning behavior data and performance data;

[0114] The learning path planning module is responsible for recommending learning resources and paths that meet students' interests and needs through collaborative filtering algorithms and content-based recommendation algorithms. It also supports the learning path planning function to plan reasonable learning paths for students based on their learning goals and current learning status, and dynamically adjust them to ensure that students achieve their learning goals in the shortest time. The accuracy and effectiveness of learning path planning are improved through continuous data analysis and model optimization. Machine learning technology is used to continuously learn and adapt to students' learning habits and changes in needs, providing students with more personalized learning support.

[0115] The visualization and customized report generation modules include:

[0116] The real-time data dashboard supports real-time data updates, allowing users to freely configure the displayed information according to their needs, including student participation, grade fluctuations, resource access, etc., and presents them in the form of intuitive charts; whether it is a bar chart, line chart or pie chart, it can help users quickly identify trends and anomalies in the data;

[0117] Chart display module: In order to analyze data more deeply, this module provides a variety of chart types, each of which is carefully designed to best display a specific type of data. Users can switch between charts or overlay multiple data layers to discover deeper connections by simply dragging and dropping. In addition, the chart also supports rich interactive functions, such as hovering over data points to display detailed information, clicking on the chart area to filter or explore in depth, etc., which greatly enhances the user experience;

[0118] The geospatial visualization module is responsible for realizing the visualization of data with geographic location attributes. For data with geographic location attributes, this module introduces geospatial visualization functions. Users can intuitively see information such as student distribution and campus facility layout on the map, and use visual elements such as color and size to reflect data characteristics such as density and intensity. This method not only helps to discover spatial laws and trends, but also provides strong support for the optimal allocation of educational resources;

[0119] The report module provides monthly summaries, teaching evaluations, student performance analysis, etc. Users can select appropriate templates based on actual conditions and adjust fields and styles through simple editing operations to quickly generate reports that meet requirements. In addition, users can also customize report templates and save them as their own exclusive styles for easy reuse in the future;

[0120] Report generation module, which has a built-in intelligent report generator. Users only need to set the basic parameters of the report, such as time range, data source, etc., and the system can automatically analyze the data and generate reports containing key indicators, trend analysis and comparative analysis. This one-click operation greatly simplifies the report generation process and improves work efficiency. At the same time, the generated reports also support export in multiple formats (such as PDF, Excel, PPT, etc.), which is convenient for users to use in different occasions;

[0121] The collaboration module supports the function of multi-person collaborative report editing. Users can invite team members to participate in the preparation and revision of reports. During the collaboration process, the system will save the change records of each version in real time to ensure the accuracy and traceability of the data. In addition, users can also share the generated reports with relevant personnel for review and discussion through emails, links, etc., to promote information exchange and sharing;

[0122] The permission management and security audit modules include:

[0123] Permission management module: A highly refined role division strategy is adopted, which classifies users into different role groups according to their responsibilities and needs in the platform, such as administrators, teachers, students, property management, third-party service providers, etc. Each role group is assigned a corresponding set of permissions, which strictly limit the scope of data resources that can be accessed and the types of operations that can be performed. Through this mechanism, we ensure that only users who have undergone strict identity authentication and authorization can access the data necessary for their work and study, effectively preventing the risk of unauthorized access and data leakage;

[0124] Dynamic permission management module: allows temporary adjustment of user permissions in specific situations to adapt to temporary changes in requirements that may occur in educational activities, while maintaining strict control and records of permission changes;

[0125] Security audit module: This module can record all user operations on the platform in real time, including but not limited to data access, modification, deletion and other operations, as well as system-level activities such as login and logout. By collecting, analyzing and storing these operation logs, we can achieve comprehensive monitoring and auditing of user behavior. Once abnormal operation behavior is detected, such as frequent failed login attempts, access to unauthorized data resources, etc., the system will immediately trigger the early warning mechanism, notify relevant personnel to conduct verification and processing, and automatically track the source and path of abnormal operations, providing strong support for subsequent security incident investigations;

[0126] Regular review and analysis module: Regularly review and analyze security audit logs to identify potential security risks and loopholes, and continuously optimize and improve the platform's security protection measures accordingly. This continuously improved security audit mechanism provides a solid guarantee for the security and integrity of educational data.

[0127] The present invention also discloses a method for deep analysis of a full data resource education information platform based on the fusion of structured and unstructured data, which uses the deep analysis system for a full data resource education information platform based on the fusion of structured and unstructured data as described above for analysis, and the method comprises the following steps:

[0128] S1: Data source identification and configuration:

[0129] 1.1、Data source identification;

[0130] 1.2、Data source configuration;

[0131] S2: Data capture:

[0132] 2.1、Task scheduling;

[0133] 2.2 Data request;

[0134] 2.3、Data reception;

[0135] S3: Dynamic addition and deletion of data sources:

[0136] 3.1、Dynamic addition;

[0137] 3.2, Dynamic deletion;

[0138] S4: Data cleaning:

[0139] 4.1. Deduplication processing;

[0140] 4.2. Error handling;

[0141] 4.3, Standardization processing;

[0142] S5: Data conversion:

[0143] 5.1, Structural transformation;

[0144] 5.2 Data mapping;

[0145] S6: Data loading:

[0146] 6.1、Data storage;

[0147] 6.2、Data loading;

[0148] 6.3, Indexing and optimization;

[0149] S7: Data preprocessing:

[0150] 7.1. Deep data cleaning;

[0151] 7.2. Complex data conversion;

[0152] S8: Data Analysis and Mining:

[0153] 8.1. In-depth descriptive statistical analysis;

[0154] 8.2. Refined exploratory data analysis;

[0155] 8.3. Advanced prediction and classification;

[0156] 8.4, text and image analysis;

[0157] S9: Customized data processing and analysis;

[0158] S10: Automation and intelligence;

[0159] S11: Construction of personalized learning model:

[0160] 11.1. Data collection and preprocessing;

[0161] 11.2. Feature Engineering;

[0162] 11.3. Model selection and training;

[0163] S12: Learning Path Planning:

[0164] 12.1. Learning goal setting;

[0165] 12.2. Assessment of current learning status;

[0166] 12.3. Plan your learning path;

[0167] 12.4 Dynamic adjustment and optimization;

[0168] S13: Visual display and customized report generation; S14: Permission management and security audit.

[0169] The present invention also discloses an electronic device, comprising:

[0170] at least one processor;

[0171] a memory communicatively coupled to the at least one processor;

[0172] The memory stores a computer program that can be executed by at least one processor, and the computer program is executed by at least one processor so that the at least one processor can execute the above-mentioned method for in-depth analysis of the full-data resource education information platform based on the fusion of structured and unstructured data.

[0173] The present invention also discloses a computer-readable storage medium storing a computer program, characterized in that when the computer program is executed by a processor, it implements the above-mentioned method for deep analysis of a full-data resource education information platform based on the fusion of structured and unstructured data.

[0174] The present invention is realized by the following core concepts:

[0175] 1. Data Collection and Integration Module

[0176] This module is responsible for collecting data from multiple data sources and integrating them. By configuring data collection tasks and data integration rules, this module can automatically complete operations such as data capture, cleaning, conversion, and loading. At the same time, this module also supports the dynamic addition and deletion of data sources, which makes it convenient for users to adjust the scope of data collection and integration according to actual needs. This module mainly implements the following functions:

[0177] S1: Data source identification and configuration

[0178] 1.1. Data source identification

[0179] The system has a built-in data source identification engine that can parse and identify various types of data source formats and protocols, such as JDBC (for relational databases), ODBC (open database connectivity), RESTful API (for Web services), FTP / SFTP (file transfer protocol), MinIO (object storage), etc.

[0180] The system also supports a custom data source plug-in mechanism, allowing users or third-party developers to extend new data source types.

[0181] 1.2 Data source configuration

[0182] Users configure data sources through the web interface or configuration files provided by the system. The interface usually contains multiple tabs or forms, which are used to enter connection parameters, set data ranges, select collection frequencies, etc.

[0183] For complex data sources, such as APIs that require multi-step authentication or have complex query capabilities, the system may provide a wizard-style configuration process to guide users through the configuration step by step.

[0184] After the configuration is complete, the system will perform a connection test on the data source to ensure that the configuration is correct and the data source is accessible.

[0185] S2: Data Capture

[0186] 2.1 Task Scheduling

[0187] The system uses a distributed task scheduling framework (such as Quartz, Apache Airflow, etc.) to support the scheduling and management of large-scale data collection tasks.

[0188] The scheduler automatically triggers data collection tasks according to the task plan set by the user (such as Cron expressions, time interval-based triggers).

[0189] The task scheduler also has fault tolerance and retry mechanisms to ensure automatic recovery when encountering network fluctuations, data source failures, etc. during the data collection process.

[0190] 2.2 Data Request

[0191] For API interface data sources, the system uses HTTP client libraries (such as Apache HttpClient, OkHttp, etc.) to build HTTP requests and process HTTP responses. The request may contain query parameters, request headers (such as authentication tokens), request bodies, etc.

[0192] For database data sources, the system uses database connection technologies such as JDBC or ODBC to build and execute SQL query statements. The system also supports advanced functions such as stored procedure calls and batch data processing;

[0193] For file data sources, the system uses file transfer protocols (such as FTP, SFTP) or file reading libraries (such as JavaNIO) or object storage servers (such as MinIO) to read file contents from a remote server or local file system.

[0194] 2.3 Data Reception

[0195] The system temporarily stores the received data in a buffer in memory or a temporary file on disk to avoid direct pressure on the data source during data processing.

[0196] The system also has a data integrity verification mechanism, such as checking the response status code, calculating data checksums, etc., to ensure that the received data is intact.

[0197] S3: Dynamic addition and deletion of data sources

[0198] 3.1. Dynamic addition

[0199] The user selects the "Add Data Source" option through the system interface and fills in the basic information of the data source (such as name, type, description) and detailed configuration (such as connection parameters, data range, etc.).

[0200] The system verifies the validity and integrity of the configuration information, including checking the legitimacy of the connection parameters and testing the accessibility of the data source.

[0201] After verification, the new data source is added to the system's data source management list and automatically synchronized to the task scheduler, becoming a candidate data source for the data collection task.

[0202] 3.2 Dynamic Deletion

[0203] The user selects the data source to be deleted through the system interface and confirms the deletion operation.

[0204] The system first checks whether the data source is being used by any data collection task. If it is, the system prompts the user to stop the relevant task or change the task configuration to remove the dependence on the data source.

[0205] If the data source is not in use, the system removes it from the data source management list and cleans up all configuration information, temporary data, and related records in the task scheduler related to the data source.

[0206] S4: Data cleaning

[0207] 4.1. Deduplication

[0208] The system uses data structures such as hash tables and Bloom filters to deduplicate the received data. For batch data, the system may sort the data first and then use algorithms such as double pointers or sliding windows to deduplicate.

[0209] For real-time data streams, the system may use a stream processing framework (such as Apache Kafka Streams, Apache Flink, etc.) for deduplication.

[0210] 4.2 Error Handling

[0211] The system defines a set of error handling rules, including null value processing (such as filling default values, deleting null value records), abnormal value processing (such as replacing with average value, median, etc.), format verification (such as date and time format, value range verification), etc.

[0212] During the data cleaning process, the system checks and processes the data one by one according to the rule set to ensure the accuracy and consistency of the data.

[0213] 4.3 Standardization

[0214] The system provides a data standardization toolset, including date and time formatting, text encoding conversion, and numerical unit unification.

[0215] Users can select the corresponding standardization tool according to actual needs and configure the corresponding parameters (such as date and time format template, encoding method, unit conversion factor, etc.).

[0216] During the data cleaning process, the system applies these standardized tools to process the data to ensure the consistency and comparability of the data.

[0217] S5: Data Conversion

[0218] 5.1. Structural transformation

[0219] For unstructured or semi-structured data (such as text, images, videos, etc.), the system uses NLP, OCR (optical character recognition), image recognition and other technologies to extract information and structure it.

[0220] The extracted key information (such as entity names, relationship attributes, image tags, etc.) is converted into structured data formats (such as JSON, XML, etc.) for subsequent processing and analysis.

[0221] The system also supports a custom data conversion plug-in mechanism, allowing users or third-party developers to extend new data conversion functions.

[0222] 5.2 Data Mapping

[0223] The system provides a data mapping tool or visual interface that allows users to define the mapping relationship between fields in different data sources.

[0224] The mapping relationship can be one-to-one, one-to-many, or many-to-one, etc. The system supports rule-based mapping (such as regular expression matching, conditional expression judgment) and learning-based mapping (such as machine learning algorithms automatically discovering mapping relationships).

[0225] During the data integration process, the system associates and integrates data fields from different data sources based on mapping relationships to form a unified data view.

[0226] S6: Data loading

[0227] 6.1 Data Storage

[0228] The system supports a variety of data storage media and formats, including relational databases (such as MySQL, PostgreSQL, etc.), non-relational databases (such as MongoDB, Cassandra, etc.), data warehouses (such as Hive, Spark), etc.

[0229] Users can choose a suitable data storage solution according to actual needs and configure corresponding storage parameters (such as database address, port, user name, password, partition strategy, index strategy, etc.).

[0230] 6.2 Data Loading

[0231] The system loads the cleaned and converted data into the data storage medium according to the specified format and rules. The loading process may involve optimization operations such as data compression, encryption, and partitioning.

[0232] The system supports batch loading and real-time streaming loading. Batch loading is usually used to process large-scale historical data; real-time streaming loading is suitable for processing real-time data streams.

[0233] The system also has a data consistency verification mechanism to ensure that the data loaded into the data storage medium is consistent with the original data.

[0234] 6.3 Indexing and Optimization

[0235] The system automatically or manually creates indexes (such as B-tree indexes, hash indexes, full-text indexes, etc.) based on the characteristics of the data storage medium and user query requirements to speed up data retrieval.

[0236] The system also provides optimization strategies such as data compression and partitioning to reduce storage space usage and improve query performance. These optimization strategies can be flexibly configured and adjusted according to user needs.

[0237] 2. Data Processing and Analysis Module

[0238] This module is responsible for in-depth processing and analysis of the integrated data. By providing a variety of data processing and analysis tools and technical support, this module can help users quickly extract valuable information and knowledge. At the same time, this module also supports the development and deployment of custom data processing rules and algorithms to meet the personalized needs of different users. This module mainly implements the following functions:

[0239] S7: Data preprocessing

[0240] 7.1. Deep Data Cleaning

[0241] Missing value strategy: Analyze the causes of missing values ​​in detail (such as random missing, completely random missing, and non-random missing), and use the most appropriate strategy to handle them, such as multiple imputation, model-based prediction filling, or filling according to business logic.

[0242] Outlier detection and correction: Use statistical methods (such as box plots, IQR rules) combined with business common sense to identify outliers, and use smoothing, binning, Winsorization and other methods to correct them, or treat them separately as special values.

[0243] Data denoising: For data sets containing noise, such as sensor data, filtering techniques (such as moving average and median filtering) are used to remove noise and improve data quality.

[0244] Data consistency check: Ensure the logical consistency and business rationality of data through consistency check across fields, tables or data sources.

[0245] 7.2 Complex Data Conversion

[0246] Data coding and classification: For categorical data, more sophisticated coding strategies (such as ordered coding and hierarchical coding) are adopted to retain the inherent structure and information of the data.

[0247] Feature engineering: Enhance the expressiveness of data and improve the subsequent analysis results through methods such as feature selection (such as filtering, packaging, and embedding), feature extraction (such as PCA and LDA), and feature construction (such as polynomial features and interactive features).

[0248] Data aggregation and stratification: Based on business needs, perform multi-level and multi-dimensional aggregation analysis on data, such as seasonal decomposition of time series data and hierarchical clustering of user behavior data.

[0249] S8: Data Analysis and Mining

[0250] 8.1. In-depth descriptive statistical analysis

[0251] In addition to basic statistical calculations, it also involves data distribution fitting (such as normal distribution, Poisson distribution test), data correlation analysis (such as Pearson correlation coefficient, Spearman rank correlation coefficient), data variability analysis (such as coefficient of variation), etc., in order to fully understand the characteristics of the data.

[0252] 8.2. Refined Exploratory Data Analysis (EDA)

[0253] Use advanced visualization techniques (such as interactive dashboards, dynamic charts, and 3D visualization) to present the complex structure and internal relationships of data.

[0254] Introduce methods such as hypothesis testing and variance analysis to conduct in-depth exploration of the data and discover hidden patterns and trends in the data.

[0255] 8.3 Advanced Prediction and Classification

[0256] Advanced algorithms such as ensemble learning methods (such as random forests, gradient boosting trees), deep learning models (such as convolutional neural networks (CNN), recurrent neural networks (RNN), and long short-term memory networks (LSTM)) are used to improve the accuracy and robustness of prediction and classification.

[0257] Implement cross-validation, grid search, random search and other strategies to tune the model and ensure optimal model performance.

[0258] Use tools such as feature importance analysis and partial dependency plots to gain a deeper understanding of the model's working mechanism and key influencing factors.

[0259] 8.4 Text and Image Analysis

[0260] For text data, natural language processing (NLP) techniques, such as word embedding (Word2Vec, GloVe) and BERT model, are used to perform tasks such as sentiment analysis, topic modeling, and entity recognition.

[0261] For image data, computer vision techniques such as convolutional neural networks (CNN), target detection, and image segmentation are used to extract key information and features from the image.

[0262] S9: Customized data processing and analysis

[0263] Highly configurable: Provides a wide range of parameter settings and configuration options, allowing users to adjust data processing and analysis processes according to specific needs.

[0264] Low-code / no-code platform: Through a graphical interface and drag-and-drop operation, it lowers the technical threshold and enables non-technical personnel to participate in the data processing and analysis process.

[0265] Open API and scalability: Provides an open API interface to support users to integrate third-party tools and services to achieve seamless data connection and extended analysis functions.

[0266] S10: Automation and intelligence

[0267] Intelligent recommendation: Based on user historical behavior and data analysis results, intelligently recommend appropriate data processing methods and analysis models.

[0268] Process automation: Through the workflow engine and scheduling system, the data processing and analysis processes can be automatically executed and monitored.

[0269] Continuous learning and optimization: Use machine learning technology to continuously optimize and improve data processing and analysis processes to improve overall efficiency and effectiveness.

[0270] 3. Personalized recommendation and learning path planning module

[0271] This module builds a personalized learning model based on students' learning behavior data and performance data. It recommends learning resources and paths that meet students' interests and needs through collaborative filtering algorithms and content-based recommendation algorithms. At the same time, this module also supports the learning path planning function to plan a reasonable learning path for students based on their learning goals and current learning status, and dynamically adjust it to ensure that students achieve their learning goals in the shortest time. This module mainly implements the following functions:

[0272] S11: Construction of personalized learning model

[0273] 11.1 Data Collection and Preprocessing

[0274] Learning behavior data: collects students' various learning behavior data on the platform, including but not limited to learning time, learning frequency, learning progress, interaction (such as questions, answers, discussions), resource access records, etc. These data are obtained through log systems, learning management systems (LMS) and other channels.

[0275] Grade data: Obtain students' test scores, homework scores, project scores, etc. to quantitatively evaluate students' learning effects and mastery. These data usually come from the school's academic affairs system or the platform's built-in evaluation system.

[0276] Data preprocessing: Clean, denoise, and standardize the collected data to ensure data accuracy and consistency. At the same time, extract and transform the data for subsequent model training and analysis.

[0277] 11.2 Feature Engineering

[0278] User characteristics: Construct a feature set that reflects the individual characteristics of students, such as age, gender, professional background, learning style (such as visual, auditory, hands-on), etc. These characteristics may be obtained through questionnaires, user portraits, etc.

[0279] Learning behavior characteristics: Features are constructed based on learning behavior data, such as learning time distribution, learning progress curve, resource preferences (such as videos, documents, exercises), interactive activity, etc. These characteristics can reflect students' learning habits and learning status.

[0280] Grade characteristics: Convert grade data into characteristics, such as average score, grade change trend, subject strengths and weaknesses, etc. These characteristics are used to evaluate students' learning effects and potential needs.

[0281] 11.3 Model Selection and Training

[0282] Collaborative filtering algorithm: Use the similarity between students to make recommendations. First, calculate the similarity between students (such as based on cosine similarity, Pearson correlation coefficient, etc.), and then recommend learning resources to students based on the historical behavior of similar students. Collaborative filtering algorithms can capture potential connections between students and discover resources that students may be interested in but have not yet accessed.

[0283] Content-based recommendation algorithm: Match the content characteristics of learning resources (such as topics, keywords, and difficulty levels) with students' interest preferences. First, analyze students' learning history and behavior data to extract their interest characteristics; then use text mining, topic modeling and other technologies to analyze the content characteristics of learning resources; finally, recommend relevant resources to students through similarity calculation.

[0284] Hybrid recommendation model: Combining the advantages of collaborative filtering and content-based recommendation algorithms, a hybrid recommendation model is constructed. The hybrid method can be weighted fusion, switching hybrid, feature combination, etc. The hybrid recommendation model can more comprehensively consider students' personal characteristics and learning needs, and improve the accuracy and diversity of recommendations.

[0285] Model training and optimization: Use machine learning or deep learning technology to train the recommendation model. Adjust model parameters through cross-validation, grid search and other methods to optimize performance. At the same time, use experimental methods such as A / B testing to evaluate the model effect and continuously iterate and optimize.

[0286] S12: Learning Path Planning

[0287] 12.1. Learning Goal Setting

[0288] Support students to set learning goals independently, such as improving grades in a certain subject, mastering a certain skill, etc. At the same time, provide intelligent suggestions to help students set reasonable and specific learning goals.

[0289] 12.2. Current Learning Status Assessment

[0290] Evaluate students' current learning status based on their learning behavior data and performance data. Analyze students' learning progress, mastery level, weak links, etc.

[0291] 12.3 Learning Path Planning

[0292] Plan a reasonable learning path for students based on their learning goals and current learning status. The learning path includes a series of orderly learning resources and activities (such as video courses, exercises, mock exams, etc.) to help students improve their knowledge and skills in a systematic way.

[0293] The learning path planning algorithm considers multiple factors, such as resource difficulty, learning order, time allocation, etc. The optimization algorithm is used to find the optimal learning path to maximize the learning effect of students.

[0294] 12.4 Dynamic Adjustment and Optimization

[0295] Monitor students' learning progress and results in real time, and dynamically adjust the learning path based on feedback. For example, when students encounter learning difficulties, add tutoring resources or adjust the learning sequence; when students have a good grasp of the content, speed up the learning process or add challenging learning tasks.

[0296] Improve the accuracy and effectiveness of learning path planning through continuous data analysis and model optimization. Use machine learning technology to continuously learn and adapt to students' learning habits and changes in needs, and provide students with more personalized learning support.

[0297] 4. Visualization and customized report generation module

[0298] This module is an important bridge for the platform to interact with users. It integrates a variety of visualization tools, such as data dashboards, charts, maps, etc., so that users can intuitively understand complex data analysis results. Users can customize report templates according to their needs and quickly generate reports containing key indicators, trend analysis, and comparative analysis. These reports not only help education managers make more informed decisions, but also facilitate teachers to evaluate teaching effectiveness and adjust teaching strategies. This module mainly implements the following functions:

[0299] S13: Visualization and customized report generation

[0300] 13.1. Real-time data dashboard

[0301] The real-time data dashboard is like an educational "cockpit", allowing users to understand the status of key indicators at a glance. The dashboard not only supports real-time data updates, but also allows users to freely configure the displayed information according to their needs, including student participation, grade fluctuations, resource access, etc., and presents it in the form of intuitive charts. Whether it is a bar chart, line chart or pie chart, it can help users quickly identify trends and anomalies in the data.

[0302] 13.2. Diversification and Interactivity Charts

[0303] In order to analyze data more deeply, the module provides a variety of chart types, each of which is carefully designed to best display a specific type of data. Users can switch between charts or overlay multiple data layers to discover deeper connections by simply dragging and dropping. In addition, the chart also supports rich interactive functions, such as hovering over data points to display detailed information, clicking on the chart area to filter or explore in depth, etc., which greatly enhances the user experience.

[0304] 13.3 Geospatial Visualization

[0305] For data with geographic location attributes, the module introduces geospatial visualization. Users can intuitively see information such as student distribution and campus facility layout on the map, and use visual elements such as color and size to reflect data density, intensity and other characteristics. This approach not only helps to discover spatial laws and trends, but also provides strong support for the optimal allocation of educational resources.

[0306] 13.4. Flexible report templates

[0307] In order to meet the needs of different users, the module has built-in multiple report templates, including monthly summary, teaching evaluation, student performance analysis, etc. Users can choose the appropriate template according to the actual situation, and adjust the fields, styles, etc. through simple editing operations to quickly generate reports that meet the requirements. In addition, users can also customize report templates and save them as their own exclusive styles for future reuse.

[0308] 13.5. Intelligent Report Generator

[0309] The module has a built-in intelligent report generator. Users only need to set the basic parameters of the report (such as time range, data source, etc.), and the system can automatically analyze the data and generate reports containing key indicators, trend analysis and comparative analysis. This one-click operation greatly simplifies the report generation process and improves work efficiency. At the same time, the generated reports also support multiple formats of export (such as PDF, Excel, PPT, etc.), which is convenient for users to use in different occasions.

[0310] 13.6. Collaboration and Sharing

[0311] The module supports the function of multi-person collaborative report editing. Users can invite team members to participate in the preparation and revision of reports. During the collaborative process, the system will save the change records of each version in real time to ensure the accuracy and traceability of the data. In addition, users can also share the generated reports with relevant personnel for review and discussion through emails, links, etc., to promote the exchange and sharing of information.

[0312] 5. Permission Management and Security Audit Module

[0313] Given the sensitivity and privacy of educational data, the platform pays special attention to data security and privacy protection. The permission management module ensures that only authorized users can access the corresponding data resources through refined role division and permission allocation. At the same time, the security audit module records user operation behaviors, warns and tracks abnormal operations, and ensures the security and integrity of data. This module mainly implements the following functions:

[0314] S14: Permission Management and Security Audit

[0315] 14.1. The authority management module is the cornerstone of the entire data security system

[0316] We have adopted a highly refined role division strategy, classifying users into different role groups according to their responsibilities and needs in the platform, such as administrators, teachers, students, property management, third-party service providers, etc. Each role group is assigned a corresponding set of permissions, which strictly limit the scope of data resources that can be accessed and the types of operations that can be performed. Through this mechanism, we ensure that only users who have undergone strict identity authentication and authorization can access the data necessary for their work and study, effectively preventing the risk of unauthorized access and data leakage.

[0317] 14.2. Introduced dynamic permission management mechanism

[0318] It allows users to temporarily adjust their permissions under specific circumstances to accommodate temporary changes in demand that may occur in educational activities, while maintaining strict control and records of permission changes.

[0319] 14.3. Deployed a comprehensive security audit module

[0320] This module can record all user operations on the platform in real time, including but not limited to data access, modification, deletion and other operations, as well as system-level activities such as login and logout. By collecting, analyzing and storing these operation logs, we can achieve comprehensive monitoring and auditing of user behavior. Once abnormal operation behavior is detected, such as frequent failed login attempts, access to unauthorized data resources, etc., the system will immediately trigger the early warning mechanism, notify relevant personnel to verify and handle, and automatically track the source and path of abnormal operations, providing strong support for subsequent security incident investigations.

[0321] 14.4. Regular review and analysis

[0322] The security audit logs are reviewed and analyzed regularly to identify potential security risks and loopholes, and the security protection measures of the platform are continuously optimized and improved accordingly. This continuously improved security audit mechanism provides a solid guarantee for the security and integrity of educational data.

[0323] Compared with the prior art, the present invention has at least the following technical effects:

[0324] 1. Comprehensive integration of educational resources: By integrating structured and unstructured data, comprehensive integration and efficient use of educational resources can be achieved. The automated and efficient data collection and integration process significantly reduces the need for manual intervention and improves data processing efficiency. At the same time, it supports the function of dynamically adding and deleting data sources, allowing the platform to flexibly respond to changing data environments and meet data collection needs in different educational scenarios. By timely and accurately collecting and integrating educational data from multiple channels, a solid foundation is laid for subsequent data processing and analysis, which helps educational institutions to have a more comprehensive understanding of the current teaching situation and provide strong support for decision-making.

[0325] 2. In-depth data analysis: Using intelligent analysis engines, we can conduct in-depth mining of all data resources to discover hidden teaching rules and trends. We provide a variety of data processing and analysis tools, which not only simplifies the data processing process, but also improves the depth and breadth of data analysis. We support the development and deployment of custom data processing rules and algorithms, so that the platform can flexibly respond to complex and changing analysis needs and meet the personalized analysis requirements of different users. Through in-depth data processing and analysis, we can dig out valuable information and knowledge hidden behind the data, provide scientific decision-making basis for education managers, help teachers optimize teaching strategies and improve teaching quality, and also provide students with more accurate learning guidance.

[0326] 3. Personalized services and improved teaching efficiency and quality: Provide personalized educational resource recommendations and learning path planning to meet the differentiated needs of different users. Utilizing advanced recommendation algorithms and learning path planning technologies, accurate predictions of students’ learning needs and personalized recommendations are achieved. By comprehensively considering factors such as students’ learning behaviors and performance data, learning resources and learning paths are tailored for students, effectively improving the pertinence and efficiency of learning. The implementation of personalized recommendations and learning path planning functions can help stimulate students’ interest and motivation in learning, improve the accuracy of recommendation results and user satisfaction, and improve learning outcomes. At the same time, it also helps to optimize the allocation of educational resources, reduce the waste of ineffective learning time, and promote educational equity and quality.

[0327] 4. Visualization display and report generation: Provide teachers with accurate teaching decision support, help students achieve efficient learning, and improve teaching efficiency and quality overall. Through rich visualization tools and flexible report template design, complex data analysis results can be presented intuitively and clearly. Users can customize report content according to their own needs and quickly generate high-quality reports, which greatly improves the efficiency and effectiveness of data display and report generation. The application of visualization display and report generation modules enables education managers and teachers to understand and use data analysis results more conveniently, providing more powerful support for decision-making and teaching adjustments.

[0328] 5. Security and privacy protection: The authority management module refines the role division and authority allocation to achieve decentralization and domain division. At the same time, the security audit module records the user's operation behavior, warns and tracks abnormal operations, and ensures the security and integrity of the data. The implementation of the authority management and security audit modules effectively prevents the occurrence of security incidents such as data leakage and abuse, and ensures the confidentiality and integrity of educational data. This not only helps to maintain the reputation and interests of educational institutions, but also provides a safer and more reliable educational service environment for the majority of users.

[0329] This invention aims to promote the comprehensive integration and efficient use of educational resources through technological innovation, promote the development of personalized teaching and precision teaching, and contribute to the advancement of education.

[0330] Parts or structures not specifically described in the present invention may adopt existing technologies or existing products and will not be described in detail here.

[0331] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A deep analysis system for the full data resource education information platform based on the fusion of structured and unstructured data, characterized in that: include: Data collection and integration module, responsible for collecting data from multiple data sources and integrating them; Data processing and analysis module, responsible for in-depth processing and analysis of the integrated data; Personalized recommendation and learning path planning module; Visual display and customized report generation module; The authority management and security audit modules ensure the security and integrity of data.

2. According to claim 1, a deep analysis system for the full data resource education information platform based on the fusion of structured and unstructured data, characterized in that: The data collection and integration module includes: Data source identification and configuration module, responsible for completing data source identification and data source configuration; The data capture module is responsible for the scheduling and management of data collection tasks, data request and data reception; The data source dynamic addition and deletion module is responsible for completing the dynamic addition and deletion of data sources; The data cleaning module is responsible for completing data deduplication, error handling, and standardization processing; Data conversion module, responsible for completing the structured conversion and mapping of data; The data loading module is responsible for completing data storage, loading, indexing and optimization.

3. The in-depth analysis system of the full data resource education information platform based on the fusion of structured and unstructured data according to claim 1 is characterized in that: The data processing and analysis module includes: Data preprocessing module, responsible for deep data cleaning and complex data conversion; The data analysis and mining module is responsible for implementing in-depth descriptive statistical analysis of data, refined exploratory data analysis, advanced prediction and classification, and text and image analysis; Customized data processing and analysis modules; Intelligent recommendation module, responsible for intelligently recommending appropriate data processing methods and analysis models based on user historical behavior and data analysis results; The monitoring module is responsible for automating and monitoring the data processing and analysis processes; The optimization module is responsible for using machine learning technology to continuously optimize and improve data processing and analysis processes to improve overall efficiency and effectiveness.

4. The in-depth analysis system of the full data resource education information platform based on the fusion of structured and unstructured data according to claim 1 is characterized in that: The personalized recommendation and learning path planning module includes: The personalized learning model building module is responsible for building a personalized learning model based on students’ learning behavior data and performance data; The learning path planning module is responsible for recommending learning resources and paths that meet students' interests and needs through collaborative filtering algorithms and content-based recommendation algorithms. It also supports the learning path planning function to plan reasonable learning paths for students based on their learning goals and current learning status, and dynamically adjust them to ensure that students achieve their learning goals in the shortest time.

5. The in-depth analysis system of the full data resource education information platform based on the fusion of structured and unstructured data according to claim 1 is characterized in that: The visualization and customized report generation module includes: Real-time data dashboard, which supports real-time data updates, allows users to freely configure the displayed information according to their needs, and presents it in an intuitive chart form; Chart display module; The geospatial visualization module is responsible for visualizing data with geographic location attributes; Reporting module; Report generation module; Collaboration module.

6. The in-depth analysis system of the full data resource education information platform based on the fusion of structured and unstructured data according to claim 1 is characterized in that: The authority management and security audit module includes: Permission management module; Dynamic rights management module; Security audit module; Regular review and analysis of modules.

7. A method for deep analysis of the full data resource education information platform based on the fusion of structured and unstructured data, characterized in that: The method is performed by using a deep analysis system for an all-data resource education information platform based on the fusion of structured and unstructured data as described in any one of claims 1 to 6, and the method comprises the following steps: S1: Data source identification and configuration: 1.

1. Data source identification; 1.2、Data source configuration; S2: Data capture: 2.1、Task scheduling; 2.2 Data request; 2.3、Data reception; S3: Dynamic addition and deletion of data sources: 3.1、Dynamic addition; 3.2, Dynamic deletion; S4: Data cleaning: 4.

1. Deduplication processing; 4.

2. Error handling; 4.3, Standardization processing; S5: Data conversion: 5.1, Structural transformation; 5.2 Data mapping; S6: Data loading: 6.1、Data storage; 6.2、Data loading; 6.3, Indexing and optimization; S7: Data preprocessing: 7.

1. Deep data cleaning; 7.

2. Complex data conversion; S8: Data Analysis and Mining: 8.

1. In-depth descriptive statistical analysis; 8.

2. Refined exploratory data analysis; 8.

3. Advanced prediction and classification; 8.4, text and image analysis; S9: Customized data processing and analysis; S10: Automation and intelligence; S11: Construction of personalized learning model: 11.

1. Data collection and preprocessing; 11.

2. Feature Engineering; 11.

3. Model selection and training; S12: Learning Path Planning: 12.

1. Learning goal setting; 12.

2. Assessment of current learning status; 12.

3. Plan your learning path; 12.4 Dynamic adjustment and optimization; S13: Visualization display and customized report generation; S14: Permission management and security audit.

8. An electronic device, characterized in that: include: at least one processor; a memory communicatively coupled to the at least one processor; The memory stores a computer program that can be executed by at least one processor, and the computer program is executed by at least one processor so that the at least one processor can execute the method for in-depth analysis of the full-data resource education information platform based on the fusion of structured and unstructured data as described in claim 7.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by the processor, a method for deep analysis of an education information platform of all data resources based on the fusion of structured and unstructured data as described in claim 7 is implemented.