Intelligent prediction system based on statistical learning resource optimization
By designing a hybrid computing processing framework and a multi-data storage engine in an intelligent prediction system, the resource allocation conflicts and inefficiency caused by resource management differences between different computing frameworks are solved, and more efficient data storage and computing are achieved.
Patent Information
- Application Number
- CN202411866471.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-18
- Publication Date
- 2025-03-14
AI Technical Summary
In the existing intelligent prediction systems, the resource management systems supported by each computing framework are different, resulting in resource allocation conflicts and inefficiency.
An intelligent prediction system based on statistical learning resource optimization is designed, and a big data absorption module, data warehouse module, analysis and prediction algorithm model library and analysis and prediction module are used to realize unified resource management and collaborative operation of multiple computing frameworks through a hybrid computing processing framework.
It effectively avoids resource allocation conflicts, improves data storage and computing efficiency, and reduces redundant storage and resource recycling time.
Smart Images
Figure CN119621020A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of learning resource optimization, and in particular to an intelligent prediction system based on statistical learning resource optimization. Background Art
[0002] Online learning resources are the core elements of the "Internet" + education system. In recent years, open knowledge communities have attracted experts, researchers and enthusiasts in various knowledge fields to participate in the collaborative construction of knowledge communities with the characteristics of "co-construction, sharing, openness and interconnection"; typical open knowledge communities include Wikipedia, Google Knol, Github, Baidu Encyclopedia, Learning Meta Platform, Yuque Platform, etc. This type of community generates knowledge sharing documents in the collaborative editing of a large number of users, promoting the inheritance, circulation, creation and evolution of knowledge. However, due to the imbalance and complexity of the knowledge level of the user group, the open knowledge community also causes certain quality problems in the online learning resources generated by collaborative editing. The explosively growing online learning resources are of varying quality, which is not conducive to the creation and dissemination of knowledge.
[0003] Since various distributed computing frameworks are added to the intelligent prediction system to introduce distributed features into model training, data processing and other functions, multiple different types of computing processing frameworks will run simultaneously during the system operation. Each computing framework has its own supported resource management system and different resource allocation processes. This will lead to resource allocation conflicts when allocating resources between tasks running on different frameworks, resulting in anomalies and low efficiency in data storage and computing. Summary of the invention
[0004] The main purpose of the present invention is to provide an intelligent prediction system based on statistical learning resource optimization, which can effectively solve the problem that each computing framework in the background technology has its own supported resource management system and has different resource allocation processes. This will lead to resource allocation conflicts when allocating resources between tasks running on different frameworks, resulting in anomalies and low efficiency of data storage and calculation.
[0005] To achieve the above-mentioned purpose, the technical solution adopted by the present invention is as follows: the intelligent prediction system based on statistical learning resource optimization of the present invention includes a big data absorption module, a data warehouse module, an analysis and prediction algorithm model library and an analysis and prediction module; Big data absorption module: absorbs various types of data from external data sources, completes data preprocessing and data interaction functions for various types of data in external data sources, imports external data sources into the data warehouse module after preprocessing, and designs a hybrid computing processing framework to realize the function of multiple distributed computing frameworks running simultaneously; Data warehouse module: The data warehouse module is responsible for storing all types of data used by this system. It integrates and introduces data storage engines such as MongoDB, Redis, HDFS, and Alluxio. Multiple data storage engines work together to store in a distributed manner. Structured data suitable for database storage is stored in MongoDB; storage data supporting auxiliary functions such as operation monitoring is stored in Redis to achieve fast real-time reading; metadata, unstructured data, and big data files are stored in HDFS; data frequently used by various algorithm models is stored in Alluxio, and an in-memory storage engine is also built; Analysis and prediction algorithm model library module: provides users with an analysis and prediction algorithm model library containing multiple algorithm models in a pre-integrated manner, and encapsulates each algorithm model into a basic bundle package to form an algorithm model component. Each algorithm model component is associated in the form of a data stream to form data dependency. Multiple algorithm model components are connected in series to form the entire data analysis task. Analysis and prediction module: The analysis and prediction module provides a graphical interactive interface Stduio. Users can combine the algorithm models provided by the analysis and prediction algorithm model library module by clicking and selecting. For each selected algorithm model, the algorithm panel information is configured to finally form a process algorithm model for model training in the form of a data flow diagram, complete the construction of a custom model, and realize the prediction function.
[0006] Preferably, the big data absorption module provides a python usage interface, and performs data preprocessing on various types of data in the external data source through the python language in the python usage interface, and then transfers the processed data to the data warehouse module for persistent storage. If the authority permits, the data in the data warehouse module can also be exported to an external storage system. The big data absorption module also provides two computing engines, sparksql and mapreduce, and provides distributed data preprocessing functions through the distributed characteristics of the distributed computing processing framework spark and hadoop, thereby realizing the function of preprocessing various types of data in the external data source.
[0007] Preferably, the data interaction function in the big data absorption module is completed through the process architecture of external data source - cache - data exchange tool - data warehouse module, with redis as the cache storage. Through its memory storage characteristics, it provides a cache engine for high-speed data reading in data exchange. At the same time, socket programming is adopted to realize the function of reading redis data information, and the Sqoop tool is integrated to realize the exchange between the data warehouse module and the data source in a distributed manner, thereby realizing the function of data interaction.
[0008] Preferably, the hybrid computing processing framework is designed based on the Yarn resource management framework, and the TensorFlowonSpark open source project is introduced as the framework of the intermediate processing layer to realize the function of multiple distributed frameworks running simultaneously, and assist in data preprocessing and data interaction.
[0009] Preferably, some algorithm models in the analysis and prediction algorithm model library module implement distributed model training by combining a distributed computing framework.
[0010] Preferably, the analysis and prediction algorithm model library module includes two parts: DAG parsing and OSGI scheduling. DAG is the representation of the entire data analysis task, in which a node represents an algorithm model component, and the edge between two nodes represents the data dependency relationship between the two algorithm model components. After receiving the data analysis task, the analysis and prediction algorithm model library module will parse the corresponding DAG according to the topological sorting method, and schedule the operation of the parsed nodes one by one to obtain the OSGI services represented by the nodes.
[0011] Preferably, the analysis and prediction algorithm model library module also includes a task monitoring function, which is responsible for monitoring the task information provided to the hybrid processing framework for execution, including the running component task name, running component task ID, start running time, task end time, task duration and current running component task status.
[0012] Compared with the prior art, the present invention has the following beneficial effects: 1. In the present invention, by setting up multiple data storage engines in the data warehouse module, all data can be provided with data access services in the storage facility in the form of files, thereby reducing redundant storage and resource recovery time in frequent iterative calculations, and improving operating efficiency.
[0013] 2. In the present invention, through the setting of a hybrid computing framework, unified resource management of each computing framework can be achieved, providing unified resource allocation and scheduling for the big data absorption module, avoiding conflicts between resource allocations, and then supporting multiple computing frameworks to run simultaneously without conflict.
[0014] 3. In the present invention, Spark is used to store the intermediate results of the calculation process in memory, avoiding repeated reading and writing of HDFS, improving the efficiency of iterative calculation, and effectively reducing the load rate of the computing cluster used.
[0015] 4. In the present invention, by analyzing the setting of the prediction module, the data of all external data sources are utilized, thereby obtaining a global optimization model and protecting the data from being leaked.
[0016] 5. In the present invention, the system implements an algorithm model set based on a hybrid computing processing framework, which satisfies the requirement of realizing the algorithm required for the analysis task by combining a finite number of algorithm models, and each algorithm model will be constructed in a componentized form to achieve its high reuse characteristics, thereby improving the efficiency of data storage and computing. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Figure 1 is a flow chart of the present invention; Figure 2 It is an architectural diagram of the hybrid processing framework in the present invention; Figure 3 Flowchart for encapsulating basic bundle packages for each algorithm model in the present invention. DETAILED DESCRIPTION
[0018] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the present invention is further explained below in conjunction with specific implementation methods.
[0019] In the description of the present invention, it should be noted that the terms "upper", "lower", "inner", "outer", "front end", "rear end", "two ends", "one end", "the other end" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance.
[0020] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "provided with", "connected", etc. should be understood in a broad sense. For example, "connected" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
[0021] Please refer to Figure 1 As shown in FIG. 3 , this embodiment includes a big data absorption module, a data warehouse module, an analysis and prediction algorithm model library, and an analysis and prediction module; Big data absorption module: absorbs various types of data from external data sources, completes data preprocessing and data interaction functions for various types of data in external data sources, imports external data sources into the data warehouse module after preprocessing, and designs a hybrid computing processing framework to realize the function of multiple distributed computing frameworks running simultaneously; Through the setting of hybrid computing framework, unified resource management of each computing framework can be achieved, providing unified resource allocation and scheduling for the big data absorption module, avoiding conflicts between resource allocations, and thus supporting the simultaneous operation of multiple computing frameworks without conflicts; Data warehouse module: The data warehouse module is responsible for storing all types of data used by this system. It integrates and introduces data storage engines such as MongoDB, Redis, HDFS, and Alluxio. Multiple data storage engines work together to store in a distributed manner. Structured data suitable for database storage is stored in MongoDB; storage data supporting auxiliary functions such as operation monitoring is stored in Redis to achieve fast real-time reading; metadata, unstructured data, and big data files are stored in HDFS; data frequently used by various algorithm models is stored in Alluxio, and an in-memory storage engine is also built; By setting up multiple data storage engines in the data warehouse module, all data can be stored in the storage facility in the form of files to provide data access services, reducing redundant storage and resource recovery time in frequent iterative calculations, and improving operational efficiency; Redis is the most typical open source key-value NoSQL database. One of the most important features of Redis is that it runs in memory and stores data in memory. The high-speed access feature of memory ensures high efficiency of data processing. MongoDB is another document-based nosql database; Analysis and prediction algorithm model library module: provides users with an analysis and prediction algorithm model library containing multiple algorithm models in a pre-integrated manner, and encapsulates each algorithm model into a basic bundle package to form an algorithm model component. Each algorithm model component is associated in the form of a data stream to form data dependency. Multiple algorithm model components are connected in series to form the entire data analysis task. Analysis and prediction module: The analysis and prediction module provides a graphical interactive interface Stduio. Users can combine the algorithm models provided by the analysis and prediction algorithm model library module by clicking and selecting. For each selected algorithm model, the algorithm panel information is configured to finally form a process algorithm model for model training in the form of a data flow diagram, complete the construction of a custom model, and realize the prediction function.
[0022] In this way, all data from external data sources can be utilized to obtain a global optimization model and protect data from being leaked. After the custom model is built, it will be sent to the analysis and prediction algorithm model library module for storage; And based on the hybrid computing processing framework, an algorithm model set is implemented. This set satisfies the algorithm required for the analysis task by combining a limited number of algorithm models. Each algorithm model will be constructed in a componentized form to achieve its high reuse characteristics, thereby improving the efficiency of data storage and computing. The big data absorption module provides a Python usage interface, which uses the Python language in the Python usage interface to preprocess various types of data in the external data source, and then transfers the processed data to the data warehouse module for persistent storage. If permissions permit, the data in the data warehouse module can also be exported to the external storage system. The big data absorption module also provides two computing engines, sparksql and mapreduce. Through the distributed characteristics of the distributed computing processing framework spark and hadoop, it provides distributed data preprocessing functions, thereby realizing the functions of preprocessing various types of data in external data sources.
[0023] Hadoop has the ability to distribute massive amounts of data to different machine nodes in a server cluster for rapid processing. Hadoop includes several excellent open source products, including HDFS, MapReduce, and Yarn. HDFS is the file storage system in the Hadoop architecture, responsible for storing massive amounts of data, while the data computing part of Hadoop is MapReduce, which distributes the entire computing logic to multiple computing machine nodes for data computing and works with HDFS, making great use of the high parallel computing capabilities brought by the distributed system. Yarn is responsible for unified resources, job scheduling, and job monitoring in the Hadoop architecture, and has the highest control over all resources in the architecture. MapReduce is a distributed batch processing computing framework with excellent interactivity for big data computing scenarios; Spark uses memory storage to store the intermediate results of the calculation process, avoiding repeated reading and writing of HDFS, improving the efficiency of iterative calculations, and effectively reducing the load rate of the computing cluster used.
[0024] The data interaction function in the big data absorption module is completed through the process architecture of external data source - cache - data exchange tool - data warehouse module. Redis is used as the cache storage. Through its memory storage characteristics, it provides a cache engine for high-speed data reading in data exchange. At the same time, socket programming is adopted to realize the function of reading redis data information, and the Sqoop tool is integrated to realize the exchange between the data warehouse module and the data source in a distributed manner, thereby realizing the data interaction function.
[0025] The hybrid computing processing framework is designed based on the Yarn resource management framework. At the same time, the TensorFlowonSpark open source project is introduced as the framework of the intermediate processing layer to realize the function of multiple distributed frameworks running simultaneously and assist in data preprocessing and data interaction.
[0026] The TensorFlow on Spark open source project supports various heterogeneous systems, including multiple CPUs / GPUs, servers, and mobile devices, and has good cross-system features. Some algorithm models in the analysis and prediction algorithm model library module implement distributed model training by combining with the distributed computing framework.
[0027] The analysis and prediction algorithm model library module includes two parts: DAG parsing and OSGI scheduling. DAG is the representation of the entire data analysis task. A node represents an algorithm model component, and the edge between two nodes represents the data dependency relationship between the two algorithm model components. After receiving the data analysis task, the analysis and prediction algorithm model library module will parse the corresponding DAG according to the topological sorting method, and schedule the OSGI services represented by the parsed nodes one by one.
[0028] The design of OSGI specification has many advantages. The first one is hot-swappable feature, that is, without affecting the operation of OSGI service system, developers can dynamically install, uninstall, start and stop any module unit of the application deployed in OSGI service system. The second advantage is the powerful version management function. For any module unit in the deployed application, OSGI service system supports running different versions of the same module unit at the same time. The analysis and prediction algorithm model library module also includes a task monitoring function, which is responsible for monitoring the task information provided to the hybrid processing framework for execution, including the running component task name, running component task ID, start running time, task end time, task duration and current running component task status.
[0029] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.
Claims
1. An intelligent prediction system based on statistical learning resource optimization, characterized by: It includes big data absorption module, data warehouse module, analysis and prediction algorithm model library and analysis and prediction module; Big data absorption module: absorbs various types of data from external data sources, completes data preprocessing and data interaction functions for various types of data in external data sources, imports external data sources into the data warehouse module after preprocessing, and designs a hybrid computing processing framework to realize the function of multiple distributed computing frameworks running simultaneously; Data warehouse module: The data warehouse module is responsible for storing all types of data used by this system. It integrates and introduces data storage engines such as MongoDB, Redis, HDFS, and Alluxio. Multiple data storage engines work together to store structured data suitable for database storage in MongoDB. The storage data supporting auxiliary functions such as operation monitoring is stored in Redis to achieve fast real-time reading; metadata, unstructured data, and big data files are stored in HDFS; data frequently used by various algorithm models is stored in Alluxio, and an in-memory storage engine is also built; Analysis and prediction algorithm model library module: provides users with an analysis and prediction algorithm model library containing multiple algorithm models in a pre-integrated manner, and encapsulates each algorithm model into a basic bundle package to form an algorithm model component. Each algorithm model component is associated in the form of a data stream to form data dependency. Multiple algorithm model components are connected in series to form the entire data analysis task. Analysis and prediction module: The analysis and prediction module provides a graphical interactive interface Stduio. Users can combine the algorithm models provided by the analysis and prediction algorithm model library module by clicking and selecting. For each selected algorithm model, the algorithm panel information is configured to finally form a process algorithm model for model training in the form of a data flow diagram, complete the construction of a custom model, and realize the prediction function.
2. The intelligent prediction system based on statistical learning resource optimization according to claim 1, characterized in that: The big data absorption module provides a python usage interface, and uses the python language in the python usage interface to perform data preprocessing on various types of data in the external data source, and then transfers the processed data to the data warehouse module for persistent storage. If the authority permits, the data in the data warehouse module can also be exported to the external storage system. The big data absorption module also provides two computing engines, sparksql and mapreduce. Through the distributed characteristics of the distributed computing processing framework spark and hadoop, it provides distributed data preprocessing functions, thereby realizing the functions of preprocessing various types of data in the external data source.
3. The intelligent prediction system based on statistical learning resource optimization according to claim 2, characterized in that: The data interaction function in the big data absorption module is completed through the process architecture of external data source - cache - data exchange tool - data warehouse module. Redis is used as the cache storage. Through its memory storage characteristics, it provides a cache engine for high-speed data reading in data exchange. At the same time, socket programming is adopted to realize the function of reading redis data information, and the Sqoop tool is integrated to realize the exchange between the data warehouse module and the data source in a distributed manner, thereby realizing the data interaction function.
4. The intelligent prediction system based on statistical learning resource optimization according to claim 3 is characterized in that: The hybrid computing processing framework is designed based on the Yarn resource management framework, and introduces the TensorFlow on Spark open source project as the framework of the intermediate processing layer to realize the function of multiple distributed frameworks running simultaneously, and assist in data preprocessing and data interaction.
5. The intelligent prediction system based on statistical learning resource optimization according to claim 4 is characterized in that: Some algorithm models in the analysis and prediction algorithm model library module implement distributed model training by combining a distributed computing framework.
6. The intelligent prediction system based on statistical learning resource optimization according to claim 5, characterized in that: The analysis and prediction algorithm model library module includes two parts: DAG parsing and OSGI scheduling. DAG is the representation of the entire data analysis task, in which a node represents an algorithm model component, and the edge between two nodes represents the data dependency relationship between the two algorithm model components. After receiving the data analysis task, the analysis and prediction algorithm model library module will parse the corresponding DAG according to the topological sorting method, and schedule the operation of the parsed nodes one by one to obtain the OSGI services represented by the nodes.
7. The intelligent prediction system based on statistical learning resource optimization according to claim 6, characterized in that: The analysis and prediction algorithm model library module also includes a task monitoring function, which is responsible for monitoring the task information provided to the hybrid processing framework for execution, including the running component task name, running component task ID, start running time, task end time, task duration and current running component task status.