A campus big data comprehensive service system

Abstract

The embodiment of the present application discloses a campus big data comprehensive service system, which belongs to the field of campus data service technology, including a user management module for collecting campus teacher and student accounts, user management and portrait analysis, collecting teacher and student data, establishing a distributed computing framework, and performing data analysis and distributed processing on teachers and students. The business service module modularizes campus business applications to obtain multiple independent business project applications, divides functions and defines boundaries for each business project application, clarifies data communication and dependency relationships between each module, and the component deployment module performs environmental configuration, component installation, resource deployment and interface configuration on components in the system, installs and configures an operating system for each component, and installs and configures corresponding application programs and services according to the functions of each component. The embodiment of the present application can better manage and utilize big data and help teacher and student users get better services from big data.

Description

Technical Field

[0001] The present application belongs to the field of campus data service technology, and specifically, relates to a campus big data comprehensive service system. Background Art

[0002] The wave of cloud computing and big data technology has brought tremendous changes to all walks of life, and has also brought opportunities and changes to school education life. In the context of the digital age, campus management also needs to move towards big data. Digital campuses are becoming a construction focus in the information age.

[0003] In the existing campus informatization construction process, each information system is independent, and the data between various service function modules cannot be integrated and unified. Most of them adopt a separate management method. The management and control method not only increases the difficulty of campus service data management, but also makes it difficult to conduct unified analysis of the big data in the system, making it inconvenient for campus teachers and students to obtain campus information, and campus teachers and students cannot get better data services. Summary of the invention

[0004] In order to solve the above problems and technical defects, the embodiment of the present application adopts the following technical solution, a campus big data comprehensive service system, including:

[0005] User management module, used for collecting campus teacher and student accounts, user management and portrait analysis;

[0006] Establish data pipelines, design data pipelines to collect teacher and student data, establish a distributed computing framework, use data processing and analysis engines to perform data analysis and distributed processing on teachers and students, and then establish data cleaning strategies to check data quality;

[0007] The business service module is used to modularize campus business applications to obtain multiple independent business project applications, and to manage these independent business project applications in a unified and distributed manner;

[0008] Perform functional division and boundary definition for each business project application, define the interface and boundary of each module, and clarify the data communication and dependency relationship between each module;

[0009] Component deployment module, used to perform environment configuration, component installation, resource deployment and interface configuration for components within the system;

[0010] Install and configure the operating system for each component, update the component's drivers and firmware, configure the network settings of each component, and install and configure the corresponding applications and services based on the functions of each component.

[0011] Preferably, the establishment of a distributed computing framework and the use of a data processing and analysis engine to perform data analysis and distributed processing on teachers and students is to first establish a Kafka message queue, collect teacher and student data from a data source in real time, use an Apache Flink stream processing engine to process and analyze the real-time data, perform data cleaning on the data, and calculate data indicators of the data in real time;

[0012] Then establish a data lake, and then regularly import teacher and student data into the data lake, use the MapReduce batch processing engine to analyze and process the data sets in the data lake, use TensorFlow technology for data mining and machine learning, implement aggregation analysis and association analysis, identify correlations and associations in the data set, and discover hidden data patterns and trends in teacher and student data.

[0013] Furthermore, the process of establishing a data lake is as follows:

[0014] Select the Apache Hadoop ecosystem as the data lake technology stack;

[0015] Redesign the data lake architecture and standardize data storage, data partitioning strategies, and data naming conventions;

[0016] Establish access control policies and authorize access users and operation users;

[0017] Establish data collection and integration strategies to regulate and restrict ETL processes and real-time data stream processing;

[0018] Perform metadata management, including data description information, data source and processing history, and data version, to ensure data accuracy, completeness, and consistency;

[0019] Use data warehouse connectivity tools and the Apache Spark analytics platform to query and analyze data;

[0020] Establish a monitoring mechanism, implement data lake maintenance strategies, and perform data backup, data recovery, capacity planning, and performance optimization.

[0021] Preferably, the specific steps of modularizing the campus business application are as follows:

[0022] According to the system functions and business requirements of the service system, the campus business application is modularly split into different functional modules to obtain multiple independent business application modules;

[0023] Formulate module specifications to ensure the consistency of each module during development and maintenance, define clear interfaces for each module, and reduce coupling between modules;

[0024] Establish a modular architecture and adopt the single responsibility principle to make each module responsible for only one clear function or responsibility to avoid overlapping functions. Use dependency injection and reverse dependency technology to manage the dependencies between modules to avoid module circular dependencies and excessive coupling. Use Kafka message queue communication to establish data communication between modules to ensure loose coupling between modules.

[0025] Perform separate unit tests for each module to verify the correctness of its functions and interfaces, and then perform module integration tests to verify the correctness and stability of the modules' collaborative work.

[0026] Furthermore, the steps for unified management of business project applications are as follows:

[0027] Clarify the organizational structure and naming conventions of each business module, formulate branch management strategies and version control strategies, define version number rules and release cycles, and use the Git version control system to manage the change history of each module;

[0028] Develop a dependency management strategy, manage and review the dependencies of each module, make the dependencies clear and reasonable, and use dependency management tools to automatically resolve and install dependencies;

[0029] Use integrated automated testing and static code analysis tools to implement automated continuous integration and continuous deployment processes, configure automated build tools, deploy, define and automatically trigger build processes, manage deployment methods and configuration files of different modules in a unified manner, and use Docker containerization technology to achieve unified deployment and environmental isolation;

[0030] Use Puppet configuration management tools to manage and automate the deployment and maintenance of each module environment, write repeatable and scalable configuration scripts, and ensure the consistency and manageability of each module environment;

[0031] Set monitoring alarm rules, configure a unified monitoring system, monitor the operating status, performance indicators and resource consumption of each module, respond to and resolve module performance degradation or abnormalities in a timely manner, use performance testing tools to simulate high-load scenarios, evaluate the performance of each module under different loads, and perform performance evaluation and optimization based on the performance results;

[0032] Conduct periodic security audits, assess the security and vulnerability risks of each module, regularly update related dependencies and libraries, design and implement unified authentication and authorization strategies, and ensure that each module can only access restricted resources and data.

[0033] Furthermore, the steps of distributing and managing the business project applications are as follows:

[0034] Use the rules of high cohesion and low coupling to clarify the functional boundaries of each module. Each module solves a specific problem or implements a specific function.

[0035] Define the interfaces and communication methods between standard modules, use interface specification protocols to ensure the consistency of interaction between modules, and use standardized communication protocols and data formats to enable integration and communication between modules;

[0036] Use semantic versioning to manage the version of each module, clarify the meaning of the version number and the type of change, and use dependency management tools to manage and resolve dependencies between modules so that each module can obtain and use the corresponding dependencies;

[0037] Use Docker containerization technology to achieve modular deployment and management, design and deploy independent units for each module, update and expand individual modules as needed, and use load balancing and automatic expansion technology to dynamically adjust resource allocation according to the load and performance requirements of each module;

[0038] Carry out separate fault-tolerant design for each module and establish a fault-tolerant mechanism;

[0039] Use the Grafana performance monitoring tool to monitor and adjust the performance of the modules, perform performance analysis and optimization on each module, and make each module respond quickly under various load conditions.

[0040] Preferably, the steps of configuring the interfaces of the components in the system are as follows:

[0041] According to the functional requirements of each component, clarify the function and purpose of each component interface. The components include campus network equipment, campus servers and campus data storage equipment;

[0042] Configure network devices, servers and storage devices in the service system and configure their interface parameters;

[0043] After configuration is complete, perform functional testing and verification on each component and interface;

[0044] Conduct security audits, restrict access to interfaces, and record and back up the configuration information of each component interface.

[0045] Furthermore, the configuration of the campus network equipment includes:

[0046] Use an SSH client to connect to the component's IP address or domain name and enter the configuration interface for configuration.

[0047] Connect to the component's IP address or domain name through Telnet and enter the component's command line interface for configuration;

[0048] Configure the component using an HTTPS connection in the web management interface by entering the component's IP address or domain name;

[0049] Connect to the console of a component through a serial connection to configure initial network settings or for troubleshooting.

[0050] Furthermore, the configuration of the campus server includes:

[0051] Use the iLO server remote management card to connect to the management card's IP address over the network to access the server console for remote server control and management;

[0052] Configure through the local operating system, BIOS or UEFI setup interface;

[0053] Remotely access, manage and configure the server's operating system and network settings through the SSH protocol.

[0054] Furthermore, the configuration of the campus data storage device includes:

[0055] Manage storage pools, storage volumes, and RAID settings by entering the component's IP address or domain name in a browser.

[0056] Connect to the storage device's CLI through SSH or Telnet and use the CLI to perform configuration.

[0057] Compared with the prior art, the beneficial effects of the embodiments of the present application are:

[0058] (1) This application designs key components and functional modules for effective data processing and analysis. The campus big data comprehensive service system can help organizations extract valuable information and insights from massive data to support business decision-making and operation optimization. The campus big data comprehensive service system can effectively support data management, processing, analysis and visualization needs. At the same time, a data lake is established. The data lake provides a highly flexible and scalable data storage and processing solution, which can better manage and utilize big data and help teachers and students get better services from big data.

[0059] (2) This application splits the service system into modules and then adopts unified management and distributed management strategies, so that the modular system can achieve efficient development, integration and management while maintaining flexibility and maintainability. Unified management improves the stability and security of the entire system and accelerates development efficiency. Distributed management also supports continuous development and deployment processes, improving the maintainability and scalability of the system.

[0060] (3) When deploying components within the service system, the present application configures the interfaces of various components to ensure that the system can run stably and meet business needs after deployment. According to the specific component type and environmental requirements, the most appropriate method is selected to enter the configuration interface or console of the device to complete the interface configuration and management tasks, so that each component can effectively support the functional and performance requirements of the system, can run stably and reliably, and is easy to maintain and manage. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In the attached picture:

[0062] Figure 1 It is a schematic diagram of the structure of an embodiment of the present application. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Generally, the components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0064] Example

[0065] like Figure 1 As shown, a campus big data comprehensive service system includes:

[0066] The user management module is used to collect campus teacher and student accounts, user management and portrait analysis, establish data pipelines, design data pipelines to collect teacher and student data, establish a distributed computing framework, use data processing and analysis engines to perform data analysis and distributed processing on teachers and students, and then establish data cleaning strategies to check data quality.

[0067] The establishment of a distributed computing framework and the use of a data processing and analysis engine to perform data analysis and distributed processing on teachers and students is to first establish a Kafka message queue, collect teacher and student data from the data source in real time, use the Apache Flink stream processing engine to process and analyze the real-time data, clean the data, and calculate the data indicators of the data in real time;

[0068] Then establish a data lake, and then regularly import teacher and student data into the data lake, use the MapReduce batch processing engine to analyze and process the data sets in the data lake, use TensorFlow technology for data mining and machine learning, implement aggregation analysis and association analysis, identify correlations and associations in the data set, and discover hidden data patterns and trends in teacher and student data.

[0069] The process of building a data lake is as follows:

[0070] Select the Apache Hadoop ecosystem as the data lake technology stack;

[0071] Redesign the data lake architecture and standardize data storage, data partitioning strategies, and data naming conventions;

[0072] Establish access control policies and authorize access users and operation users;

[0073] Establish data collection and integration strategies to regulate and restrict ETL processes and real-time data stream processing;

[0074] Perform metadata management, including data description information, data source and processing history, and data version, to ensure data accuracy, completeness, and consistency;

[0075] Use data warehouse connectivity tools and the Apache Spark analytics platform to query and analyze data;

[0076] Establish a monitoring mechanism, implement data lake maintenance strategies, and perform data backup, data recovery, capacity planning, and performance optimization.

[0077] The business service module is used to modularize the campus business applications to obtain multiple independent business project applications, and to simultaneously perform unified management and distributed management on these independent business project applications. It divides the functions and defines the boundaries of each business project application, defines the interfaces and boundaries of each module, and clarifies the data communication and dependency relationships between each module.

[0078] The specific steps for modularizing campus business applications are as follows:

[0079] According to the system functions and business requirements of the service system, the campus business application is modularly split into different functional modules to obtain multiple independent business application modules;

[0080] Formulate module specifications to ensure the consistency of each module during development and maintenance, including naming specifications, code style, and documentation requirements. Define clear interfaces for each module to reduce coupling between modules, including input and output, function calls, and data formats.

[0081] Establish a modular architecture and adopt the single responsibility principle to make each module responsible for only one clear function or responsibility to avoid overlapping functions. Use dependency injection and reverse dependency technology to manage the dependencies between modules to avoid module circular dependencies and excessive coupling. Use Kafka message queue communication to establish data communication between modules to ensure loose coupling between modules.

[0082] Perform separate unit tests for each module to verify the correctness of its functions and interfaces, and then perform module integration tests to verify the correctness and stability of the modules' collaborative work.

[0083] The steps for unified management of business project applications are as follows:

[0084] Clarify the organizational structure and naming conventions of each business module, formulate branch management strategy and version control strategy, define version number rules and release cycles, use the main branch for stable versions, the development branch for technical development, and the repair branch for vulnerability repairs, and use the Git version control system to manage the change history of each module;

[0085] Develop a dependency management strategy, manage and review the dependencies of each module, make the dependencies clear and reasonable, and use dependency management tools to automatically resolve and install dependencies;

[0086] Use integrated automated testing and static code analysis tools to implement automated continuous integration and continuous deployment processes, configure automated build tools, deploy, define and automatically trigger build processes, manage deployment methods and configuration files of different modules in a unified manner, and use Docker containerization technology to achieve unified deployment and environmental isolation;

[0087] Use Puppet configuration management tools to manage and automate the deployment and maintenance of each module environment, write repeatable and scalable configuration scripts, and ensure the consistency and manageability of each module environment;

[0088] Set monitoring alarm rules, configure a unified monitoring system, monitor the operating status, performance indicators and resource consumption of each module, respond to and resolve module performance degradation or abnormalities in a timely manner, use performance testing tools to simulate high-load scenarios, evaluate the performance of each module under different loads, and perform performance evaluation and optimization based on the performance results;

[0089] Conduct periodic security audits, assess the security and vulnerability risks of each module, regularly update related dependencies and libraries, design and implement unified authentication and authorization strategies, and ensure that each module can only access restricted resources and data.

[0090] The steps for distributed management of business project applications are as follows:

[0091] Use the rules of high cohesion and low coupling to clarify the functional boundaries of each module. Each module solves a specific problem or implements a specific function.

[0092] Define the interfaces and communication methods between standard modules, use interface specification protocols to ensure the consistency of interaction between modules, and use standardized communication protocols and data formats to enable integration and communication between modules;

[0093] Use semantic versioning to manage the version of each module, clarify the meaning of the version number and the type of change, and use dependency management tools to manage and resolve dependencies between modules so that each module can obtain and use the corresponding dependencies;

[0094] Use Docker containerization technology to achieve modular deployment and management, design and deploy independent units for each module, update and expand individual modules as needed, and use load balancing and automatic expansion technology to dynamically adjust resource allocation according to the load and performance requirements of each module;

[0095] Conduct separate fault-tolerance design for each module and establish fault-tolerance mechanisms, such as implementing retry logic, backup service endpoints, and failover;

[0096] Use the Grafana performance monitoring tool to monitor and adjust the performance of the modules, perform performance analysis and optimization on each module, and make each module respond quickly under various load conditions.

[0097] The component deployment module is used to perform environmental configuration, component installation, resource deployment and interface configuration for components within the system, install and configure the operating system for each component, update the component's drivers and firmware, configure the network settings of each component, and install and configure corresponding applications and services according to the functions of each component, including monitoring software, management tools and device drivers.

[0098] The steps for configuring the interfaces of the components in the system are as follows:

[0099] According to the functional requirements of each component, clarify the function and purpose of each component interface. The components include campus network equipment, campus servers and campus data storage equipment;

[0100] Configure the network devices, servers and storage devices in the service system and their interface parameters. The network device interface parameter configuration includes: configuring IP address, subnet mask, gateway, and VLAN settings;

[0101] Server interface parameter configuration includes: configuration of IP address, network connection parameters, RAID configuration, and operating system installation;

[0102] Storage device interface parameter configuration includes: configuration of IP address, storage pool, storage volume, access control list, etc.

[0103] After configuration is complete, perform functional tests on each component and interface, check whether network devices can correctly forward data packets, and confirm network connectivity through ping tests;

[0104] Confirm whether the server can start the operating system normally and confirm the network connection through access test;

[0105] Check the status of storage volumes and storage pools on storage devices to ensure that storage resources can be accessed and used normally;

[0106] Conduct security audits, restrict access to interfaces, and record and back up the configuration information of each component interface.

[0107] Campus network equipment configuration includes:

[0108] Use an SSH client to connect to the component's IP address or domain name and enter the configuration interface for configuration.

[0109] Connect to the component's IP address or domain name through Telnet and enter the component's command line interface for configuration;

[0110] Configure the component using an HTTPS connection in the web management interface by entering the component's IP address or domain name;

[0111] Connect to the console of a component through a serial connection to configure initial network settings or for troubleshooting.

[0112] The campus server configuration includes:

[0113] Use the iLO server remote management card to connect to the management card's IP address over the network to access the server console for remote server control and management;

[0114] Configure through the local operating system, BIOS or UEFI setup interface;

[0115] Remotely access, manage and configure the server's operating system and network settings through the SSH protocol.

[0116] Campus data storage equipment is configured to include:

[0117] Manage storage pools, storage volumes, and RAID settings by entering the component's IP address or domain name in a browser.

[0118] Connect to the storage device's CLI through SSH or Telnet and use the CLI to perform configuration.

[0119] The above-mentioned embodiments only express the preferred implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for ordinary technicians in this field, several modifications, improvements and substitutions can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application.

Claims

1. A campus big data comprehensive service system, characterized in that: include: User management module, used for collecting campus teacher and student accounts, user management and portrait analysis; Establish data pipelines, design data pipelines to collect teacher and student data, establish a distributed computing framework, use data processing and analysis engines to perform data analysis and distributed processing on teachers and students, and then establish data cleaning strategies to check data quality; Data analysis and distributed processing first establish a Kafka message queue, collect teacher and student data from the data source in real time, use the Apache Flink stream processing engine to process and analyze the real-time data, clean the data, and calculate the data indicators in real time; Then build a data lake, and then regularly import teacher and student data into the data lake, use the MapReduce batch processing engine to analyze and process the data sets in the data lake, use TensorFlow technology for data mining and machine learning, implement aggregation analysis and association analysis, identify correlations and associations in the data set, and discover hidden data patterns and trends in teacher and student data; The business service module is used to modularize campus business applications to obtain multiple independent business project applications, and to manage these independent business project applications in a unified and distributed manner; The steps for unified management of business project applications are as follows: Clarify the organizational structure and naming conventions of each business module, formulate branch management strategies and version control strategies, define version number rules and release cycles, and use the Git version control system to manage the change history of each module; Develop a dependency management strategy, manage and review the dependencies of each module, make the dependencies clear and reasonable, and use dependency management tools to automatically resolve and install dependencies; Use integrated automated testing and static code analysis tools to implement automated continuous integration and continuous deployment processes, configure automated build tools, deploy, define and automatically trigger build processes, manage deployment methods and configuration files of different modules in a unified manner, and use Docker containerization technology to achieve unified deployment and environmental isolation; Use Puppet configuration management tools to manage and automate the deployment and maintenance of each module environment, write repeatable and scalable configuration scripts, and ensure the consistency and manageability of each module environment; Set monitoring alarm rules, configure a unified monitoring system, monitor the operating status, performance indicators and resource consumption of each module, respond to and resolve module performance degradation or abnormalities in a timely manner, use performance testing tools to simulate high-load scenarios, evaluate the performance of each module under different loads, and perform performance evaluation and optimization based on the performance results; Conduct periodic security audits to assess the security and vulnerability risks of each module, regularly update related dependencies and libraries, and design and implement unified authentication and authorization strategies to ensure that each module can only access restricted resources and data; The steps for distributed management of business project applications are as follows: Use the rules of high cohesion and low coupling to clarify the functional boundaries of each module. Each module solves a specific problem or implements a specific function. Define the interfaces and communication methods between standard modules, use interface specification protocols to ensure the consistency of interaction between modules, and use standardized communication protocols and data formats to enable integration and communication between modules; Use semantic versioning to manage the version of each module, clarify the meaning of the version number and the type of change, and use dependency management tools to manage and resolve dependencies between modules so that each module can obtain and use the corresponding dependencies; Use Docker containerization technology to achieve modular deployment and management, design and deploy independent units for each module, update and expand individual modules as needed, and use load balancing and automatic expansion technology to dynamically adjust resource allocation according to the load and performance requirements of each module; Carry out separate fault-tolerant design for each module and establish a fault-tolerant mechanism; Use the Grafana performance monitoring tool to monitor and adjust the performance of the modules, perform performance analysis and optimization on each module, and make each module respond quickly under various load conditions; Perform functional division and boundary definition for each business project application, define the interface and boundary of each module, and clarify the data communication and dependency relationship between each module; Component deployment module, used to perform environment configuration, component installation, resource deployment and interface configuration for components within the system; Install and configure the operating system for each component, update the component's drivers and firmware, configure the network settings of each component, and install and configure the corresponding applications and services based on the functions of each component.

2. A campus big data comprehensive service system according to claim 1, characterized in that: The process of establishing a data lake is as follows: Select the Apache Hadoop ecosystem as the data lake technology stack; Redesign the data lake architecture and standardize data storage, data partitioning strategies, and data naming conventions; Establish access control policies and authorize access users and operation users; Establish data collection and integration strategies to regulate and restrict ETL processes and real-time data stream processing; Perform metadata management, including data description information, data source and processing history, and data version, to ensure data accuracy, completeness, and consistency; Use data warehouse connectivity tools and the Apache Spark analytics platform to query and analyze data; Establish a monitoring mechanism, implement data lake maintenance strategies, and perform data backup, data recovery, capacity planning, and performance optimization.

3. A campus big data comprehensive service system according to claim 1, characterized in that: The specific steps of modularizing the campus business application are as follows: According to the system functions and business requirements of the service system, the campus business application is modularly split into different functional modules to obtain multiple independent business application modules; Formulate module specifications to ensure the consistency of each module during development and maintenance, define clear interfaces for each module, and reduce coupling between modules; Establish a modular architecture and adopt the single responsibility principle to make each module responsible for only one clear function or responsibility to avoid overlapping functions. Use dependency injection and reverse dependency technology to manage the dependencies between modules to avoid module circular dependencies and excessive coupling. Use Kafka message queue communication to establish data communication between modules to ensure loose coupling between modules. Perform separate unit tests for each module to verify the correctness of its functions and interfaces, and then perform module integration tests to verify the correctness and stability of the modules' collaborative work.

4. A campus big data comprehensive service system according to claim 1, characterized in that: The steps for configuring the interfaces of the components in the system are as follows: According to the functional requirements of each component, clarify the function and purpose of each component interface. The components include campus network equipment, campus servers and campus data storage equipment; Configure network devices, servers and storage devices in the service system and configure their interface parameters; After configuration is complete, perform functional testing and verification on each component and interface; Conduct security audits, restrict access to interfaces, and record and back up the configuration information of each component interface.

5. A campus big data comprehensive service system according to claim 4, characterized in that: The configuration of the campus network equipment includes: Use an SSH client to connect to the component's IP address or domain name and enter the configuration interface for configuration. Connect to the component's IP address or domain name through Telnet and enter the component's command line interface for configuration; Configure the component using an HTTPS connection in the web management interface by entering the component's IP address or domain name; Connect to the console of a component through a serial connection to configure initial network settings or for troubleshooting.

6. A campus big data comprehensive service system according to claim 4, characterized in that: The configuration of the campus server includes: Use the iLO server remote management card to connect to the management card's IP address over the network to access the server console for remote server control and management; Configure through the local operating system, BIOS or UEFI setup interface; Remotely access, manage and configure the server's operating system and network settings through the SSH protocol.

7. A campus big data comprehensive service system according to claim 4, characterized in that: The campus data storage equipment is configured to include: Manage storage pools, storage volumes, and RAID settings by entering the component's IP address or domain name in a browser. Connect to the storage device's CLI through SSH or Telnet and use the CLI to perform configuration.

Images