Parallel computing optimized school-enterprise cooperation big data analysis training system and method

Abstract

This invention discloses a parallel computing-optimized university-enterprise collaborative big data analysis training system and method, including a university-enterprise resource matching module, a distributed data storage module, a parallel task scheduling module, a parallel computing optimization module, a training project management module, a training process monitoring module, a results evaluation and feedback module, and a system management and access control module. This invention uses an intelligent matching algorithm to achieve precise matching between real enterprise data and business needs and university training requirements, improving the authenticity and relevance of training projects and helping students quickly adapt to actual enterprise business scenarios. It adopts a distributed hierarchical storage architecture and diverse indexing technologies to ensure the secure storage and efficient access of massive training data, while supporting dynamic data flow and expansion to meet the data storage needs of training projects of different scales.

Description

Technical Field

[0001] This invention relates to the field of university-enterprise cooperation technology, specifically to a parallel computing-optimized university-enterprise cooperation big data analysis training system and method. Background Technology

[0002] With the rapid development of the big data industry, enterprises have an increasingly urgent need for big data analysis talents with practical skills. As a crucial link connecting university talent cultivation with the actual needs of enterprises, existing university-enterprise collaborative training systems face the following technical challenges:

[0003] The training data is disconnected from real-world enterprise scenarios, often using simulated datasets, making it difficult for students to experience large-scale, high-dimensional data processing in actual business operations. Data processing efficiency is low, lacking targeted parallel computing optimization mechanisms, leading to task blocking and wasted computing resources when faced with massive amounts of training data. There is a lack of systematic adaptation mechanisms for university-enterprise resource integration, making it difficult to accurately match enterprise needs with university training course objectives. The training process lacks real-time monitoring and dynamic adjustment capabilities, failing to promptly detect student operational deviations and computational task anomalies. The evaluation criteria for training results are singular, failing to form a comprehensive evaluation system that combines actual enterprise business requirements with university knowledge assessments. The system lacks scalability, making it difficult to be compatible with different enterprise data source formats and the diverse training course needs of universities. Summary of the Invention

[0004] The purpose of this invention is to provide a parallel computing-optimized big data analysis training system and method for university-enterprise cooperation, in order to solve the problems existing in the prior art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a parallel computing-optimized university-enterprise collaborative big data analysis training system, comprising a university-enterprise resource docking module, a distributed data storage module, a parallel task scheduling module, a parallel computing optimization module, a training project management module, a training process monitoring module, a results evaluation and feedback module, and a system management and access control module; the university-enterprise resource docking module, distributed data storage module, parallel task scheduling module, parallel computing optimization module, training project management module, training process monitoring module, and results evaluation and feedback module are respectively connected to the system management and access control module; the parallel task scheduling module is respectively connected to the distributed data storage module, the parallel computing optimization module, and the training project management module; the training process monitoring module is respectively connected to the parallel task scheduling module, the parallel computing optimization module, and the training project management module; the results evaluation and feedback module is connected to the training project management module;

[0006] The school-enterprise resource matching module is used to achieve precise matching between real enterprise resources and the practical training needs of universities;

[0007] The distributed data storage module adopts a distributed architecture to perform hierarchical storage and management of training data;

[0008] The parallel task scheduling module is based on the characteristics of the training tasks and the status of computing resources;

[0009] The parallel computing optimization module reduces computational complexity, shortens task execution time, and optimizes computing resource utilization through various parallel computing optimization strategies.

[0010] The training project management module is used for the creation, configuration, release, and lifecycle management of training projects;

[0011] The training process monitoring module monitors the data flow, task execution status and student operation behavior in real time during the training process, and promptly detects and warns of abnormal situations.

[0012] The results evaluation and feedback module combines enterprise business standards with university knowledge assessment requirements to conduct a multi-dimensional comprehensive evaluation of training results and generate targeted feedback suggestions.

[0013] The system management and access control module is responsible for system configuration, user access control, and operation log recording.

[0014] Preferably, the school-enterprise resource matching module includes an enterprise resource access unit, a university training needs analysis unit, a resource matching unit, and a resource update and synchronization unit; the enterprise resource access unit and the university training needs analysis unit are respectively connected to the resource matching unit; the resource matching unit is connected to the resource update and synchronization unit.

[0015] Preferably, the distributed data storage module includes a data preprocessing unit, a data hierarchical storage unit, a data index construction unit, and a data access control unit; the data preprocessing unit is connected to the data hierarchical storage unit; and the data hierarchical storage unit is connected to the data index construction unit and the data access control unit, respectively.

[0016] Preferably, the parallel task scheduling module includes a task parsing and splitting unit, a resource status monitoring unit, a task priority calculation unit, and a task scheduling execution unit; the task parsing and splitting unit and the resource status monitoring unit are respectively connected to the task priority calculation unit; the task priority calculation unit is connected to the task scheduling execution unit.

[0017] The task priority calculation unit uses a priority calculation model to determine the execution priority of subtasks. The formula for the priority calculation model is as follows:

[0018]

[0019] in Let i be the priority of the i-th subtask. This represents the urgency level coefficient. To calculate the complexity coefficient, For data size coefficient, The dependency coefficient, , , , The weighting coefficients are and satisfy the following conditions: .

[0020] Preferably, the parallel computing optimization module includes a data sharding optimization unit, a computing task parallelization unit, a load balancing optimization unit, and a computing result merging optimization unit; the data sharding optimization unit and the computing task parallelization unit are respectively connected to the load balancing optimization unit; the load balancing optimization unit is connected to the computing result merging optimization unit.

[0021] The load balancing optimization unit uses a load balancing algorithm to adjust the load on the computing nodes. The formula for the load balancing algorithm is as follows:

[0022]

[0023] in To adjust the load of the j-th computing node, To adjust the load of the j-th computing node, To calculate the total number of nodes, Let be the load of the k-th computing node. The average load across all compute nodes. This represents the load migration weight between the j-th node and the k-th node.

[0024] Preferably, the training project management module includes a project creation unit, a project configuration unit, a project release unit, and a project lifecycle management unit; the project creation unit is connected to the project configuration unit; the project configuration unit is connected to the project release unit; and the project release unit is connected to the project lifecycle management unit.

[0025] Preferably, the training process monitoring module includes a data flow monitoring unit, a task execution status monitoring unit, a student operation behavior monitoring unit, and an anomaly early warning unit; the data flow monitoring unit, the task execution status monitoring unit, and the student operation behavior monitoring unit are respectively connected to the anomaly early warning unit.

[0026] Preferably, the results evaluation and feedback module includes an evaluation indicator construction unit, a results data collection unit, a comprehensive evaluation calculation unit, and a feedback generation unit; the evaluation indicator construction unit and the results data collection unit are respectively connected to the comprehensive evaluation calculation unit; the comprehensive evaluation calculation unit is connected to the feedback generation unit.

[0027] The comprehensive evaluation calculation unit uses a weighted summation algorithm to calculate the comprehensive score of the training results. The formula for the weighted summation algorithm is as follows:

[0028]

[0029] in For comprehensive scoring, To evaluate the total number of indicators, The weight of the k-th evaluation indicator and satisfying , The score for the k-th evaluation indicator.

[0030] Preferably, the system management and access control module includes a system configuration unit, a user management unit, a permission allocation unit, and an operation log recording unit; the system configuration unit, user management unit, and permission allocation unit are each connected to the operation log recording unit.

[0031] The training method for a university-industry collaborative big data analysis training system optimized for parallel computing includes the following steps:

[0032] Step 1: System Initialization and Resource Integration

[0033] After the system is started, the system management and access control module completes system parameter initialization and user permission loading, and the university-enterprise resource docking module enters standby mode; enterprises submit real datasets, business analysis requirements and evaluation index systems through the enterprise resource access unit, which are stored in the system after verification; universities submit training requirement information through the university training requirement parsing unit, the system extracts core requirements and constructs requirement feature vectors; the resource matching unit calculates the matching degree between requirements and enterprise resources, determines the optimal resource combination and feeds it back to the university, and the resource docking is completed after the university confirms it;

[0034] Step 2: Creating and Configuring the Training Project

[0035] University teachers create training projects through the project creation unit of the training project management module and enter basic project information; they configure training data permissions, task splitting rules, parallel computing parameters (such as data partition size, number of parallel tasks), and evaluation indicator weight parameters through the project configuration unit; after configuration, they publish the training project to the designated student group through the project publishing unit, and the system sends a training start notification to the students.

[0036] Step 3: Data Preprocessing and Distributed Storage

[0037] The distributed data storage module's data preprocessing unit cleans, transforms, integrates, and de-identifies the accessed real enterprise data; the data hierarchical storage unit allocates the processed training data to the corresponding storage level according to the data access frequency and importance; the data index building unit builds corresponding indexes for various types of data; and the data access control unit sets data access permissions according to preset permission rules to complete the storage deployment of the training data.

[0038] Step 4: Parallel Task Scheduling and Computational Optimization

[0039] After logging into the system, students select the corresponding training project and start the training task; the task parsing and splitting unit of the parallel task scheduling module splits the complex training task into multiple sub-tasks and constructs a task dependency graph; the resource status monitoring unit collects the resource status of computing nodes in real time, and the task priority calculation unit calculates the priority of each sub-task through the priority formula; the task scheduling and execution unit schedules the sub-tasks to the appropriate computing nodes based on the priority and resource status.

[0040] The data sharding optimization unit of the parallel computing optimization module performs mixed sharding processing on the training data, the computing task parallelization unit executes sub-tasks using corresponding parallel strategies, the load balancing optimization unit dynamically adjusts the node load through the load balancing formula, and the calculation result merging optimization unit efficiently merges the calculation results of each node to generate preliminary training results.

[0041] Step 5: Real-time monitoring of the training process

[0042] The training process monitoring module monitors the entire training process: the data flow monitoring unit tracks the data flow status, the task execution status monitoring unit updates the execution progress, results and time consumption of sub-tasks and the overall task in real time, the student operation behavior monitoring unit records various student operation behaviors, and the anomaly warning unit provides real-time warnings for detected anomalies (such as task execution timeout, data access anomalies, operation violations) and notifies relevant teachers and administrators.

[0043] Step 6: Evaluation and Feedback of Practical Training Results

[0044] After completing the practical training tasks, students submit their training results (analysis reports, code, calculation results, etc.); the results evaluation and feedback module's results data collection unit collects the results information submitted by students and the subjective evaluations of teachers and industry mentors; the comprehensive evaluation calculation unit calculates the comprehensive score of the practical training results using the comprehensive evaluation formula; the feedback generation unit generates personalized improvement suggestions and learning resource recommendations based on the score results and indicator performance, and provides feedback to students and teachers.

[0045] Step 7: Project Archiving and System Maintenance

[0046] After the training project is completed, the training project management module archives and stores project data, student results, evaluation reports, etc.; the system management and access control module records all user operation logs, and the system administrator checks and optimizes the system operation status through the system configuration unit to maintain system stability; the school-enterprise resource docking module updates enterprise resources and university training needs in real time to support subsequent training projects.

[0047] Compared with the prior art, the beneficial effects of the present invention are:

[0048] This invention utilizes an intelligent matching algorithm to precisely connect real enterprise data and business needs with university training requirements, enhancing the authenticity and relevance of training projects and helping students quickly adapt to actual business scenarios. It employs a distributed hierarchical storage architecture and diverse indexing technologies to ensure secure storage and efficient access to massive amounts of training data, while supporting dynamic data flow and expansion to meet the data storage needs of training projects of varying sizes. Through task splitting, intelligent scheduling, and load balancing optimization, it fully leverages the advantages of parallel computing, reducing computational complexity, shortening training task execution time, and improving computing resource utilization, effectively addressing large-scale data processing scenarios. It covers the entire lifecycle management of training projects—from creation, execution, monitoring, evaluation, and archiving—supporting diverse training scenario deployments, real-time monitoring of various anomalies during training, and ensuring smooth training implementation. Finally, it constructs a multi-dimensional evaluation system combining enterprise business standards and university knowledge assessments to achieve objective and comprehensive evaluation of training results, while generating targeted feedback suggestions to help students precisely improve their abilities. Attached Figure Description

[0049] Figure 1 This is a system module diagram of the present invention;

[0050] Figure 2 This is a schematic diagram of the school-enterprise resource docking module of the present invention;

[0051] Figure 3 This is a schematic diagram of the distributed data storage module of the present invention;

[0052] Figure 4 This is a schematic diagram of the parallel task scheduling module of the present invention;

[0053] Figure 5 This is a flowchart of the method of the present invention. Detailed Implementation

[0054] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0055] Please see Figure 1-5 This invention provides a parallel computing-optimized university-industry collaborative big data analysis training system, comprising a university-industry resource docking module, a distributed data storage module, a parallel task scheduling module, a parallel computing optimization module, a training project management module, a training process monitoring module, an outcome evaluation and feedback module, and a system management and access control module. The university-industry resource docking module, distributed data storage module, parallel task scheduling module, parallel computing optimization module, training project management module, training process monitoring module, and outcome evaluation and feedback module are each connected to the system management and access control module. The parallel task scheduling module is connected to the distributed data storage module, parallel computing optimization module, and training project management module. The training process monitoring module is connected to the parallel task scheduling module, parallel computing optimization module, and training project management module. The outcome evaluation and feedback module is connected to the training project management module.

[0056] The university-enterprise resource matching module is used to achieve precise matching between real enterprise resources and the practical training needs of universities;

[0057] The distributed data storage module uses a distributed architecture to store and manage training data in layers.

[0058] The parallel task scheduling module is based on the characteristics of the training tasks and the status of computing resources;

[0059] The parallel computing optimization module reduces computational complexity, shortens task execution time, and optimizes computing resource utilization through various parallel computing optimization strategies.

[0060] The training project management module is used for the creation, configuration, release, and lifecycle management of training projects;

[0061] The training process monitoring module monitors data flow, task execution status and student operation behavior in real time during the training process, and promptly detects and warns of abnormal situations.

[0062] The results evaluation and feedback module combines enterprise business standards with university knowledge assessment requirements to conduct a multi-dimensional comprehensive evaluation of training results and generate targeted feedback suggestions.

[0063] The system management and access control module is responsible for system configuration, user access control, and operation log recording.

[0064] The university-enterprise resource matching module includes an enterprise resource access unit, a university training needs analysis unit, a resource matching unit, and a resource update and synchronization unit; the enterprise resource access unit and the university training needs analysis unit are respectively connected to the resource matching unit; the resource matching unit is connected to the resource update and synchronization unit.

[0065] Enterprise Resource Access Unit: Provides standardized data interfaces and task description specifications to support the access and verification of real enterprise datasets (including structured, semi-structured, and unstructured data), business analysis needs, and evaluation indicator systems, ensuring the legality, integrity, and availability of accessed resources;

[0066] The University Practical Training Needs Analysis Unit: Receives information from universities regarding their practical training course objectives, students' knowledge levels, practical training duration, and expected competency development directions. It then extracts core requirement keywords through semantic analysis technology and constructs a requirement feature vector.

[0067] Resource matching unit: Based on the demand feature vector and the enterprise resource feature vector, the matching degree between the two is calculated by the cosine similarity algorithm to select the optimal matching resource combination;

[0068] Resource Update and Synchronization Unit: Establish a dynamic resource update mechanism to synchronize changes in enterprise resources in real time, periodically verify the matching effectiveness between resources and training needs, and trigger a rematch process when the matching degree is lower than the threshold.

[0069] The distributed data storage module includes a data preprocessing unit, a data hierarchical storage unit, a data index building unit, and a data access control unit; the data preprocessing unit is connected to the data hierarchical storage unit; the data hierarchical storage unit is connected to the data index building unit and the data access control unit, respectively.

[0070] Data preprocessing unit: Cleans, transforms, integrates and de-identifies the raw data from the enterprise, removes redundant data, corrects outliers, converts data of different formats into a standard format, and encrypts and de-identifies sensitive information to ensure data security;

[0071] Data tiered storage unit: Adopting a hot data-warm data-cold data tiered storage strategy, the high-frequency access real-time training data is stored in the in-memory database, the recently used training data is stored in the distributed file system, and the low-frequency access historical data and backup data are stored in the low-cost distributed storage medium. The dynamic flow of data at different levels is realized through the data migration strategy.

[0072] Data index building unit: For different types of training data, build diverse index structures, including structured data index based on B+ tree, unstructured data index based on inverted index, and time-series data index based on spatiotemporal index, to improve data query and retrieval efficiency;

[0073] Data access control unit: Role-based access control policies assign different data access permissions to different users (enterprise personnel, university teachers, students), record data access logs, and prevent unauthorized access and data leakage.

[0074] The parallel task scheduling module includes a task parsing and splitting unit, a resource status monitoring unit, a task priority calculation unit, and a task scheduling execution unit; the task parsing and splitting unit and the resource status monitoring unit are respectively connected to the task priority calculation unit; the task priority calculation unit is connected to the task scheduling execution unit.

[0075] Task parsing and decomposition unit: Semantic parsing is performed on complex analysis tasks in the training project. Based on data dependencies and computational logic, the tasks are decomposed into multiple independent or weakly dependent subtasks to form a task dependency graph.

[0076] Resource Status Monitoring Unit: Collects real-time resource status information such as CPU utilization, memory usage, disk I / O rate, and network bandwidth of computing nodes, constructs a resource status matrix, and dynamically updates the availability of computing resources;

[0077] Task priority calculation unit: Taking into account factors such as the urgency of the task, computational complexity, data size, and dependencies with other tasks, the execution priority of each subtask is determined through a priority calculation model.

[0078] Task scheduling and execution unit: Based on task priority, resource state matrix and task dependency graph, an improved greedy algorithm is used to schedule subtasks to the optimal computing node for execution, while dynamically adjusting the task execution order to avoid resource conflicts and task blocking.

[0079] The formula for calculating task priority is as follows:

[0080]

[0081] in This represents the priority of the i-th subtask, with a value range of [0, 10]. The larger the value, the higher the priority. The urgency coefficient of the i-th subtask is calculated based on the deadline of the training task and the current remaining time, and its value ranges from [0,10]. The computational complexity coefficient of the i-th subtask is determined based on the number and difficulty of the computational operations required by the task, and its value ranges from [0, 10]. is the data scale coefficient of the i-th subtask, which is calculated based on the amount of data processed by the task, and its value range is [0,10]. The dependency coefficient of the i-th subtask is determined based on the number of other tasks that depend on this task, and its value ranges from [0, 10]. , , , Let be the weighting coefficient, satisfying It can be dynamically adjusted according to the needs of the training scenario.

[0082] The parallel computing optimization module includes a data sharding optimization unit, a computing task parallelization unit, a load balancing optimization unit, and a computing result merging optimization unit; the data sharding optimization unit and the computing task parallelization unit are respectively connected to the load balancing optimization unit; the load balancing optimization unit is connected to the computing result merging optimization unit.

[0083] Data Sharding Optimization Unit: Based on the data distribution characteristics and task computation logic, a hybrid sharding strategy combining hash sharding and range sharding is adopted to evenly shard large-scale training data to multiple computing nodes, reducing data transmission overhead;

[0084] Computational task parallelization unit: For different types of analysis tasks, corresponding parallelization execution strategies are adopted, including data parallelism (data is sharded and processed in parallel), task parallelism (multiple independent tasks are executed simultaneously), and pipeline parallelism (tasks are divided into multiple stages and different stages are executed in parallel).

[0085] Load balancing optimization unit: Real-time monitoring of task execution load on each computing node, and dynamic adjustment of task allocation through load balancing algorithm to avoid situations where some nodes are overloaded while others are idle;

[0086] The calculation result merging and optimization unit efficiently merges the parallel calculation results of each computing node. It adopts a divide-and-conquer strategy to reduce data transmission and computational overhead during the merging process, ensuring the accuracy of the merged results.

[0087] The load balancing algorithm formula is as follows:

[0088]

[0089] in, To adjust the load of the j-th computing node, To adjust the load of the j-th computing node, To calculate the total number of nodes, Let be the load of the k-th computing node. The average load across all compute nodes. , The load migration weight between the j-th node and the k-th node is determined based on factors such as network bandwidth and distance between the nodes, and its value ranges from [0,1].

[0090] The training project management module includes a project creation unit, a project configuration unit, a project release unit, and a project lifecycle management unit; the project creation unit is connected to the project configuration unit; the project configuration unit is connected to the project release unit; and the project release unit is connected to the project lifecycle management unit.

[0091] Project creation unit: Supports college teachers to create training projects according to their training needs, and set basic information such as project name, training objectives, training duration, scope of participating students, and associated enterprise resources;

[0092] Project configuration unit: Provides a visual configuration interface, supporting flexible configuration of training data permissions, task splitting rules, parallel computing parameters, evaluation index weights, etc., to adapt to different types of training scenarios;

[0093] Project publishing unit: Publishes the configured training projects to students, supporting both batch and targeted publishing, and notifies students of the project launch information through the system notification function;

[0094] Project Lifecycle Management Unit: Manages the entire lifecycle of training projects, including creation, configuration, release, execution, and termination. Supports pause, resume, modification, and archiving of projects, and retains complete historical data of projects.

[0095] The training process monitoring module includes a data flow monitoring unit, a task execution status monitoring unit, a student operation behavior monitoring unit, and an anomaly early warning unit; the data flow monitoring unit, the task execution status monitoring unit, and the student operation behavior monitoring unit are respectively connected to the anomaly early warning unit.

[0096] Data flow monitoring unit: Tracks the flow path and status changes of training data in real time during storage, transmission, and calculation, and records data reading, writing, modification, deletion and other operations to ensure the traceability of data flow;

[0097] Task execution status monitoring unit: collects information such as the execution progress, execution result, and time consumption of each subtask in real time, and displays the task execution status through visual charts. It triggers an alert when the task execution times out, fails, or resource consumption is abnormal.

[0098] Student Operation Behavior Monitoring Unit: Records students' login and logout times, data query operations, task submission records, code writing and modification traces, and other behavioral data during the practical training process to construct a student operation behavior profile;

[0099] Anomaly Warning Unit: Based on preset anomaly judgment rules, it provides real-time warnings for data flow anomalies, task execution anomalies, and student operation anomalies, and supports notification to teachers and administrators via system messages, emails, and other means.

[0100] The results evaluation and feedback module includes an evaluation indicator construction unit, a results data collection unit, a comprehensive evaluation calculation unit, and a feedback generation unit; the evaluation indicator construction unit and the results data collection unit are respectively connected to the comprehensive evaluation calculation unit; the comprehensive evaluation calculation unit is connected to the feedback generation unit.

[0101] Evaluation indicator construction unit: Combining enterprise business evaluation standards (such as the accuracy of analysis results, business relevance, and efficiency indicators) with university knowledge assessment requirements (such as theoretical application ability, methodological correctness, and logical integrity), a multi-dimensional evaluation indicator system is constructed, supporting the custom addition of indicators and adjustment of weights;

[0102] Results Data Collection Unit: Automatically collects results information such as training analysis reports, data processing codes, and calculation results submitted by students, while also collecting subjective evaluation opinions from enterprise mentors and university teachers;

[0103] Comprehensive evaluation calculation unit: The weighted summation algorithm is used to comprehensively calculate the scores of each evaluation indicator to obtain the comprehensive score of the training results;

[0104] Feedback generation unit: Based on the comprehensive score and the scores of each indicator, analyze the students' strengths and weaknesses in knowledge mastery, skills application, and business understanding, and generate targeted improvement suggestions and learning resource recommendations.

[0105] The comprehensive evaluation calculation formula is as follows:

[0106]

[0107] in For comprehensive scoring, the value range is [0, 100]. To evaluate the total number of indicators, The weight of the k-th evaluation indicator and satisfying , The score for the k-th evaluation indicator ranges from [0, 100].

[0108] The system management and access control module includes a system configuration unit, a user management unit, a permission allocation unit, and an operation log recording unit; the system configuration unit, user management unit, and permission allocation unit are each connected to the operation log recording unit.

[0109] System configuration unit: Supports the configuration and modification of system operating parameters (such as data storage threshold, task scheduling cycle, early warning threshold), interface configuration, third-party service integration parameters, etc., to ensure that the system adapts to different operating environments;

[0110] User Management Unit: Implements the registration, review, activation, freezing, and information modification management of users (including enterprise users, university teacher users, student users, and system administrator users), and maintains the user information database;

[0111] Permission allocation unit: Based on the RBAC (role-based access control) model, it assigns corresponding system operation permissions and data access permissions to different roles, supporting fine-grained control and dynamic adjustment of permissions;

[0112] Operation Log Recording Unit: Comprehensively records all users' system operation behaviors (including login, data access, task submission, configuration modification, etc.), system operation status changes and abnormal events. The log information is tamper-proof and supports log querying, exporting and audit analysis.

[0113] The training method for a university-industry collaborative big data analysis training system optimized for parallel computing includes the following steps:

[0114] Step 1: System Initialization and Resource Integration

[0115] After the system is started, the system management and access control module completes system parameter initialization and user permission loading, and the university-enterprise resource docking module enters standby mode; enterprises submit real datasets, business analysis requirements and evaluation index systems through the enterprise resource access unit, which are stored in the system after verification; universities submit training requirement information through the university training requirement parsing unit, the system extracts core requirements and constructs requirement feature vectors; the resource matching unit calculates the matching degree between requirements and enterprise resources, determines the optimal resource combination and feeds it back to the university, and the resource docking is completed after the university confirms it;

[0116] Step 2: Creating and Configuring the Training Project

[0117] University teachers create training projects through the project creation unit of the training project management module and enter basic project information; they configure training data permissions, task splitting rules, parallel computing parameters (such as data partition size, number of parallel tasks), and evaluation indicator weight parameters through the project configuration unit; after configuration, they publish the training project to the designated student group through the project publishing unit, and the system sends a training start notification to the students.

[0118] Step 3: Data Preprocessing and Distributed Storage

[0119] The distributed data storage module's data preprocessing unit cleans, transforms, integrates, and de-identifies the accessed real enterprise data; the data hierarchical storage unit allocates the processed training data to the corresponding storage level according to the data access frequency and importance; the data index building unit builds corresponding indexes for various types of data; and the data access control unit sets data access permissions according to preset permission rules to complete the storage deployment of the training data.

[0120] Step 4: Parallel Task Scheduling and Computational Optimization

[0121] After logging into the system, students select the corresponding training project and start the training task; the task parsing and splitting unit of the parallel task scheduling module splits the complex training task into multiple sub-tasks and constructs a task dependency graph; the resource status monitoring unit collects the resource status of computing nodes in real time, and the task priority calculation unit calculates the priority of each sub-task through the priority formula; the task scheduling and execution unit schedules the sub-tasks to the appropriate computing nodes based on the priority and resource status.

[0122] The data sharding optimization unit of the parallel computing optimization module performs mixed sharding processing on the training data, the computing task parallelization unit executes sub-tasks using corresponding parallel strategies, the load balancing optimization unit dynamically adjusts the node load through the load balancing formula, and the calculation result merging optimization unit efficiently merges the calculation results of each node to generate preliminary training results.

[0123] Step 5: Real-time monitoring of the training process

[0124] The training process monitoring module monitors the entire training process: the data flow monitoring unit tracks the data flow status, the task execution status monitoring unit updates the execution progress, results and time consumption of sub-tasks and the overall task in real time, the student operation behavior monitoring unit records various student operation behaviors, and the anomaly warning unit provides real-time warnings for detected anomalies (such as task execution timeout, data access anomalies, operation violations) and notifies relevant teachers and administrators.

[0125] Step 6: Evaluation and Feedback of Practical Training Results

[0126] After completing the practical training tasks, students submit their training results (analysis reports, code, calculation results, etc.); the results evaluation and feedback module's results data collection unit collects the results information submitted by students and the subjective evaluations of teachers and industry mentors; the comprehensive evaluation calculation unit calculates the comprehensive score of the practical training results using the comprehensive evaluation formula; the feedback generation unit generates personalized improvement suggestions and learning resource recommendations based on the score results and indicator performance, and provides feedback to students and teachers.

[0127] Step 7: Project Archiving and System Maintenance

[0128] After the training project is completed, the training project management module archives and stores project data, student results, evaluation reports, etc.; the system management and access control module records all user operation logs, and the system administrator checks and optimizes the system operation status through the system configuration unit to maintain system stability; the school-enterprise resource docking module updates enterprise resources and university training needs in real time to support subsequent training projects.

[0129] Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A parallel computing-optimized university-industry collaborative big data analysis training system, characterized by: The system includes a school-enterprise resource docking module, a distributed data storage module, a parallel task scheduling module, a parallel computing optimization module, a training project management module, a training process monitoring module, an outcome evaluation and feedback module, and a system management and access control module. The school-enterprise resource docking module, distributed data storage module, parallel task scheduling module, parallel computing optimization module, training project management module, training process monitoring module, and outcome evaluation and feedback module are all connected to the system management and access control module. The parallel task scheduling module is connected to the distributed data storage module, parallel computing optimization module, and training project management module. The training process monitoring module is connected to the parallel task scheduling module, parallel computing optimization module, and training project management module. The outcome evaluation and feedback module is connected to the training project management module. The school-enterprise resource matching module is used to achieve precise matching between real enterprise resources and the practical training needs of universities; The distributed data storage module adopts a distributed architecture to perform hierarchical storage and management of training data; The parallel task scheduling module is based on the characteristics of the training tasks and the status of computing resources; The parallel computing optimization module reduces computational complexity, shortens task execution time, and optimizes computing resource utilization through various parallel computing optimization strategies. The training project management module is used for the creation, configuration, release, and lifecycle management of training projects; The training process monitoring module monitors the data flow, task execution status and student operation behavior in real time during the training process, and promptly detects and warns of abnormal situations. The results evaluation and feedback module combines enterprise business standards with university knowledge assessment requirements to conduct a multi-dimensional comprehensive evaluation of training results and generate targeted feedback suggestions. The system management and access control module is responsible for system configuration, user access control, and operation log recording.

2. The parallel computing optimized university-enterprise collaborative big data analysis training system according to claim 1, characterized in that: The school-enterprise resource matching module includes an enterprise resource access unit, a university training needs analysis unit, a resource matching unit, and a resource update and synchronization unit; the enterprise resource access unit and the university training needs analysis unit are respectively connected to the resource matching unit; the resource matching unit is connected to the resource update and synchronization unit.

3. The parallel computing optimized university-enterprise collaborative big data analysis training system according to claim 1, characterized in that: The distributed data storage module includes a data preprocessing unit, a data hierarchical storage unit, a data index construction unit, and a data access control unit; the data preprocessing unit is connected to the data hierarchical storage unit; the data hierarchical storage unit is connected to the data index construction unit and the data access control unit, respectively.

4. The parallel computing optimized university-enterprise collaborative big data analysis training system according to claim 1, characterized in that: The parallel task scheduling module includes a task parsing and splitting unit, a resource status monitoring unit, a task priority calculation unit, and a task scheduling execution unit; the task parsing and splitting unit and the resource status monitoring unit are respectively connected to the task priority calculation unit; the task priority calculation unit is connected to the task scheduling execution unit. The task priority calculation unit uses a priority calculation model to determine the execution priority of subtasks. The formula for the priority calculation model is as follows: in Let i be the priority of the i-th subtask. This represents the urgency level coefficient. To calculate the complexity coefficient, For data size coefficient, The dependency coefficient, , , , The weighting coefficients are and satisfy the following conditions: .

5. The parallel computing optimized university-enterprise collaborative big data analysis training system according to claim 1, characterized in that: The parallel computing optimization module includes a data sharding optimization unit, a computing task parallelization unit, a load balancing optimization unit, and a computing result merging optimization unit; the data sharding optimization unit and the computing task parallelization unit are respectively connected to the load balancing optimization unit; the load balancing optimization unit is connected to the computing result merging optimization unit. The load balancing optimization unit uses a load balancing algorithm to adjust the load on the computing nodes. The formula for the load balancing algorithm is as follows: in To adjust the load of the j-th computing node, To adjust the load of the j-th computing node, To calculate the total number of nodes, Let be the load of the k-th computing node. The average load across all compute nodes. This represents the load migration weight between the j-th node and the k-th node.

6. The parallel computing optimized university-enterprise collaborative big data analysis training system according to claim 1, characterized in that: The training project management module includes a project creation unit, a project configuration unit, a project release unit, and a project lifecycle management unit; the project creation unit is connected to the project configuration unit; the project configuration unit is connected to the project release unit; and the project release unit is connected to the project lifecycle management unit.

7. The parallel computing optimized university-enterprise collaborative big data analysis training system according to claim 1, characterized in that: The training process monitoring module includes a data flow monitoring unit, a task execution status monitoring unit, a student operation behavior monitoring unit, and an anomaly early warning unit; the data flow monitoring unit, the task execution status monitoring unit, and the student operation behavior monitoring unit are respectively connected to the anomaly early warning unit.

8. The parallel computing optimized university-enterprise collaborative big data analysis training system according to claim 1, characterized in that: The results evaluation and feedback module includes an evaluation indicator construction unit, a results data collection unit, a comprehensive evaluation calculation unit, and a feedback generation unit; the evaluation indicator construction unit and the results data collection unit are respectively connected to the comprehensive evaluation calculation unit; the comprehensive evaluation calculation unit is connected to the feedback generation unit. The comprehensive evaluation calculation unit uses a weighted summation algorithm to calculate the comprehensive score of the training results. The formula for the weighted summation algorithm is as follows: in For comprehensive scoring, To evaluate the total number of indicators, The weight of the k-th evaluation indicator and satisfying , The score for the k-th evaluation indicator.

9. The parallel computing optimized university-enterprise collaborative big data analysis training system according to claim 1, characterized in that: The system management and access control module includes a system configuration unit, a user management unit, an access control unit, and an operation log recording unit; the system configuration unit, user management unit, and access control unit are each connected to the operation log recording unit.

10. The training method of the parallel computing optimized school-enterprise cooperation big data analysis training system according to any one of claims 1-9, characterized in that: Includes the following steps: Step 1: System Initialization and Resource Integration After the system is started, the system management and access control module completes system parameter initialization and user permission loading, and the university-enterprise resource docking module enters standby mode; enterprises submit real datasets, business analysis requirements and evaluation index systems through the enterprise resource access unit, which are stored in the system after verification; universities submit training requirement information through the university training requirement parsing unit, the system extracts core requirements and constructs requirement feature vectors; the resource matching unit calculates the matching degree between requirements and enterprise resources, determines the optimal resource combination and feeds it back to the university, and the resource docking is completed after the university confirms it; Step 2: Creating and Configuring the Training Project University teachers create training projects through the project creation unit of the training project management module and enter basic project information; they configure training data permissions, task splitting rules, parallel computing parameters, and evaluation indicator weight parameters through the project configuration unit; after configuration, they publish the training project to the designated student group through the project publishing unit, and the system sends a training start notification to the students. Step 3: Data Preprocessing and Distributed Storage The data preprocessing unit of the distributed data storage module cleans, transforms, integrates, and de-identifies the real enterprise data it receives. The data tiered storage unit allocates the processed training data to the corresponding storage tier based on the data access frequency and importance. The data indexing construction unit builds corresponding indexes for various types of data, and the data access control unit sets data access permissions according to preset permission rules to complete the storage and deployment of training data. Step 4: Parallel Task Scheduling and Computational Optimization After logging into the system, students select the corresponding training project and start the training task; the task parsing and splitting unit of the parallel task scheduling module breaks down complex training tasks into multiple sub-tasks and constructs a task dependency graph. The resource status monitoring unit collects the resource status of computing nodes in real time, and the task priority calculation unit calculates the priority of each subtask using the priority formula. The task scheduling and execution unit schedules subtasks to appropriate computing nodes based on priority and resource status; The data sharding optimization unit of the parallel computing optimization module performs mixed sharding processing on the training data, the computing task parallelization unit executes sub-tasks using corresponding parallel strategies, the load balancing optimization unit dynamically adjusts the node load through the load balancing formula, and the calculation result merging optimization unit efficiently merges the calculation results of each node to generate preliminary training results. Step 5: Real-time monitoring of the training process The training process monitoring module monitors the entire training process: the data flow monitoring unit tracks the data flow status, the task execution status monitoring unit updates the execution progress, results and time of sub-tasks and the overall task in real time, the student operation behavior monitoring unit records various student operation behaviors, and the anomaly warning unit provides real-time warnings for detected anomalies and notifies relevant teachers and administrators. Step 6: Evaluation and Feedback of Practical Training Results After completing the practical training tasks, students submit their practical training results; the results evaluation and feedback module's results data collection unit collects the results information submitted by students and the subjective evaluations from teachers and corporate mentors; The comprehensive evaluation calculation unit calculates the comprehensive score of the training results using the comprehensive evaluation formula; Based on the scoring results and indicator performance, the feedback generation unit generates personalized improvement suggestions and learning resource recommendations, which are then fed back to students and teachers. Step 7: Project Archiving and System Maintenance After the training project is completed, the training project management module archives and stores the project data, student results, and evaluation reports; the system management and access control module records all user operation logs, and the system administrator checks and optimizes the system's operating status through the system configuration unit to maintain system stability; the school-enterprise resource docking module updates enterprise resources and university training needs in real time to support subsequent training projects.

Images