Industrial unit-oriented cross-level cross-platform data acquisition system

Abstract

The invention discloses a cross-level cross-platform data acquisition system oriented to an industrial unit, and relates to the technical field of computer data processing, and the system comprises a semantic abstraction layer which is used for establishing a semantic description model for original data in the industrial unit and generating a feature set; the dynamic arrangement layer is used for analyzing, splitting and arranging a model training task according to a machine learning training requirement to generate a training task process; the algorithm adaptation layer is used for providing a unified training interface for different machine learning algorithms and training frameworks and outputting a model result obtained by training; the resource management layer is used for uniformly monitoring, scheduling and optimizing resources in a model training process; and the result convergence layer is used for converging, evaluating and managing model results from the model training process to form model output. The invention aims to solve the problems of lack of intelligent processing capability after data acquisition, low training efficiency of a machine learning model, difficulty in parameter tuning and low utilization rate of training resources.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer data processing, and particularly relates to an industry unit-oriented cross-level cross-platform data acquisition system. BACKGROUND

[0002] With the rapid development of industrial internet, artificial intelligence and big data technology, machine learning models are increasingly widely used in industry units, and have gradually become an important technical means to support production optimization, quality prediction, equipment operation and maintenance and intelligent decision-making. In the fields of manufacturing, energy and transportation, industry units usually rely on a large number of equipment systems and business management systems to accumulate massive data, and expect to deeply mine these data through machine learning models to realize business efficiency improvement and operation cost reduction.

[0003] However, in the actual application process, the construction, training and application of machine learning models still face many technical challenges. First, the data sources in the industry unit are diverse in level and heterogeneous in platform, and the data is distributed in equipment layer systems, control systems and various business platforms. The data structure, semantic definition and update frequency between different systems are quite different. The existing technology usually preprocesses multi-source data through data acquisition, cleaning and integration to meet the subsequent analysis requirements, but such data engineering process is often separated from the machine learning model training process, lacking a unified organization and coordination mechanism, resulting in a large amount of manual participation in the model training stage. The existing machine learning model training process highly depends on human experience, especially in the feature engineering and parameter optimization links. Feature selection, feature transformation and feature combination usually need to be completed by engineers with professional experience through repeated experiments, which not only consumes time and effort, but also is difficult to reuse in different business scenarios. At the same time, the setting of model parameters often depends on manual parameter tuning or simple grid search method, which is low in training efficiency and high in resource consumption under the condition of high model complexity or large data scale, and is difficult to meet the requirements of efficiency and stability of industrial-level application.

[0004] With the continuous expansion of the scale of machine learning model application in industrial units, the model training task gradually presents the characteristics of multi-model parallel, frequent iteration and continuous optimization. The traditional single machine training or static resource allocation mode is difficult to effectively support the large-scale concurrent training demand, there is a lack of unified scheduling between training tasks, the utilization rate of computing resources is low, and the training period is difficult to predict. When the model training relies on distributed computing resources, the existing scheme usually needs to be customized and developed for a specific computing framework, the system complexity is high, and the maintenance cost is large. After the model training is completed, the model evaluation, version management and deployment process also generally lack systematic support, the existing technology is not uniform in the selection of model evaluation indicators, the evaluation results are difficult to form effective feedback with the training process, the model version information is scattered, the model update and rollback operation is complex, and it is difficult to form a closed-loop management of the whole life cycle of the model. This makes it difficult to continuously optimize the performance of the model, and also increases the uncertainty of the application of the model in the actual production environment. SUMMARY

[0005] The purpose of the present application is to provide an industrial unit-oriented cross-level cross-platform data acquisition system to solve the problems of lack of intelligent processing capability after data acquisition, low efficiency of machine learning model training, difficulty in parameter optimization, low utilization rate of training resources and lack of global intelligent management capability in the prior art

[0006] To solve the above technical problems, the present application provides the following technical solutions:

[0007] An industrial unit-oriented cross-level cross-platform data acquisition system, which adopts an innovative paradigm based on a unified semantic model and intelligent machine learning training, and constructs a system-level architecture that works collaboratively with a semantic abstraction layer, a dynamic orchestration layer, an algorithm adaptation layer, a resource management layer and a result aggregation layer, to realize the full-process automation of data acquisition, intelligent feature processing, efficient model training and application of different business platforms from the device level to the enterprise level in the industrial unit.

[0008] The system comprises a semantic abstraction layer, a dynamic orchestration layer, an algorithm adaptation layer, a resource management layer and a result aggregation layer.

[0009] The semantic abstraction layer is used to establish a unified and standardized semantic description model for all data entities and their attributes to be acquired in the industrial unit, and to perform intelligent feature processing on the acquired data. The semantic description model adopts an ontology method to define the type, attribute, relationship and business context of the data entity. Specifically, the semantic abstraction layer includes a model construction module and a feature processing module.

[0010] The model construction module constructs a core ontology library according to international or industry standard data models and in combination with specific business domain knowledge of industrial units. The core ontology library defines general concepts such as equipment, process, product, order and their hierarchical relationships, and also defines a type classification system of features, including standardized feature types such as numerical features, discrete features, time features and text features.

[0011] The feature processing module is responsible for intelligent feature engineering processing of the collected raw data. First, the feature processing module automatically identifies the feature type of each data field based on the distribution characteristics, statistical properties and field naming rules of the data, and classifies it into the feature type classification system defined in the core ontology library. Second, the feature processing module integrates a useless feature recognition engine based on regular expressions. The engine maintains an extensible rule set and can identify and filter out features that are useless for machine learning model training, such as unique identifiers, fixed constants, high null rate, etc. Third, the feature processing module performs feature correlation analysis to calculate the correlation coefficient matrix between features, identifies highly correlated feature groups and redundant features, and displays the feature relationship network to the user through a visual interface. Finally, the feature processing module automatically generates feature optimization suggestions based on statistical analysis and historical training experience, including feature combination suggestions, feature transformation suggestions (such as logarithmic transformation, standardization) and feature selection suggestions. Through the processing of the semantic abstraction layer, the raw data from heterogeneous sources is not only normalized at the semantic level, but also converted into a high-quality feature set suitable for machine learning model training.

[0012] The dynamic orchestration layer, as the intelligent control center of the system, is used to dynamically generate, schedule and execute cross-level and cross-platform model training task flows according to the machine learning training requirements of the upper layer applications. The dynamic orchestration layer includes a task analysis module, a process orchestration module and an execution engine module.

[0013] The task analysis module receives machine learning training requirements defined in a high-level description language or a graphical interface. The requirements are based on the unified semantics and feature types defined by the semantic abstraction layer. The training requirements include target variable definition, feature selection range, model type preference, performance constraints (such as maximum training time, minimum accuracy) and resource constraints. The task analysis module decomposes the macro training requirements into a series of atomic training operations, each corresponding to a specific machine learning task unit such as data preprocessing, feature transformation, model training, cross-validation, hyperparameter optimization, etc.

[0014] The flow orchestration module automatically arranges the execution order and parallel relationship of these atomic operations according to the data dependency relationship between atomic operations, task characteristics, and preset optimization strategies, and generates an executable machine learning training task flow. The flow orchestration module integrates task scheduling capabilities based on the Ray distributed computing framework, which can decompose large-scale training tasks into multiple subtasks and distribute them to different nodes of a distributed computing cluster for parallel execution. Optimization strategies include minimizing total training time, maximizing GPU / CPU utilization, avoiding repeated access to the same data source, and balancing the load between computing nodes.

[0015] Further, the flow orchestration module integrates an intelligent parameter optimization algorithm based on the Optuna framework. This algorithm uses a Bayesian optimization strategy to dynamically search for the optimal hyperparameter combination of the model during training. The flow orchestration module first defines the parameter search space based on the model type, then iteratively trains and evaluates the model, uses historical test results to guide subsequent parameter selection, and gradually approaches the optimal parameters. For scenarios that require training multiple models simultaneously, the flow orchestration module implements a multi-model joint optimization mechanism that adjusts the parameters of each model in coordination under a unified optimization objective to generate the optimal model combination solution. The flow orchestration module solves the multi-objective optimization problem under the constraints of training quality requirements and resources to generate a training task flow that balances training efficiency, model performance, and resource consumption.

[0016] The execution engine module is responsible for loading and running the task flow generated by the flow orchestration module, driving the corresponding adapters in the algorithm adaptation layer to perform specific training actions, and managing the state, exceptions, and intermediate results during task execution. The execution engine monitors training progress in real time, collects performance metrics such as loss function values and validation accuracy during training, and supports pausing, resuming, and early termination of training tasks.

[0017] The algorithm adaptation layer provides a unified interface for various machine learning frameworks and algorithms, as well as the ability to physically connect and interact with various data sources within the industry unit. The algorithm adaptation layer is composed of multiple pluggable algorithm adapters and data source adapters.

[0018] The algorithm adapter types cover a variety of mainstream machine learning frameworks and algorithm libraries, including the scikit-learn adapter for traditional machine learning algorithms, the XGBoost and LightGBM adapters for gradient boosting algorithms, the TensorFlow and PyTorch adapters for deep learning, and the Prophet adapter for time series analysis. Each algorithm adapter encapsulates all the details required to interact with a specific machine learning framework, including parameter format conversion, data format adaptation, training interface calls, model serialization and deserialization, etc. The algorithm adapter receives standardized training instructions from the dynamic orchestration layer. These instructions, based on a unified semantic definition, are internally converted into native API calls for specific machine learning frameworks to execute training. Subsequently, the trained model object is converted into an internal standard format and appended with metadata (including algorithm type, hyperparameter configuration, training time, and performance metrics) before being passed up to the result aggregation layer.

[0019] Data source adapters retain the ability to connect to various data sources within the industry unit, used to acquire training data in real time or in batches during the training process. Data source adapter types include device layer protocol adapters, platform layer interface adapters, and file system adapters. Device layer protocol adapters include industrial protocol adapters for programmable logic controllers, distributed control systems, and CNC machine tools, as well as IoT protocol adapters for various sensors. Platform layer interface adapters include application programming interface adapters, database connection adapters, and message queue adapters for business systems such as manufacturing execution systems, enterprise resource planning, and product lifecycle management. File system adapters are used to read training data files from shared directories or cloud storage.

[0020] The resource management layer is used for global monitoring, scheduling, and optimization of the computing, storage, and network resources required for distributed machine learning training, ensuring the stable and efficient execution of large-scale concurrent training tasks. The resource management layer includes a resource monitoring module, a load balancing module, and an elastic scaling module.

[0021] The resource monitoring module collects key performance indicators (KPIs) for each computing node in the distributed training cluster in real time. These KPIs cover three main categories: hardware resources, training progress, and network status. Hardware resource KPIs include the percentage utilization of each CPU core, GPU utilization and memory usage, used memory and total memory bytes, disk partition utilization and I / O read / write speeds, and network interface inflow and outflow bandwidth. Training progress KPIs primarily target the individual training task instances running on the node, including CPU usage, GPU usage, memory usage, training iteration speed, and current loss function value. Network status KPIs include the round-trip latency from the node to the data source and other training nodes, and connection stability.

[0022] The load balancing module dynamically distributes training tasks to nodes with lighter loads and matching resources based on real-time load information from the resource monitoring module and the characteristics of the training tasks issued by the dynamic orchestration layer (such as whether GPUs are needed, estimated training time, and data size). The task allocation strategy comprehensively considers the node's processing capacity, current load level, network latency with the training data source, and the task's resource requirements. For distributed training tasks, the load balancing module selects multiple nodes with similar performance and good network connectivity to form a training cluster, maximizing parallel training efficiency.

[0023] The elastic scaling module automatically adjusts the size of the training cluster based on preset load threshold strategies and training task queue status. When system training tasks are backlogged and node load remains above the high-water mark threshold, the elastic scaling module automatically creates new training node instances in the resource pool to horizontally scale processing capacity; when training tasks are completed and node load remains below the low-water mark threshold, it automatically reclaims idle node instances to save costs. Furthermore, the elastic scaling module integrates with the cloud platform's Infrastructure as a Service (IaaS) interface. When resource expansion is needed, the elastic scaling module directly requests and configures new virtual machine or container instances (especially GPU instances) on the cloud platform through the application programming interface (API), and automatically deploys the required training environment, algorithm adapter software, and their configurations on the instances, achieving seamless elastic scaling of training resources and cloud computing resources.

[0024] The results aggregation layer is used to receive, evaluate, manage, and visualize model results from distributed training tasks, forming a unified, high-quality model output. The results aggregation layer includes a training results aggregation module, a model evaluation module, and a model management module.

[0025] The training results aggregation module adopts a distributed message queue architecture to receive training result data streams from various algorithm adapters, including trained model objects, training process logs, intermediate evaluation metrics, etc., and provides buffering capabilities to cope with the mismatch between the upstream training result production rate and the downstream model evaluation consumption rate.

[0026] The model evaluation module performs a comprehensive quality assessment of the trained model. First, it automatically selects an appropriate set of evaluation metrics based on the machine learning task type (classification, regression, clustering, etc.). For classification tasks, it calculates accuracy, precision, recall, F1 score, and AUC-ROC curve; for regression tasks, it calculates mean absolute error, root mean square error, and coefficient of determination (R²); for clustering tasks, it calculates silhouette coefficient and Davidson-Borging index. The module also performs feature importance analysis, calculating the contribution of each feature to the model's prediction results (using SHAP values, ranking importance, or the model's built-in `feature_importance`), generating feature importance rankings and visualizations to help users understand the model's decision-making logic. The module supports user-defined evaluation rules and dynamic thresholds. Users can write complex evaluation rules based on business logic through the rule definition interface. Furthermore, for continuously trained models, the module can automatically calculate and dynamically adjust the threshold for judging model quality based on the statistical distribution of historical model performance to accommodate normal fluctuations in business data.

[0027] The model management module is responsible for version management, storage, and deployment of evaluated models. It assigns a unique version identifier to each trained model, records complete metadata (including training data source, feature set, algorithm type, hyperparameter configuration, training time, evaluation metrics, and trainer information), and serializes the model object before storing it in the model repository. The module provides model retrieval, comparison, and rollback functions, allowing users to query historical model versions, compare performance differences between versions, and roll back to previous versions when necessary. For models requiring application deployment, the module provides a model export interface, supporting exporting models to standard formats or direct deployment to the inference service engine for online prediction.

[0028] Furthermore, the feature processing module in the semantic abstraction layer supports self-learning and optimization of feature processing rules. This module records historical feature processing operations and their impact on subsequent model training, and uses machine learning algorithms to analyze the effectiveness of feature processing rules. When a new data source or a change in data pattern is detected, the feature processing module can recommend potential feature processing strategies based on historical learning experience. After administrator confirmation, the feature processing rule base is automatically updated, thereby reducing the complexity of system maintenance.

[0029] Furthermore, the workflow orchestration module in the dynamic orchestration layer integrates a resource-aware scheduling algorithm. This algorithm, when orchestrating training task flows, considers not only training efficiency and model performance but also the cost of training resources as one of the optimization objectives. Resource costs are quantitatively modeled based on the usage costs of computing node types (CPU nodes, GPU nodes), training duration, and storage costs. The workflow orchestration module generates a training task flow that balances training efficiency, model quality, and resource costs by solving a multi-objective optimization problem that satisfies model performance requirements and training time constraints.

[0030] Furthermore, the algorithm adapter in the algorithm adaptation layer possesses adaptive model training capabilities. When the distribution of training data changes significantly or model performance deteriorates, the algorithm adapter can automatically trigger model retraining or incremental training processes. The adapter has a built-in data drift detection mechanism that continuously monitors changes in the distribution of input data through statistical tests (such as KS test and chi-square test). When data drift is detected to exceed a preset threshold, the adapter sends a retraining request to the dynamic orchestration layer, automatically initiating the model update process to ensure that the model always maintains good predictive performance.

[0031] Furthermore, the elastic scaling module of the resource management layer supports multi-cloud hybrid deployment. When resources on a single cloud platform are insufficient or costs are too high, the elastic scaling module can distribute training tasks across multiple cloud platforms (such as Alibaba Cloud, AWS, and Azure) through a unified cloud management interface, achieving cross-cloud resource scheduling and cost optimization.

[0032] Furthermore, the model evaluation module in the results aggregation layer provides real-time model performance monitoring and visualization capabilities. Users can view the progress of the training task, the current loss function value, the performance metric changes on the validation set, and resource consumption during training in real time via a web interface. The model evaluation module also supports multi-model performance comparison visualization, allowing users to compare the performance of models with different algorithms and hyperparameter configurations in the same chart, assisting in making the optimal model selection.

[0033] Compared with the prior art, the beneficial technical effects of the present invention are as follows:

[0034] This invention introduces a semantic abstraction layer based on a unified semantic model, enabling unified semantic modeling and intelligent feature processing of heterogeneous data across levels and platforms within an industry unit. The system can complete the standardized description of data entities, attributes, and feature types during the data acquisition phase, and perform feature type identification, invalid feature filtering, and feature correlation analysis on the raw data. This transforms multi-source heterogeneous data into a high-quality feature set suitable for machine learning model training, effectively reducing the degree of manual intervention in the data preparation process and improving the consistency and reusability of feature engineering.

[0035] This invention achieves automatic parsing, streamlined organization, and intelligent scheduling of machine learning model training tasks by setting up a dynamic orchestration layer. The system can decompose the model training process into multiple atomic training operations based on training requirements and generate an executable training task flow according to the dependencies between operations, supporting parallel execution and dynamic optimization of training tasks. Simultaneously, the dynamic orchestration layer adaptively searches and optimizes the model parameter space during training by incorporating historical training results, enabling the model training process to continuously approach the optimal parameter configuration while meeting performance constraints, thereby effectively improving model training efficiency and model performance stability.

[0036] This invention achieves unified encapsulation and invocation of various machine learning algorithms and training frameworks through a pluggable architecture design of the algorithm adaptation layer. The system shields the interface differences between different algorithms and frameworks through standardized training instructions, decoupling the model training process from the specific implementation framework. Users can flexibly switch model types without repeatedly developing training logic for different algorithms, significantly improving the system's flexibility, scalability, and algorithm reusability.

[0037] This invention provides unified monitoring and scheduling of computing, storage, and network resources in a distributed training environment through a resource management layer. It dynamically adjusts resource allocation based on training task load, enabling on-demand allocation and elastic scaling of training resources. Through a reasonable load balancing and resource scheduling mechanism, the system can improve overall resource utilization efficiency while ensuring model training stability, avoiding idle or overloaded computing resources, thereby reducing the operating costs of large-scale machine learning training.

[0038] This invention achieves centralized management and unified evaluation of model training results through a result aggregation layer. The system can automatically select appropriate model evaluation metrics based on different machine learning task types, perform performance analysis and comparison of trained models, and manage and maintain models throughout their lifecycle through a model management module, thus forming a complete closed loop from data collection, feature processing, model training to model evaluation and application. This mechanism helps improve the controllability and sustainable optimization capabilities of model applications, enhancing the application effectiveness and reliability of machine learning models in actual business scenarios within industrial units. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of the overall technical architecture of the cross-level, cross-platform data acquisition system for industrial units proposed in this invention. Detailed Implementation

[0040] The features and exemplary embodiments of various aspects of the present invention will now be described in detail. To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely intended to explain the present invention and not to limit the present invention. For those skilled in the art, the present invention can be practiced without some of these specific details. The following description of the embodiments is merely to provide a better understanding of the present invention by illustrating examples of the invention.

[0041] Example 1

[0042] This embodiment provides a specific implementation of a cross-level, cross-platform data acquisition, intelligent feature processing, and machine learning model training and application system for industrial units. This system is applicable to manufacturing, energy industries, and other data environments with multi-level equipment systems and multiple business platforms, enabling automated training, optimization, and application of machine learning models under complex industrial data conditions.

[0043] In this embodiment, the system is deployed in the private cloud, industry cloud, or hybrid cloud computing environment of the industry unit, and establishes communication connections with the internal device layer system, business management system, and data storage system of the industry unit through the network. The system as a whole adopts an innovative paradigm based on a unified semantic model and intelligent machine learning training, and constructs a system-level architecture in which a semantic abstraction layer, a dynamic orchestration layer, an algorithm adaptation layer, a resource management layer, and a result aggregation layer work together to achieve full-process automation from raw data acquisition, feature construction, model training, parameter optimization to model evaluation and model management.

[0044] During system operation, the semantic abstraction layer first performs unified modeling and processing of data from different levels and platforms. Data sources within industrial units include, but are not limited to, equipment systems at the production site, industrial control systems, and various business management platforms. The semantic abstraction layer constructs a core ontology library through a model building module. This core ontology library, based on industry-standard data models and incorporating business domain knowledge of the industrial unit, defines data entities such as equipment, processes, products, and orders, along with their attribute relationships. It also establishes a unified feature type classification system for the standardized description of features required for training machine learning models. When the system receives data, it maps the raw data fields to the corresponding data entities and feature types in the core ontology library, providing a consistent semantic foundation for subsequent feature processing and model training.

[0045] After semantic modeling is completed, the feature processing module in the semantic abstraction layer performs intelligent feature engineering on the collected raw data. The feature processing module analyzes the input data fields, automatically identifying the feature type corresponding to each field based on the statistical distribution characteristics, value variation characteristics, and field naming information, and classifying them into numerical features, categorical features, temporal features, or textual features. Based on this, the feature processing module uses its built-in rule engine to filter the identified features, automatically identifying and filtering feature fields that contribute little to model training or are noisy. Subsequently, the feature processing module performs correlation analysis on the retained features, identifying the relationships between features and potential redundancy, and generating feature optimization suggestions based on historical model training results. After the above processing, the raw data is transformed into a structured feature set with quality labels, serving as input for subsequent machine learning model training.

[0046] After the feature set is generated, the system enters the model training phase. The dynamic orchestration layer, acting as the system's intelligent control center, receives machine learning training requests submitted by users or upper-layer applications. These training requests are defined using a high-level description language or a graphical interface, and include information such as target variable definitions, feature selection ranges, model type preferences, performance constraints, and resource constraints. The task parsing module in the dynamic orchestration layer parses the training requests and, in conjunction with feature information provided by the semantic abstraction layer, decomposes the overall training requirements into multiple interrelated atomic training operations. These atomic training operations include, but are not limited to, feature transformation, training data partitioning, model training, model validation, and model evaluation.

[0047] After completing the training requirements analysis, the workflow orchestration module in the dynamic orchestration layer automatically generates an executable machine learning training task flow based on the data dependencies and execution conditions between each atomic training operation. This training task flow is organized in a process-oriented structure and supports the parallel execution of multiple training tasks. In this embodiment, the workflow orchestration module divides the training task into multiple sub-tasks based on the training task size, model complexity, and resource constraints, and schedules them to be executed in parallel on distributed computing nodes to improve overall training efficiency and shorten the training cycle. Simultaneously, during training, the workflow orchestration module incorporates parameter optimization strategies to automatically search and adjust the model's hyperparameters, continuously optimizing model performance while meeting constraints.

[0048] The training task flow generated by the dynamic orchestration layer is loaded and run by the execution engine module. The execution engine module is responsible for coordinating the execution order of each atomic training operation and driving the corresponding algorithm adapters in the algorithm adaptation layer to complete specific model training tasks. During model training, the execution engine module monitors the training progress and running status in real time, collects intermediate performance metrics generated during model training, and executes corresponding control strategies when abnormal situations are detected or the training effect does not meet expectations, including terminating inefficient training tasks or triggering parameter adjustment processes.

[0049] In this embodiment, the algorithm adaptation layer provides a unified algorithm calling interface for the dynamic orchestration layer. The algorithm adapter encapsulates the differences between various machine learning frameworks and algorithm libraries, converting standardized training instructions issued by the execution engine into training processes executable by specific machine learning frameworks, and converting the model results into a unified model representation format within the system after training. Simultaneously, the data source adapter in the algorithm adaptation layer acquires training data in real-time or in batches as needed during model training, ensuring that the training task continuously receives the required data input.

[0050] During model training, the resource management layer provides unified monitoring and scheduling of the distributed training environment. It collects real-time data on the resource usage status of each computing node, including computing, storage, and network resources, and feeds the monitoring results back to the dynamic orchestration layer. The system dynamically adjusts the scheduling strategy for training tasks based on resource usage, allocating tasks to computing nodes with sufficient resources and low load to ensure the stability and efficiency of the training process. When the scale of training tasks changes, the resource management layer can automatically expand or reclaim computing nodes based on load conditions, achieving elastic scaling of training resources.

[0051] After model training is complete, the training results are passed to the results aggregation layer for centralized management. The results aggregation layer summarizes the model results from each training task and performs a unified evaluation of model performance. The system automatically selects appropriate evaluation metrics based on the type of machine learning task, analyzes model performance, and generates evaluation results and visualizations. Through the model management module, the system manages the versions of trained models, recording training information, feature sets, parameter configurations, and evaluation results to support subsequent model comparisons, rollbacks, and deployment.

[0052] Through the above implementation methods, the system provided in this embodiment can achieve full-process automation from data collection and feature processing to machine learning model training, optimization and management in complex industrial data environments, significantly reducing model development and maintenance costs, and improving model training efficiency and model application effects.

[0053] Example 2

[0054] Based on Embodiment 1, this embodiment further provides a specific implementation method for multi-model joint training and intelligent parameter optimization, which is applicable to application scenarios in which industrial units need to simultaneously build, compare and optimize multiple machine learning models in the same business scenario.

[0055] In this embodiment, when the system receives a machine learning training request submitted by a user or an upper-layer application, the training request includes not only the target variable and feature range, but also multiple candidate model types and an overall optimization objective. The candidate model types can include models with various algorithmic structures, used for parallel training and performance comparison under the same dataset and business objective. The system identifies this type of training request as a multi-model joint training task, and the dynamic orchestration layer uniformly schedules and manages it.

[0056] During system operation, the semantic abstraction layer still first performs unified modeling and feature processing on the input data. The feature processing module, based on the feature type system defined in the core ontology library, performs feature type identification, invalid feature filtering, and feature correlation analysis on the raw data, generating a structured feature set. This feature set serves as the unified input for multi-model joint training, enabling different models to be trained in the same feature space, ensuring fairness and consistency in model performance comparisons.

[0057] After the feature set is prepared, the task parsing module in the dynamic orchestration layer analyzes the multi-model joint training requirements. Based on the multiple model types specified in the training requirements, the task parsing module breaks down the overall training requirements into multiple parallel model training branches. Each model training branch corresponds to the training process of a specific model, which includes atomic training operations such as training data partitioning, model training, model validation, and performance evaluation. During the decomposition process, the task parsing module also attaches unified performance evaluation metrics and optimization objectives to each model training branch to ensure that the training results of different models can be compared under the same evaluation system.

[0058] After breaking down the training requirements, the workflow orchestration module organizes multiple model training processes into parallel training task flows based on the independence of each model training branch. The module also allocates resources to different model training tasks based on the system's current computing resources and schedules them to distributed computing nodes for execution. During training, each model training task can run in parallel, and the system coordinates the execution pace of each task through a unified scheduling mechanism, thereby shortening the overall training cycle while ensuring efficient resource utilization.

[0059] During multi-model joint training, the workflow orchestration module further integrates parameter optimization strategies to automatically search and optimize the hyperparameter space of each model. The system establishes a corresponding parameter search space for each model type and dynamically adjusts the parameter search strategy based on model performance feedback during training. The parameter optimization process gradually approaches the optimal parameter combination through multiple rounds of training and evaluation iterations. In this process, the parameter optimization of different models proceeds in parallel under a unified optimization objective constraint, allowing the system to simultaneously obtain the performance of multiple models under their respective optimal parameter configurations.

[0060] During the training task execution phase, the execution engine module is responsible for driving the specific operation of each model training process. The execution engine module monitors the execution status and performance index changes of each model training task in real time, and dynamically adjusts the training strategy of the model when it detects that the training effect of a certain model is significantly lower than expected or the convergence speed is slow. This includes: terminating the training task of an inefficient model in advance or reducing the resource investment in its subsequent parameter search, thereby avoiding ineffective resource consumption.

[0061] After model training and parameter optimization are completed, the training results of each model are passed to the result aggregation layer for centralized processing. The result aggregation layer manages the model results from different model training branches in a unified manner and compares and analyzes the performance of each model based on preset evaluation metrics. The system generates multi-model performance comparison results through the model evaluation module and performs correlation analysis on model performance, parameter configuration, and training resource consumption, providing users with multi-dimensional model selection criteria.

[0062] After completing the model performance evaluation, the model management module categorizes the models according to the evaluation results. Models whose performance meets application requirements are included in the deployable model set, and their complete training and version information is recorded. For models whose performance does not meet the requirements, their training results are recorded as historical reference data for adjusting subsequent training and parameter optimization strategies. When a model needs to be deployed, users can select the optimal model from multiple candidate models or choose multiple models to form a model combination scheme based on business needs.

[0063] Through the above implementation methods, the system provided in this embodiment can efficiently support parallel training and joint optimization of multiple models in the same business scenario, enabling comprehensive comparison and intelligent selection of model performance. The system, through a unified scheduling and parameter optimization mechanism, significantly reduces the manual and time costs of multi-model experimentation, improves the scientific rigor and reliability of model selection, and is suitable for industrial application scenarios with high model performance requirements and the need for continuous optimization.

[0064] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Therefore, all equivalent changes made in accordance with the structure, shape, and principle of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A cross-level, cross-platform data acquisition system for industrial units, characterized in that, include: The semantic abstraction layer is used to access raw data collected from different levels and business platforms within the industry unit, establish a unified and standardized semantic description model for the raw data, and perform intelligent feature processing on the collected data based on the semantic description model to generate a feature set suitable for training machine learning models. The dynamic orchestration layer is used to parse, split, and orchestrate model training tasks according to machine learning training requirements, generate an executable machine learning training task flow, and schedule and control the execution of the training task flow. The algorithm adaptation layer provides a unified training interface for different machine learning algorithms and training frameworks, and performs model training operations according to the training instructions issued by the dynamic orchestration layer, and outputs the trained model results. The resource management layer is used to uniformly monitor, schedule, and optimize the computing, storage, and network resources used during model training. The results aggregation layer is used to aggregate, evaluate, and manage the model results from the model training process, forming a unified model output and supporting the subsequent application of the model.

2. The cross-level, cross-platform data acquisition system for industrial units according to claim 1, characterized in that: The semantic abstraction layer includes a model building module and a feature processing module. The model building module is used to build a core ontology library, which is used to define the types, attributes and relationships of data entities, and to establish a unified feature type classification system.

3. The cross-level, cross-platform data acquisition system for industrial units according to claim 2, characterized in that: The feature processing module is used to perform automatic feature type identification, invalid feature identification, and feature correlation analysis on the original data fields, and generate feature optimization suggestions based on the analysis results.

4. The cross-level, cross-platform data acquisition system for industrial units according to claim 1, characterized in that: The dynamic orchestration layer includes a task parsing module, a process orchestration module, and an execution engine module. The task parsing module is used to decompose machine learning training requirements into multiple atomic training operations, and the process orchestration module is used to generate training task processes based on the dependencies between the atomic training operations.

5. The cross-level, cross-platform data acquisition system for industrial units according to claim 4, characterized in that: The process orchestration module supports splitting the model training task into multiple sub-tasks and scheduling them to distributed computing nodes for parallel execution, and adaptively searching and optimizing the model parameter space based on historical training results during the training process.

6. The cross-level, cross-platform data acquisition system for industrial units according to claim 1, characterized in that: The algorithm adaptation layer includes multiple pluggable algorithm adapters, which are used to encapsulate the interface differences between different machine learning algorithms or training frameworks, and to convert standardized training instructions into training operations that can be executed by specific algorithms or training frameworks.

7. The cross-level, cross-platform data acquisition system for industrial units according to claim 6, characterized in that: The algorithm adaptation layer also includes a data source adapter, which is used to obtain training data from the device layer system, business management system or data storage system during model training.

8. The cross-level, cross-platform data acquisition system for industrial units according to claim 1, characterized in that: The resource management layer includes a resource monitoring module, a load balancing module, and an elastic scaling module, which are used to dynamically adjust the configuration of computing resources according to the training task load.

9. The cross-level, cross-platform data acquisition system for industrial units according to claim 1, characterized in that: The result aggregation layer includes a model evaluation module and a model management module. The model evaluation module is used to evaluate the performance of the trained model, and the model management module is used to manage the versioning of the model.

10. The cross-level, cross-platform data acquisition system for industrial units according to claim 1, characterized in that: The system supports parallel training of multiple models and performs unified evaluation and comparison of the training results of different models to assist in model selection or deployment.

Images