A data processing method and system for industrial internet based on big data

Abstract

This invention discloses a data processing method for the industrial internet based on big data. It includes designing a visual data processing workflow orchestration tool that allows users to flexibly define the logic and steps of data processing. During workflow orchestration, common data processing operations are implemented through a predefined data processing operator library. A visual configuration interface allows users to select the required data processing components through drag-and-drop operations and set the corresponding data types and processing conditions to form a complete data processing flowchart. The method acquires the user-configured data type information and performs initial conditional filtering based on this information, selecting raw data that meets the conditions. It determines whether dynamic data calculation over time is needed based on whether the conditions change. A real-time data processing engine is introduced to perform real-time data acquisition, cleaning, transformation, and analysis. Through a visual interface, NLP interaction, real-time processing, and intelligent scheduling, the flexibility of data processing is improved.

Description

Technical Field

[0001] This invention relates to the field of big data processing technology, and in particular to a data processing method and system for industrial internet based on big data. Background Technology

[0002] In today's information society, big data processing and business data visualization are crucial aspects of enterprise management and decision-making. With advancements in equipment data acquisition technology, the ability to process and analyze these massive amounts of data has become paramount. Traditional data processing methods typically rely on writing fixed code to handle specific data formats and computational needs, which limits their flexibility and efficiency.

[0003] Existing data processing relies on writing and maintaining specific data processing code, such as Python and Java. This requires highly skilled developers, which is not only costly but also makes it difficult to adapt to rapidly changing business needs. Furthermore, the long code development cycle hinders flexible adjustments.

[0004] Insufficient flexibility: Existing systems require manual intervention or custom development for scenarios with high real-time requirements and dynamic changes in conditions for task scheduling and historical data acquisition. Summary of the Invention

[0005] In order to solve the above-mentioned technical problems, the present invention provides an industrial internet data processing method and system based on big data.

[0006] The technical solution of this invention is implemented as follows:

[0007] A data processing method for the industrial internet based on big data includes the following steps:

[0008] S1. Design a visual data processing workflow orchestration tool that allows users to flexibly define the logic and steps of data processing. During the workflow orchestration process, common data processing operations can be implemented through a predefined data processing operator library.

[0009] S2. Through the visual configuration interface, users can select the required data processing components by dragging and dropping, and set the corresponding data types and processing conditions to form a complete data processing flowchart.

[0010] S3. Obtain the data type information configured by the user, and perform initial condition filtering judgment on the data based on this information to filter out the original data that meets the conditions.

[0011] S4. Determine whether data needs to be dynamically calculated over time based on whether the conditions change.

[0012] S5 introduces a real-time data processing engine to collect, clean, transform, and analyze data in real time.

[0013] Furthermore, the design of the visual data processing workflow orchestration tool in step S1 specifically includes constructing a visual component library containing common data processing operations based on a predefined data processing operator library, with each component corresponding to a data processing function;

[0014] Design a visual drag-and-drop interface that allows users to select the required data processing components from the component library and drag them into the workflow orchestration area. In the workflow orchestration area, connect the various data processing components according to business logic using connection lines.

[0015] Design a data flow mechanism to ensure that data is passed and processed between components in a predefined logical order, and the output of the previous component automatically becomes the input of the next component;

[0016] Design a task scheduling mechanism to automatically generate task scheduling scripts based on user-defined data processing flows, and control the execution order and data flow of each data processing component;

[0017] The introduction of a real-time preview function allows users to view the output of each processing node in real time during the orchestration of data processing workflows, so as to adjust and optimize the processing logic in a timely manner.

[0018] Furthermore, step S2 also includes automatically generating corresponding data processing logic code based on the data processing flowchart;

[0019] By introducing NLP technology, users can describe their data processing needs in natural language, and the natural language instructions are converted into corresponding data processing flow configurations.

[0020] The dependencies between tasks are represented using a directed acyclic graph, and the execution order of tasks is determined by a task scheduling algorithm.

[0021] Furthermore, the process of filtering the original data that meets the conditions in step S3 is as follows: obtain user configuration information, determine parameters such as data type, result identifier, data identifier and manual input value, establish the initial filtering conditions for the original data based on the obtained user configuration information, and monitor the accuracy, integrity and other indicators of the data in real time by setting data quality monitoring rules.

[0022] Iterate through the original data, make conditional judgments for each data point, and clean the data in the initial filtering result set according to predefined data quality rules to remove noisy data;

[0023] An incremental data processing approach is adopted to process newly added or changed data in real time, avoiding redundant calculations;

[0024] The cleaned data is formatted and structured according to preset conversion rules, a data lineage diagram is established, the entire process from data collection to output is recorded, and the converted data subset that meets the conditions is used as the input for the next data processing step.

[0025] Furthermore, in step S4, it is determined whether data needs to be dynamically calculated over time. If the conditions change, the configured data is dynamically obtained from the data warehouse based on the time values ​​stored in the database, and the allocation of computing resources is automatically adjusted according to the complexity of the data changes and the computing requirements, in conjunction with adaptive computing resource allocation.

[0026] Specifically, based on preset business rules, it is determined whether the triggering conditions for condition changes are met. If they are met, the real-time calculation process is initiated; otherwise, the current data state is maintained.

[0027] Retrieve configuration parameters related to the current time value from the database, determine the target data range and granularity to be extracted from the data warehouse, adopt an incremental data synchronization strategy, dynamically retrieve changed data from the data warehouse, and merge and update it with existing local data to obtain the updated complete dataset;

[0028] By analyzing the complexity of data changes and the resource requirements of computing tasks, the allocation ratio of computing resources is dynamically adjusted. For the updated complete dataset, a classification model of data changes is constructed using the decision tree algorithm to predict the data change trend in the future. Based on the prediction results of the data change trend, the time cycle of data acquisition and computing is dynamically adjusted.

[0029] Furthermore, in step S5, a real-time data processing engine is introduced. By listening to data change events, corresponding data processing tasks are triggered. If a data change is detected, the changed data is obtained, and the data processing task to be executed is determined.

[0030] According to the predefined data processing algorithm, the acquired real-time data is cleaned and noisy data is removed. If the data quality does not meet the requirements, the data cleaning process is triggered to obtain clean data.

[0031] Data transformation rules are used to convert and structure the cleaned data. By parsing the data structure and field meanings, and based on the preset transformation template, the data format is standardized, and it is determined whether the transformed data meets the requirements of subsequent analysis.

[0032] Using an event-driven architecture, the execution order of tasks is determined by a task scheduling algorithm based on the priority of data processing tasks.

[0033] Obtain task dependencies, construct a directed acyclic graph to obtain the task execution topology, and use a stream computing framework to perform real-time filtering, transformation and aggregation operations on the transformed data during real-time task processing. Select appropriate computation functions and window types according to data characteristics and business needs, and select machine learning algorithms for real-time data analysis for different data processing scenarios.

[0034] It also includes step S6, which specifically involves dynamically adjusting the task execution order and resource allocation based on the data processing task dependencies and resource consumption.

[0035] Furthermore, the process of dynamically adjusting the task execution order and resource allocation in step S6 specifically involves obtaining the dependencies and resource consumption of data processing tasks, constructing a task dependency graph and a resource consumption model, establishing directed edges to connect relevant task nodes in the task dependency graph if there are dependencies between tasks, using a heuristic algorithm to generate an initial task execution order and resource allocation scheme based on the task dependency graph and resource consumption model, and obtaining an optimized scheduling scheme through iterative optimization.

[0036] By introducing deep reinforcement learning algorithms, the task scheduling problem is modeled as a Markov decision process. Through the interaction between the agent and the environment, the optimal task execution order and resource allocation strategy are dynamically learned. During the task execution process, the task progress and resource consumption are monitored in real time. If the task execution progress is found to be lagging or the resource utilization is low, a dynamic adjustment mechanism is triggered to re-optimize the task execution order and resource allocation for independent data processing tasks.

[0037] A parallel execution strategy is adopted to make full use of resources. By executing tasks in parallel, the efficiency of task execution is improved. For tasks with dependencies, they are executed sequentially according to the optimized execution order. The execution order of tasks is determined according to the task dependency graph, and task priorities are set according to the urgency and importance of the tasks.

[0038] After setting task priorities in step S6, resources are allocated to high-priority tasks during resource contention. A monitoring platform for task execution and resource consumption is built to collect and analyze task execution data in real time. By evaluating the effectiveness of the scheduling algorithm, the scheduling strategy and algorithm parameters are continuously optimized to achieve adaptive intelligent task scheduling.

[0039] Based on historical and real-time monitoring data, the task execution order and resource allocation are dynamically adjusted. Deep learning algorithms are used to predict the task execution status and resource consumption trends in the future, and resource scheduling and task optimization are carried out in advance.

[0040] An industrial internet data processing system based on big data includes a visual process orchestration module, a data preprocessing module, a real-time data processing engine, a task scheduling and resource management module, and an intelligent decision support module.

[0041] The visual workflow orchestration module provides a user interface that allows users to define data processing workflows by dragging and dropping components. The visual workflow orchestration module includes a visual component library from which users can select data processing components and set data types and processing conditions. The visual workflow orchestration module is responsible for generating task scheduling scripts and controlling the execution order of data processing components.

[0042] The data preprocessing module is responsible for initial screening and filtering of data based on user-configured information, establishing data screening conditions, performing data quality monitoring, and cleaning and format conversion of data to ensure that only data that meets the conditions is further processed and that the data quality meets the requirements of subsequent processing.

[0043] The real-time data processing engine listens for data change events, triggers corresponding data processing tasks, and performs real-time data acquisition, cleaning, transformation, and analysis.

[0044] The task scheduling and resource management module dynamically adjusts the task execution order and resource allocation based on the dependencies and resource consumption of data processing tasks. This includes the construction of a task dependency graph, the establishment of a resource consumption model, the application of deep reinforcement learning algorithms, and the monitoring of task execution and resource consumption, in order to achieve intelligent task scheduling and resource optimization.

[0045] The intelligent decision support module uses deep learning algorithms to predict task execution and resource consumption trends, performs resource scheduling and task optimization in advance, and provides decision support by analyzing historical data and real-time monitoring data, helping the system to adaptively adjust the task execution order and resource allocation to improve overall processing efficiency and response speed.

[0046] Compared with the prior art, the present invention has the following advantages:

[0047] 1. This invention designs a drag-and-drop interface, allowing users to flexibly define data processing logic and steps, and utilize a predefined operator library to perform common data operations. By introducing NLP technology, it allows users to describe processing requirements in natural language, which is automatically converted into corresponding configurations, thus lowering the technical threshold. During data processing, this invention performs initial condition filtering based on data type information and supports dynamic calculation over time. It also introduces a real-time data processing engine and combines it with an event-driven architecture to improve response speed.

[0048] 2. This invention also employs an intelligent task scheduling algorithm to dynamically adjust the execution order and resource allocation based on dependencies and resource consumption. This configuration-based approach reduces manual coding work, allowing non-technical personnel to participate in configuration and monitoring. Through innovations such as a visual interface, NLP interaction, real-time processing, and intelligent scheduling, this invention significantly improves the flexibility, efficiency, and ease of use of data processing, reduces development and maintenance costs, and provides an efficient and convenient solution for complex data processing tasks. Attached Figure Description

[0049] Figure 1 This is a flowchart illustrating an industrial internet data processing method based on big data, as described in Example 1.

[0050] Figure 2 This is a flowchart of step S6 in Example 1, which involves dynamically adjusting the task execution order and resource allocation.

[0051] Figure 3 Example 2 shows a system framework diagram of an industrial internet data processing system based on big data. Detailed Implementation

[0052] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0053] Example

[0054] like Figure 1 As shown, this embodiment provides a data processing method for the industrial internet based on big data, including the following steps:

[0055] S1. Design a visual data processing workflow orchestration tool that allows users to flexibly define the logic and steps of data processing. During the workflow orchestration process, common data processing operations can be implemented through a predefined data processing operator library.

[0056] S2. Through the visual configuration interface, users can select the required data processing components by dragging and dropping, and set the corresponding data types and processing conditions to form a complete data processing flowchart.

[0057] S3. Obtain the data type information configured by the user, and perform initial condition filtering judgment on the data based on this information to select the original data that meets the conditions.

[0058] S4. Determine whether data needs to be dynamically calculated over time based on whether the conditions change.

[0059] S5 introduces a real-time data processing engine to collect, clean, transform, and analyze data in real time.

[0060] Furthermore, the design of the visual data processing workflow orchestration tool in step S1 specifically includes constructing a visual component library containing common data processing operations based on a predefined data processing operator library, with each component corresponding to a data processing function;

[0061] Design a visual drag-and-drop interface that allows users to select the required data processing components from the component library and drag them into the workflow orchestration area. In the workflow orchestration area, connect the various data processing components according to business logic using connection lines.

[0062] Design a data flow mechanism to ensure that data is passed and processed between components in a predefined logical order, and the output of the previous component automatically becomes the input of the next component;

[0063] Design a task scheduling mechanism to automatically generate task scheduling scripts based on user-defined data processing flows, and control the execution order and data flow of each data processing component;

[0064] A real-time preview function is introduced, allowing users to view the output of each processing node in real time during the orchestration of data processing workflows, so as to adjust and optimize the processing logic in a timely manner;

[0065] When building a visualization component library, you can design corresponding components based on common data processing operations, such as data cleaning, feature engineering, and data integration. For example, for a data cleaning component, you can build in commonly used functions such as data deduplication, missing value handling, and outlier filtering.

[0066] In the drag-and-drop interface, users can add the required components to the workflow orchestration area through simple drag-and-drop operations, and connect the components according to business logic using connecting lines. For each component, users can set algorithm parameters according to actual needs through the parameter configuration interface, such as setting the threshold for data deduplication, the method for filling missing values, etc. During the data flow process, the output of the previous component is automatically used as the input of the next component to ensure smooth data transfer between components.

[0067] Using directed acyclic graphs and topological sorting algorithms, task scheduling scripts are automatically generated to control the execution order of components;

[0068] The real-time preview function, through the establishment of data caching and incremental calculation mechanism, allows users to view the output results of each processing node in real time, which facilitates timely optimization of processing logic;

[0069] A real-time data processing engine is introduced to support the collection, cleaning, transformation, and analysis of real-time data. It utilizes predefined data processing algorithms, such as real-time data stream processing algorithms (e.g., Flink, Spark Streaming), to quickly respond to business needs. Finally, a visual monitoring panel displays the running status, performance indicators (e.g., throughput, latency), and anomalies of the data processing process. Based on the monitoring results, resource allocation is dynamically adjusted to optimize system performance.

[0070] Furthermore, step S2 also includes automatically generating corresponding data processing logic code based on the data processing flowchart;

[0071] By introducing NLP technology, users can describe their data processing needs in natural language, and the natural language instructions are converted into corresponding data processing flow configurations.

[0072] The dependencies between tasks are represented using a directed acyclic graph, and the execution order of tasks is determined by a task scheduling algorithm.

[0073] When building a visualization component library, common data processing operators can be predefined, such as filtering, joining, and aggregation, with each operator corresponding to a visualization component;

[0074] When designing a drag-and-drop interface, a flowchart-like layout can be used. Users can drag components onto the canvas with the mouse and connect them with connecting lines. In the parameter configuration interface, drop-down menus or input boxes can be provided for each component, allowing users to select data types (such as strings, integers, floating-point numbers, etc.) and set parameter values ​​(such as filter conditions, aggregate functions, etc.).

[0075] When introducing natural language processing technology, algorithms such as named entity recognition and relation extraction can be used to convert user-input natural language commands into structured data processing configurations; the data flow mechanism can be implemented through message queues or data structures in memory to ensure that data is passed between components in a predefined order; the task scheduling engine can use directed acyclic graphs to represent the dependencies between tasks, determine the execution order of tasks through topological sorting algorithms, and generate corresponding scheduling scripts.

[0076] The real-time preview function allows you to add listeners to the output of each component, display intermediate results on the interface in real time, and set thresholds for data quality and processing time. When the thresholds are exceeded, an alarm is triggered.

[0077] Furthermore, the process of filtering the original data that meets the conditions in step S3 is as follows: obtain user configuration information, determine parameters such as data type, result identifier, data identifier and manual input value, establish the initial filtering conditions for the original data based on the obtained user configuration information, and monitor the accuracy, integrity and other indicators of the data in real time by setting data quality monitoring rules.

[0078] Iterate through the original data, make conditional judgments for each data point, and clean the data in the initial filtering result set according to predefined data quality rules to remove noisy data;

[0079] An incremental data processing approach is adopted to process newly added or changed data in real time, avoiding redundant calculations;

[0080] The cleaned data is formatted and structured according to preset conversion rules, a data lineage diagram is established, the entire process of data collection and output is recorded, and the converted data subset that meets the conditions is used as the input for the next data processing step.

[0081] When obtaining user configuration information, users can select parameters such as data type and result identifier through a graphical interface. For example, users can select "sales data" as the data type, "sales amount" as the result identifier, "product number" as the data identifier, and manually enter "2022" as the time range.

[0082] Based on this configuration information, an initial filtering condition can be constructed, such as data type = sales data AND time ≥ 2022-01-01 AND time ≤ 2022-12-31;

[0083] When monitoring data in real time, a data integrity rule can be set, such as "the product number of the sales record cannot be empty". If more than 5% of the records are found to be inconsistent with this rule, an alarm will be triggered.

[0084] When iterating through the original data, you can use conditional statements, such as "IF data type" = "sales data" AND time ≥ "2022-01-01" AND time ≤ "2022-12-31 THEN" to add it to the initial filtered result set;

[0085] When cleaning the initial filtered result set, some outliers can be removed, such as records with negative sales quantities.

[0086] In incremental data processing, the timestamps of new data can be compared with those of previously processed data, and only data with more recent timestamps can be processed. During data transformation, sales figures can be converted from string format to numerical format, and the time can be structured according to the "year-month" format. The data lineage graph can record information such as the source, cleaning process, and transformation rules of each sales record, which is convenient for tracing during subsequent data analysis. Finally, the transformed data is used as input for data analysis algorithms, such as using the random forest algorithm to predict sales figures for the next month.

[0087] Furthermore, in step S4, it is determined whether data needs to be dynamically calculated over time. If the conditions change, the configured data is dynamically obtained from the data warehouse based on the time values ​​stored in the database, and the allocation of computing resources is automatically adjusted according to the complexity of the data changes and the computing requirements, in conjunction with adaptive computing resource allocation.

[0088] Specifically, based on preset business rules, it is determined whether the triggering conditions for condition changes are met. If they are met, the real-time calculation process is initiated; otherwise, the current data state is maintained.

[0089] Retrieve configuration parameters related to the current time value from the database, determine the target data range and granularity to be extracted from the data warehouse, adopt an incremental data synchronization strategy, dynamically retrieve changed data from the data warehouse, and merge and update it with existing local data to obtain the updated complete dataset;

[0090] By analyzing the complexity of data changes and the resource requirements of computing tasks, the allocation ratio of computing resources is dynamically adjusted. For the updated complete dataset, a classification model of data changes is constructed using the decision tree algorithm to predict the data change trend in the future. Based on the prediction results of the data change trend, the time cycle of data acquisition and computing is dynamically adjusted.

[0091] According to the preset business rules, if the order amount exceeds 10,000 yuan and the customer's credit score is below 60, real-time calculation is triggered. The configuration parameters of the previous hour are obtained from the database to determine the order and customer data of the most recent 24 hours that need to be extracted from the data warehouse.

[0092] Incremental data synchronization is adopted, and changed data is retrieved from the data warehouse every 5 minutes and merged with local data for updating. By analyzing the complexity of data changes, such as when order data increases by 20% and customer data increases by 10%, the CPU resource allocation ratio is dynamically adjusted to 70% and the memory resource allocation ratio is adjusted to 60%.

[0093] For the updated complete dataset, a data change classification model is constructed using the C4.5 decision tree algorithm to predict the data change trend within the next hour. Based on the prediction results, if the data change rate accelerates, the data acquisition and calculation time cycle is adjusted to 2 minutes. Key indicators during task execution are acquired, such as data processing speed reaching 1000 records / second and resource utilization reaching 80%, and are displayed in real time through a visualization interface. If the data processing time exceeds the threshold by 20 seconds, predefined alarm rules are triggered to notify the relevant responsible persons.

[0094] The optimized data calculation results are stored in a MySQL database, and order analysis reports and customer credit analysis reports are generated according to business needs, which are then visualized through a web page.

[0095] Furthermore, in step S5, a real-time data processing engine is introduced. By listening to data change events, corresponding data processing tasks are triggered. If a data change is detected, the changed data is obtained, and the data processing task to be executed is determined.

[0096] According to the predefined data processing algorithm, the acquired real-time data is cleaned and noisy data is removed. If the data quality does not meet the requirements, the data cleaning process is triggered to obtain clean data.

[0097] Data transformation rules are used to convert and structure the cleaned data. By parsing the data structure and field meanings, and based on the preset transformation template, the data format is standardized, and it is determined whether the transformed data meets the requirements of subsequent analysis.

[0098] Using an event-driven architecture, the execution order of tasks is determined by a task scheduling algorithm based on the priority of data processing tasks.

[0099] The process involves obtaining task dependencies, constructing a directed acyclic graph (DAG), and obtaining the task execution topology. During real-time task processing, a stream computing framework is employed to perform real-time filtering, transformation, and aggregation operations on the converted data. Based on data characteristics and business requirements, appropriate computation functions and window types are selected. For different data processing scenarios, machine learning algorithms are chosen for real-time data analysis; for classification problems, decision tree algorithms are used; for prediction problems, linear regression algorithms are used; and for clustering problems, K-means algorithms are used.

[0100] Specifically, the Apache Flink real-time data processing engine is introduced, using the CDC (ChangeDataCapture) mechanism to monitor the binlog of the MySQL database and capture data change events in real time. When data updates are detected, the Flink job automatically triggers predefined data processing tasks, such as real-time cleaning of newly added user logs, using a Bloom filter algorithm to filter out duplicate data, and using regular expressions to validate field formats and remove invalid data. The cleaned data is then structured according to a preset Avro format, defining the table structure using RowTypeInfo and using MapFunction for field mapping and type conversion. A Directed Acyclic Graph (DAG) is then used to define task dependencies, and a priority queue algorithm is used to determine the execution order, prioritizing latency-sensitive real-time analysis tasks. The transformed data is then processed using the FlinkCEP library for complex event processing, defining event patterns using PatternAPI, filtering out user behavior events that meet the conditions in real time, and using the incremental aggregation function ReduceFunction for statistical analysis of the events. For user segmentation scenarios, the K-means clustering algorithm is selected, using Euclidean distance to measure user feature similarity, iteratively optimizing cluster centers to obtain user segmentation results.

[0101] This step may also include: acquiring key data processing metrics, such as data processing speed and resource utilization, and displaying them in real time on a visualization interface. If any abnormality is found in the metrics, such as processing speed being lower than the threshold or resource utilization being too high, a predefined alarm rule will be triggered to notify the relevant responsible persons to handle the situation in a timely manner.

[0102] By establishing a data lineage diagram, the entire process of data from collection to output is recorded, including data source, processing logic, intermediate results and final output. Through data lineage analysis, the root cause of data quality problems can be determined, the data change process can be traced, and the traceability of data processing can be improved.

[0103] An incremental data processing approach is adopted to process newly added or changed data in real time, avoiding redundant calculations. Based on the data update time or version number, it is determined whether the data needs to be reprocessed, ensuring the real-time nature and accuracy of the data processing results.

[0104] Specifically, the analysis process monitors key data processing metrics, such as records per second (RecordsperSecond) and latency. When the processing speed drops below 100 records per second or the 99th percentile latency exceeds 1 second, an alarm is triggered. Finally, the FlinkDataStreamAPI's process method is used to construct a data lineage graph, recording each step of the data transformation operation to achieve end-to-end data traceability. For incremental data, Flink uses the timestamp of the Kafka message or the update time of the MySQL table to determine whether the data has been updated, processing only newly added or changed data to avoid redundant calculations and improve real-time performance.

[0105] like Figure 2 As shown, this method also includes step S6, which specifically involves dynamically adjusting the task execution order and resource allocation based on the data processing task dependencies and resource consumption.

[0106] Furthermore, the process of dynamically adjusting the task execution order and resource allocation in step S6 specifically involves obtaining the dependencies and resource consumption of data processing tasks, constructing a task dependency graph and a resource consumption model, establishing directed edges to connect relevant task nodes in the task dependency graph if there are dependencies between tasks, using a heuristic algorithm to generate an initial task execution order and resource allocation scheme based on the task dependency graph and resource consumption model, and obtaining an optimized scheduling scheme through iterative optimization.

[0107] By introducing deep reinforcement learning algorithms, the task scheduling problem is modeled as a Markov decision process. Through the interaction between the agent and the environment, the optimal task execution order and resource allocation strategy are dynamically learned. During the task execution process, the task progress and resource consumption are monitored in real time. If the task execution progress is found to be lagging or the resource utilization is low, a dynamic adjustment mechanism is triggered to re-optimize the task execution order and resource allocation for independent data processing tasks.

[0108] A parallel execution strategy is adopted to make full use of resources. By executing tasks in parallel, the efficiency of task execution is improved. For tasks with dependencies, they are executed sequentially according to the optimized execution order. The execution order of tasks is determined according to the task dependency graph, and task priorities are set according to the urgency and importance of the tasks.

[0109] After setting task priorities in step S6, resources are allocated to high-priority tasks during resource contention. A monitoring platform for task execution and resource consumption is built to collect and analyze task execution data in real time. By evaluating the effectiveness of the scheduling algorithm, the scheduling strategy and algorithm parameters are continuously optimized to achieve adaptive intelligent task scheduling.

[0110] Based on historical and real-time monitoring data, the task execution order and resource allocation are dynamically adjusted. Deep learning algorithms are used to predict the task execution status and resource consumption trends in the future, and resource scheduling and task optimization are carried out in advance.

[0111] Specifically, when obtaining the dependencies and resource consumption of data processing tasks, a directed acyclic graph (DAG) can be used to represent the dependencies between tasks. Nodes represent tasks, edges represent dependencies, and the weight of an edge can represent the amount of data transferred between tasks or the task execution time.

[0112] By analyzing historical task execution data, a resource consumption model can be established, such as using a linear regression model, to predict the CPU, memory, I / O resources required for task execution based on characteristics such as the amount of input data and computational complexity. When using genetic algorithms to optimize task scheduling schemes, the task execution order and resource allocation scheme can be encoded as gene sequences. Through operations such as selection, crossover, and mutation, a highly fit scheduling scheme can be obtained through iterative optimization. For example, using binary encoding, each task corresponds to a gene bit, with 0 indicating unscheduled and 1 indicating scheduled. The optimal task scheduling order can then be searched using a genetic algorithm.

[0113] When applying deep reinforcement learning algorithms, the task scheduling problem can be modeled as a Markov decision process. The state represents the current task execution progress and resource usage; the action is selecting the next task to execute and allocating resources; and the reward is a weighted sum of task execution efficiency and resource utilization. Through continuous interaction with the environment, the agent learns the optimal scheduling strategy. When triggering a dynamic adjustment mechanism, thresholds for task execution progress and resource utilization can be set. For example, if the task execution progress is below 50% and the resource utilization is below 60%, rescheduling is triggered.

[0114] For independent tasks, tasks can be broken down into multiple subtasks and assigned to different CPU cores or GPU devices for parallel execution, such as using the MapReduce framework for distributed computing. For tasks with dependencies, the execution order can be determined by using a topological sorting algorithm based on the task dependency graph, such as the Kahn algorithm or the DFS algorithm.

[0115] When setting task priorities, you can calculate the task priority score based on factors such as the task's impact on the business and the deadline. For example, when using the Weighted Shortest Job First (WSJF) algorithm, the priority score = business value / task duration. When monitoring task execution in real time, you can use tools such as ELK (Elasticsearch, Logstash, Kibana) to collect task execution logs, analyze indicators such as task execution progress and resource consumption in real time, and display the effect of the scheduling algorithm through a dashboard.

[0116] Based on historical and real-time monitoring data, LSTM networks can be used to predict task execution and resource consumption trends over a future period. For example, based on task execution data from the past 24 hours, the task completion rate and resource utilization rate can be predicted for the next hour, allowing for advance resource scheduling and task optimization.

[0117] Example 2

[0118] like Figure 3 As shown, this embodiment provides an industrial internet data processing system based on big data, which is used to implement an industrial internet data processing method based on big data, including a visual process orchestration module, a data preprocessing module, a real-time data processing engine, a task scheduling and resource management module, and an intelligent decision support module;

[0119] The visual workflow orchestration module provides a user interface that allows users to define data processing workflows by dragging and dropping components. The visual workflow orchestration module includes a visual component library from which users can select data processing components and set data types and processing conditions. The visual workflow orchestration module is responsible for generating task scheduling scripts and controlling the execution order of data processing components.

[0120] The data preprocessing module is responsible for initial screening and filtering of data based on user-configured information, establishing data screening conditions, performing data quality monitoring, and cleaning and format conversion of data to ensure that only data that meets the conditions is further processed and that the data quality meets the requirements of subsequent processing.

[0121] The real-time data processing engine listens for data change events, triggers corresponding data processing tasks, and performs real-time data acquisition, cleaning, transformation, and analysis.

[0122] The task scheduling and resource management module dynamically adjusts the task execution order and resource allocation based on the dependencies and resource consumption of data processing tasks. This includes the construction of a task dependency graph, the establishment of a resource consumption model, the application of deep reinforcement learning algorithms, and the monitoring of task execution and resource consumption, in order to achieve intelligent task scheduling and resource optimization.

[0123] The intelligent decision support module uses deep learning algorithms to predict task execution and resource consumption trends, performs resource scheduling and task optimization in advance, and provides decision support by analyzing historical data and real-time monitoring data, helping the system to adaptively adjust the task execution order and resource allocation to improve overall processing efficiency and response speed.

[0124] In this embodiment, the visual process orchestration module provides an intuitive drag-and-drop user interface, which allows users to define data processing processes in a graphical way. The interface displays icons of various data processing components, and users can build processes by dragging and dropping these icons into the workspace.

[0125] Build a component library that includes various data processing operations, such as data cleaning, feature engineering, and data integration. Each component has a predefined algorithm and parameter setting interface, and users can configure these parameters as needed.

[0126] Based on the user-defined process, task scheduling scripts are automatically generated. These scripts define the execution order of data processing components, use a directed acyclic graph (DAG) to represent the dependencies between tasks, and employ a topological sorting algorithm to determine the execution order.

[0127] The data preprocessing module performs preliminary screening of the raw data based on the filtering conditions configured by the user, removing data that does not meet the conditions, and monitors the accuracy and completeness of the data in real time to ensure that the data quality meets the preset standards. The filtered data is then cleaned, including removing outliers and filling in missing values, and the data is converted into a format suitable for analysis, such as converting dates from strings to date types.

[0128] Real-time data processing engines use message queues or database change data capture (CDC) technology to listen for data change events, collect data in real time from various data sources such as sensors, databases, and APIs, clean and transform the real-time collected data to ensure data quality, and apply real-time analysis algorithms, such as stream processing algorithms, to perform instant data analysis.

[0129] The task scheduling and resource management module constructs a task dependency graph to represent the dependencies between tasks, establishes a resource consumption model to predict the resources required for task execution, applies deep reinforcement learning algorithms to dynamically learn the optimal task execution order and resource allocation strategy, monitors task progress and resource consumption in real time, and adjusts task scheduling and resource allocation based on the monitoring results.

[0130] The intelligent decision support module utilizes deep learning algorithms, such as LSTM networks, to predict task execution and resource consumption trends, analyze historical and real-time monitoring data, and provide decision support to help the system adaptively adjust task execution order and resource allocation. Based on prediction results and real-time data, it automatically adjusts task execution order and resource allocation to improve processing efficiency and response speed.

[0131] Users define data processing flows through the visual process orchestration module. The generated task scheduling scripts are used by the task scheduling and resource management module to control the execution of the data preprocessing module and the real-time data processing engine. The data processed by the data preprocessing module is further processed by the real-time data processing engine. The output of the real-time data processing engine is used by the intelligent decision support module for prediction and optimization. The task scheduling and resource management module dynamically adjusts task and resource allocation based on the prediction results of the intelligent decision support module.

[0132] The specific embodiments of the invention have been described in detail above, but these are merely examples. The invention is not limited to the specific embodiments described above. Those skilled in the art should understand that the embodiments and descriptions in the specification are only illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A data processing method for the industrial internet based on big data, characterized in that, Includes the following steps: S1. Design a visual data processing workflow orchestration tool that allows users to flexibly define the logic and steps of data processing. During the workflow orchestration process, common data processing operations can be implemented through a predefined data processing operator library. S2. Through the visual configuration interface, users can select the required data processing components by dragging and dropping, and set the corresponding data types and processing conditions to form a complete data processing flowchart. S3. Obtain the data type information configured by the user, and perform initial condition filtering judgment on the data based on this information to select the original data that meets the conditions. S4. Determine whether data needs to be dynamically calculated over time based on whether the conditions change. S5. Introduces a real-time data processing engine to perform real-time data acquisition, cleaning, transformation, and analysis; S6. Dynamically adjust the task execution order and resource allocation based on the data processing task dependencies and resource consumption. The process of dynamically adjusting the task execution order and resource allocation in S6 is as follows: obtain the dependency relationship and resource consumption of data processing tasks, construct a task dependency graph and resource consumption model. If there is a dependency relationship between tasks, establish directed edges to connect the relevant task nodes in the task dependency graph. Using a heuristic algorithm, generate an initial task execution order and resource allocation scheme based on the task dependency graph and resource consumption model. Through iterative optimization, obtain the optimized scheduling scheme. By introducing deep reinforcement learning algorithms, the task scheduling problem is modeled as a Markov decision process. Through the interaction between the agent and the environment, the optimal task execution order and resource allocation strategy are dynamically learned. During the task execution process, the task progress and resource consumption are monitored in real time. If the task execution progress is found to be lagging or the resource utilization is low, a dynamic adjustment mechanism is triggered to re-optimize the task execution order and resource allocation for independent data processing tasks. A parallel execution strategy is adopted to make full use of resources. By executing tasks in parallel, the efficiency of task execution is improved. For tasks with dependencies, they are executed sequentially according to the optimized execution order. The execution order of tasks is determined according to the task dependency graph, and task priorities are set according to the urgency and importance of the tasks. Based on historical and real-time monitoring data, the task execution order and resource allocation are dynamically adjusted. Deep learning algorithms are used to predict the task execution status and resource consumption trends in the future, and resource scheduling and task optimization are carried out in advance.

2. The industrial internet data processing method based on big data according to claim 1, characterized in that: The design of the visual data processing workflow orchestration tool in step S1 specifically includes: constructing a visual component library containing common data processing operations based on a predefined data processing operator library, with each component corresponding to a data processing function; Design a visual drag-and-drop interface that allows users to select the required data processing components from the component library and drag them into the workflow orchestration area. In the workflow orchestration area, connect the various data processing components according to business logic using connection lines. Design a data flow mechanism to ensure that data is passed and processed between components in a predefined logical order, and the output of the previous component automatically becomes the input of the next component; Design a task scheduling mechanism to automatically generate task scheduling scripts based on user-defined data processing flows, and control the execution order and data flow of each data processing component; The introduction of a real-time preview function allows users to view the output of each processing node in real time during the orchestration of data processing workflows, so as to adjust and optimize the processing logic in a timely manner.

3. The industrial internet data processing method based on big data according to claim 1, characterized in that: Step S2 further includes automatically generating corresponding data processing logic code based on the data processing flowchart; By introducing NLP technology, users can describe their data processing needs in natural language, and the natural language instructions are converted into corresponding data processing flow configurations. The dependencies between tasks are represented using a directed acyclic graph, and the execution order of tasks is determined by a task scheduling algorithm.

4. The industrial internet data processing method based on big data according to claim 1, characterized in that: The process of filtering the original data that meets the conditions in step S3 is as follows: obtain user configuration information, determine parameters such as data type, result identifier, data identifier and manual input value, establish the initial filtering conditions for the original data based on the obtained user configuration information, and monitor the accuracy, integrity and other indicators of the data in real time by setting data quality monitoring rules. Iterate through the original data, make conditional judgments for each data point, and clean the data in the initial filtering result set according to predefined data quality rules to remove noisy data; An incremental data processing approach is adopted to process newly added or changed data in real time, avoiding redundant calculations; The cleaned data is formatted and structured according to preset conversion rules, a data lineage diagram is established, the entire process from data collection to output is recorded, and the converted data subset that meets the conditions is used as the input for the next data processing step.

5. The industrial internet data processing method based on big data according to claim 1, characterized in that: In step S4, it is determined whether data needs to be dynamically calculated over time. If the conditions change, the configured data is dynamically obtained from the data warehouse based on the time value stored in the database, and the allocation of computing resources is automatically adjusted according to the complexity of the data changes and the computing requirements, in conjunction with adaptive computing resource allocation. Specifically, based on preset business rules, it is determined whether the triggering conditions for condition changes are met. If they are met, the real-time calculation process is initiated; otherwise, the current data state is maintained. Retrieve configuration parameters related to the current time value from the database, determine the target data range and granularity to be extracted from the data warehouse, adopt an incremental data synchronization strategy, dynamically retrieve changed data from the data warehouse, and merge and update it with existing local data to obtain the updated complete dataset; By analyzing the complexity of data changes and the resource requirements of computing tasks, the allocation ratio of computing resources is dynamically adjusted. For the updated complete dataset, a classification model of data changes is constructed using the decision tree algorithm to predict the data change trend in the future. Based on the prediction results of the data change trend, the time cycle of data acquisition and computing is dynamically adjusted.

6. The industrial internet data processing method based on big data according to claim 1, characterized in that: In step S5, a real-time data processing engine is introduced. By listening to data change events, corresponding data processing tasks are triggered. If a data change is detected, the changed data is obtained, and the data processing task to be executed is determined. According to the predefined data processing algorithm, the acquired real-time data is cleaned and noisy data is removed. If the data quality does not meet the requirements, the data cleaning process is triggered to obtain clean data. Data transformation rules are used to convert and structure the cleaned data. By parsing the data structure and field meanings, and based on the preset transformation template, the data format is standardized, and it is determined whether the transformed data meets the requirements of subsequent analysis. Using an event-driven architecture, the execution order of tasks is determined by a task scheduling algorithm based on the priority of data processing tasks. Obtain task dependencies, construct a directed acyclic graph to obtain the task execution topology, and use a stream computing framework to perform real-time filtering, transformation and aggregation operations on the transformed data during real-time task processing. Select appropriate computation functions and window types according to data characteristics and business needs, and select machine learning algorithms for real-time data analysis for different data processing scenarios.

7. The industrial internet data processing method based on big data according to claim 1, characterized in that: After setting task priorities in step S6, resources are allocated to high-priority tasks during resource contention. A monitoring platform for task execution and resource consumption is built to collect and analyze task execution data in real time. By evaluating the effectiveness of the scheduling algorithm, the scheduling strategy and algorithm parameters are continuously optimized to achieve adaptive intelligent task scheduling.

8. A big data-based industrial internet data processing system, used to implement the big data-based industrial internet data processing method described in claim 1, characterized in that, It includes a visual process orchestration module, a data preprocessing module, a real-time data processing engine, a task scheduling and resource management module, and an intelligent decision support module; The visual workflow orchestration module provides a user interface that allows users to define data processing workflows by dragging and dropping components. The visual workflow orchestration module includes a visual component library from which users can select data processing components and set data types and processing conditions; The visual workflow orchestration module is responsible for generating task scheduling scripts and controlling the execution order of data processing components; The data preprocessing module is responsible for initial screening and filtering of data based on user-configured information, establishing data screening conditions, performing data quality monitoring, and cleaning and format conversion of data to ensure that only data that meets the conditions is further processed and that the data quality meets the requirements of subsequent processing. The real-time data processing engine listens for data change events, triggers corresponding data processing tasks, and performs real-time data acquisition, cleaning, transformation, and analysis. The task scheduling and resource management module dynamically adjusts the task execution order and resource allocation based on the dependencies and resource consumption of data processing tasks. This includes the construction of a task dependency graph, the establishment of a resource consumption model, the application of deep reinforcement learning algorithms, and the monitoring of task execution and resource consumption, in order to achieve intelligent task scheduling and resource optimization. The intelligent decision support module uses deep learning algorithms to predict task execution and resource consumption trends, performs resource scheduling and task optimization in advance, and provides decision support by analyzing historical data and real-time monitoring data, helping the system to adaptively adjust the task execution order and resource allocation to improve overall processing efficiency and response speed.

Images