Industrial internet data processing method and system based on big data

Abstract

The invention discloses an industrial internet data processing method based on big data, which comprises the steps of designing a visual data processing flow arrangement tool, allowing a user to flexibly define logic and steps of data processing, and realizing common data processing operation through a predefined data processing operator library in the flow arrangement process. A user is allowed to select a required data processing component through dragging operation through a visual configuration interface, and corresponding data types and processing conditions are set to form a complete data processing flow chart; acquiring data type information configured by a user, performing primary condition filtering judgment on the data according to the information, and screening out original data meeting conditions; judging whether data needs to be dynamically calculated along with time or not according to whether conditions are changed or not; a real-time data processing engine is introduced, and data are collected, cleaned, converted and analyzed in real time; through a visual interface, NLP interaction, real-time processing and intelligent scheduling, the flexibility of data processing is improved.

Description

Technical Field

[0001] The present invention relates to the field of big data processing technology, and in particular to an industrial Internet data processing method and system based on big data. Background Art

[0002] In the modern information society, big data processing and business data presentation are key parts of enterprise management and decision-making. With the advancement of equipment data collection technology, the ability to process and analyze these massive data has become crucial. Traditional data processing methods usually rely on writing fixed codes to handle specific data formats and computing requirements, which has certain limitations in flexibility and efficiency.

[0003] Currently, data processing relies on writing and maintaining specific data processing codes, such as Python and Java, which require highly skilled developers to develop and maintain. This is not only costly, but also difficult to cope with rapidly changing business needs. In addition, the code development cycle is long, which is not conducive to flexible adjustment.

[0004] Lack of flexibility: The existing system requires manual intervention or customized development for scenarios with high real-time requirements and dynamic conditions for task scheduling to obtain historical data. Summary of the invention

[0005] In order to solve the above-mentioned technical problems, the present invention provides an industrial Internet data processing method and system based on big data.

[0006] The technical solution of the present invention is achieved in this way:

[0007] A method for processing industrial Internet data based on big data comprises the following steps:

[0008] S1. Design a visual data processing process orchestration tool that allows users to flexibly define the logic and steps of data processing. In the process of process orchestration, common data processing operations can be implemented through a predefined data processing operator library;

[0009] S2. Through the visual configuration interface, the user is allowed to select the required data processing components by dragging and dropping, and set the corresponding data types and processing conditions to form a complete data processing flow chart;

[0010] S3. Obtain the data type information configured by the user, perform initial condition filtering and judgment on the data based on the information, and filter out the original data that meets the conditions;

[0011] S4. Determine whether it is necessary to dynamically calculate data over time based on whether the conditions change;

[0012] S5. Introduce a real-time data processing engine to collect, clean, convert and analyze data in real time.

[0013] Furthermore, the design of the visual data processing flow arrangement tool in step S1 specifically includes, based on the predefined data processing operator library, building a visual component library containing common data processing operations, each component corresponding to a data processing function;

[0014] Design a visual drag-and-drop interface that allows users to select the required data processing components from the component library and drag them to the process orchestration area. In the process orchestration area, connect the various data processing components according to business logic through connecting lines;

[0015] Design a data flow mechanism to ensure that data is transferred and processed between components in a predefined logical order, and the output of the previous component automatically becomes the input of the next component;

[0016] Design a task scheduling mechanism to automatically generate task scheduling scripts based on the user-defined data processing flow to control the execution order and data flow of each data processing component;

[0017] The real-time preview function is introduced to allow users to view the output results of each processing node in real time during the process of arranging the data processing flow, so as to adjust and optimize the processing logic in time.

[0018] Furthermore, the step S2 also includes automatically generating corresponding data processing logic code according to the data processing flow chart;

[0019] The introduction of NLP technology allows users to describe their data processing requirements in natural language and convert natural language instructions into corresponding data processing flow configurations;

[0020] The dependency relationship between tasks is represented by a directed acyclic graph, and the execution order of tasks is determined by the task scheduling algorithm.

[0021] Furthermore, the process of screening the original data that meets the conditions in step S3 is to obtain user configuration information, determine parameters such as data type, result identifier, data identifier and manually input value, establish the initial screening conditions of the original data according to the obtained user configuration information, and monitor the accuracy and completeness of the data in real time by setting data quality monitoring rules;

[0022] Traverse the original data, make conditional judgments for each piece of data, and clean the data in the initial filtering result set according to predefined data quality rules to remove noise data;

[0023] Adopt incremental data processing method to process new or changed data in real time to avoid repeated calculations;

[0024] The cleaned data is formatted and structured according to the preset conversion rules, a data lineage diagram is established, the entire process from data collection to output is recorded, and the converted data subset that meets the conditions is used as the input for the next step of data processing.

[0025] Furthermore, in step S4, it is determined whether data needs to be dynamically calculated over time. If the conditions change, the configured data is dynamically obtained from the data warehouse according to the time value stored in the database, and the allocation of computing resources is automatically adjusted according to the complexity of data changes and computing requirements in combination with adaptive computing resource allocation;

[0026] Specifically, according to the preset business rules, determine whether the triggering conditions of the condition change are met. If so, enter the real-time calculation process, otherwise maintain the current data status;

[0027] Obtain configuration parameters related to the current time value from the database, determine the target data range and granularity that needs to be extracted from the data warehouse, adopt an incremental data synchronization strategy, dynamically obtain changed data from the data warehouse, and merge and update it with the existing local data to obtain an updated complete data set;

[0028] By analyzing the complexity of data changes and the resource requirements of computing tasks, the allocation ratio of computing resources is dynamically adjusted. For the updated complete data set, a decision tree algorithm is used to build a classification model of data changes to predict the data change trend in the future. Based on the data change trend prediction results, the time period for data acquisition and calculation is dynamically adjusted.

[0029] Furthermore, in step S5, a real-time data processing engine is introduced to trigger corresponding data processing tasks by monitoring data change events. If data changes are found, the changed data is obtained to determine the data processing tasks to be executed;

[0030] According to the predefined data processing algorithm, the acquired real-time data is cleaned and noise data is removed. If the data quality does not meet the requirements, the data cleaning process is triggered to obtain clean data;

[0031] Adopt data conversion rules to convert the format and structure the cleaned data. By analyzing the data structure and field meaning, the data format is standardized according to the preset conversion template to determine whether the converted data meets the requirements of subsequent analysis.

[0032] Using event-driven architecture, the execution order of tasks is determined through task scheduling algorithm according to the priority of data processing tasks;

[0033] Obtain the dependencies of the tasks, build a directed acyclic graph, and obtain the execution topology of the tasks. When processing tasks in real time, use the stream computing framework to perform real-time filtering, conversion, and aggregation operations on the converted data. According to the characteristics of the data and business needs, select the appropriate computing function and window type. For different data processing scenarios, select machine learning algorithms for real-time data analysis.

[0034] The method further includes step S6, wherein step S6 specifically includes dynamically adjusting the task execution sequence and resource allocation according to the dependency relationship and resource consumption of the data processing tasks.

[0035] Furthermore, the process of dynamically adjusting the task execution order and resource allocation in step S6 is specifically to obtain the dependency and resource consumption of the data processing tasks, construct a task dependency graph and a resource consumption model, and if there is a dependency between the tasks, establish a directed edge in the task dependency graph to connect the related task nodes, adopt a heuristic algorithm, and generate the initial task execution order and resource allocation plan according to the task dependency graph and the resource consumption model, and obtain the optimized scheduling plan through iterative optimization;

[0036] Introducing deep reinforcement learning algorithms, the task scheduling problem is modeled as a Markov decision process. Through the interaction between the intelligent agent and the environment, the optimal task execution sequence and resource allocation strategy are dynamically learned. During the task execution process, the task progress and resource consumption are monitored in real time. If the task execution progress is lagging behind or the resource utilization rate is low, the dynamic adjustment mechanism is triggered to re-optimize the task execution sequence and resource allocation for independent data processing tasks.

[0037] Adopt parallel execution strategy, make full use of resources, improve task execution efficiency through parallel execution of tasks, execute tasks in series according to the optimized execution order for tasks with dependencies, determine the execution order of tasks according to the task dependency graph, and set task priorities according to the urgency and importance of tasks;

[0038] After setting the task priority in step S6, when there is resource competition, resources are allocated preferentially to high-priority tasks, a monitoring platform for task execution and resource consumption is built, task execution data is collected and analyzed in real time, and scheduling strategies and algorithm parameters are continuously optimized by evaluating the effect of the scheduling algorithm to achieve adaptive intelligent task scheduling;

[0039] Based on historical data and real-time monitoring data, the task execution order and resource allocation are dynamically adjusted. Using deep learning algorithms, the task execution status and resource consumption trends in the future are predicted, and resource scheduling and task optimization are performed in advance.

[0040] An industrial Internet data processing system based on big data, including a visual process orchestration module, a data preprocessing module, a real-time data processing engine, a task scheduling and resource management module, and an intelligent decision support module;

[0041] The visual process arrangement module provides a user interface, allowing users to define data processing processes by dragging and dropping components; the visual process arrangement module includes a visual component library, from which users can select data processing components and set data types and processing conditions; the visual process arrangement module is responsible for generating task scheduling scripts and controlling the execution order of data processing components;

[0042] The data preprocessing module is responsible for performing initial screening and filtering of data according to the information configured by the user, establishing data screening conditions, performing data quality monitoring, and performing data cleaning and format conversion to ensure that only qualified data is further processed and that the data quality meets the requirements of subsequent processing;

[0043] The real-time data processing engine monitors data change events, triggers corresponding data processing tasks, and performs real-time data collection, cleaning, conversion and analysis;

[0044] The task scheduling and resource management module dynamically adjusts the task execution order and resource allocation according to the dependency relationship and resource consumption of data processing tasks, including the construction of task dependency graph, the establishment of resource consumption model, the application of deep reinforcement learning algorithm, and the monitoring of task execution and resource consumption, so as to realize intelligent task scheduling and resource optimization;

[0045] The intelligent decision support module uses deep learning algorithms to predict task execution status and resource consumption trends, performs resource scheduling and task optimization in advance, and provides decision support by analyzing historical data and real-time monitoring data, helping the system to adaptively adjust task execution order and resource allocation to improve overall processing efficiency and response speed.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] 1. The present invention uses a drag-and-drop interface to allow users to flexibly define data processing logic and steps, and use predefined operator libraries to implement common data operations. By introducing NLP technology, users are allowed to describe processing requirements in natural language, which are automatically converted into corresponding configurations, thus lowering the technical threshold. During the data processing process, the present invention performs initial conditional filtering based on data type information, supports dynamic calculation over time, introduces a real-time data processing engine, and combines an event-driven architecture to improve response speed;

[0048] 2. The present invention also uses an intelligent task scheduling algorithm to dynamically adjust the execution order and resource allocation according to dependencies and resource consumption. This configuration-based approach reduces manual coding work, and non-technical personnel can also participate in configuration and monitoring. Through innovations such as visual interface, NLP interaction, real-time processing and intelligent scheduling, the present invention significantly improves the flexibility, efficiency and ease of use of data processing, reduces development and maintenance costs, and provides an efficient and convenient solution for complex data processing tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is a flow chart of a method for processing industrial Internet data based on big data according to Example 1;

[0050] Figure 2 A flowchart of the step of dynamically adjusting the task execution order and resource allocation in step S6 of Example 1;

[0051] Figure 3 This is a system framework diagram of Example 2, an industrial Internet data processing system based on big data. DETAILED DESCRIPTION

[0052] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] Example

[0054] like Figure 1 As shown, this embodiment provides an industrial Internet data processing method based on big data, comprising the following steps:

[0055] S1. Design a visual data processing process orchestration tool that allows users to flexibly define the logic and steps of data processing. In the process of process orchestration, common data processing operations can be implemented through a predefined data processing operator library;

[0056] S2. Through the visual configuration interface, the user is allowed to select the required data processing components by dragging and dropping, and set the corresponding data types and processing conditions to form a complete data processing flow chart;

[0057] S3. Obtain the data type information configured by the user, perform initial condition filtering and judgment on the data based on the information, and filter out the original data that meets the conditions;

[0058] S4. Determine whether it is necessary to dynamically calculate data over time based on whether the conditions change;

[0059] S5. Introduce a real-time data processing engine to collect, clean, convert and analyze data in real time.

[0060] Furthermore, the design of the visual data processing flow arrangement tool in step S1 specifically includes, based on the predefined data processing operator library, building a visual component library containing common data processing operations, each component corresponding to a data processing function;

[0061] Design a visual drag-and-drop interface that allows users to select the required data processing components from the component library and drag them to the process orchestration area. In the process orchestration area, connect the various data processing components according to business logic through connecting lines;

[0062] Design a data flow mechanism to ensure that data is transferred and processed between components in a predefined logical order, and the output of the previous component automatically becomes the input of the next component;

[0063] Design a task scheduling mechanism to automatically generate task scheduling scripts based on the user-defined data processing flow to control the execution order and data flow of each data processing component;

[0064] The real-time preview function is introduced to allow users to view the output results of each processing node in real time during the process of arranging the data processing flow, so as to adjust and optimize the processing logic in time;

[0065] When building a visualization component library, you can design corresponding components based on common data processing operations, such as data cleaning, feature engineering, data integration, etc. For example, for data cleaning components, you can build in common functions such as data deduplication, missing value processing, and outlier filtering.

[0066] In the drag-and-drop interface, users can simply drag and drop to add the required components to the process arrangement area, and use connecting lines to connect the components according to business logic. For each component, the parameter configuration interface allows users to set algorithm parameters according to actual needs, such as setting the threshold for data deduplication and the method of filling missing values. During the data flow process, the output of the previous component is automatically used as the input of the next component to ensure smooth data transmission between components.

[0067] Using directed acyclic graph and topological sorting algorithm, it automatically generates task scheduling scripts and controls the execution order of components;

[0068] The real-time preview function enables users to view the output results of each processing node in real time by establishing data cache and incremental calculation mechanism, so as to optimize the processing logic in time;

[0069] A real-time data processing engine is introduced to support the collection, cleaning, conversion and analysis of real-time data. Predefined data processing algorithms, such as real-time data stream processing algorithms (such as Flink, Spark Streaming, etc.) are used to quickly respond to business needs. Finally, a visual monitoring panel is used to display the running status, performance indicators (such as throughput, latency, etc.) and abnormal conditions of the data processing process, and resource allocation is dynamically adjusted according to the monitoring results to optimize system performance.

[0070] Furthermore, the step S2 also includes automatically generating corresponding data processing logic code according to the data processing flow chart;

[0071] The introduction of NLP technology allows users to describe their data processing requirements in natural language and convert natural language instructions into corresponding data processing flow configurations;

[0072] The dependency relationship between tasks is represented by a directed acyclic graph, and the execution order of tasks is determined by the task scheduling algorithm;

[0073] When building a visualization component library, you can predefine common data processing operators, such as filtering, joining, and aggregation. Each operator corresponds to a visualization component.

[0074] When designing a drag-and-drop interface, you can use a layout similar to a flowchart. Users can drag components onto the canvas with the mouse and connect them with connecting lines. In the parameter configuration interface, you can provide a drop-down menu or input box for each component, allowing users to select data types (such as strings, integers, floating-point numbers, etc.) and set parameter values ​​(such as filter conditions, aggregate functions, etc.);

[0075] When natural language processing technology is introduced, algorithms such as named entity recognition and relationship extraction can be used to convert natural language instructions input by users into structured data processing configurations; the data flow mechanism can be implemented through message queues or data structures in memory to ensure that data is transmitted between components in a predefined order. The task scheduling engine can use a directed acyclic graph to represent the dependencies between tasks, determine the execution order of tasks through a topological sorting algorithm, and generate corresponding scheduling scripts;

[0076] The real-time preview function can display the intermediate results in real time on the interface by adding a listener at the output end of each component, and set thresholds for data quality and processing time, triggering an alarm when the thresholds are exceeded.

[0077] Furthermore, the process of screening the original data that meets the conditions in step S3 is to obtain user configuration information, determine parameters such as data type, result identifier, data identifier and manually input value, establish the initial screening conditions of the original data according to the obtained user configuration information, and monitor the accuracy and completeness of the data in real time by setting data quality monitoring rules;

[0078] Traverse the original data, make conditional judgments for each piece of data, and clean the data in the initial filtering result set according to predefined data quality rules to remove noise data;

[0079] Adopt incremental data processing method to process new or changed data in real time to avoid repeated calculations;

[0080] The cleaned data is formatted and structured according to the preset conversion rules, a data lineage diagram is established, the entire process from data collection to output is recorded, and the converted data subset that meets the conditions is used as the input for the next step of data processing;

[0081] When obtaining user configuration information, the user can select parameters such as data type and result identifier through a graphical interface. For example, the user can select "sales data" as the data type, "sales amount" as the result identifier, "product number" as the data identifier, and manually enter "2022" as the time range;

[0082] Based on these configuration information, you can build a primary screening condition, such as data type = sales data AND time ≥ 2022-01-01' AND time ≤ 2022-12-31;

[0083] When monitoring data in real time, you can set a data integrity rule, such as "the product number of the sales record cannot be empty". If more than 5% of the records do not meet the rule, an alarm will be triggered;

[0084] When traversing the original data, you can use conditional judgment statements, such as "IF data type" = "sales data" AND time ≥ "2022-01-01" AND time ≤ "2022-12-31 THEN" to add the initial filtering result set;

[0085] When performing data cleaning on the initial filtering result set, some outliers can be removed, such as records with negative sales quantities;

[0086] In incremental data processing, the timestamps of new data and processed data can be compared, and only data with newer timestamps can be processed. When converting data, sales can be converted from string format to numeric format, and time can be structured in the format of "year-month". The data lineage relationship diagram can record the source, cleaning process, conversion rules and other information of each sales record, which is convenient for tracing during subsequent data analysis. Finally, the converted data is used as input for data analysis algorithms, such as using the random forest algorithm to predict sales for the next month.

[0087] Furthermore, in step S4, it is determined whether data needs to be dynamically calculated over time. If the conditions change, the configured data is dynamically obtained from the data warehouse according to the time value stored in the database, and the allocation of computing resources is automatically adjusted according to the complexity of data changes and computing requirements in combination with adaptive computing resource allocation;

[0088] Specifically, according to the preset business rules, determine whether the triggering conditions of the condition change are met. If so, enter the real-time calculation process, otherwise maintain the current data status;

[0089] Obtain configuration parameters related to the current time value from the database, determine the target data range and granularity that needs to be extracted from the data warehouse, adopt an incremental data synchronization strategy, dynamically obtain changed data from the data warehouse, and merge and update it with the existing local data to obtain an updated complete data set;

[0090] By analyzing the complexity of data changes and the resource requirements of computing tasks, the allocation ratio of computing resources is dynamically adjusted. For the updated complete data set, a decision tree algorithm is used to build a classification model of data changes to predict the data change trend in the future. Based on the data change trend prediction results, the time period for data acquisition and calculation is dynamically adjusted;

[0091] According to the preset business rules, if the order amount exceeds 10,000 yuan and the customer's credit score is less than 60 points, real-time calculation is triggered, and the configuration parameters of the hour before the current time are obtained from the database to determine whether the order and customer data within the last 24 hours need to be extracted from the data warehouse;

[0092] Adopting incremental data synchronization, obtaining change data from the data warehouse every 5 minutes, merging and updating with local data, and analyzing the complexity of data changes, such as when order data increases by 20% and customer data increases by 10%, dynamically adjusting the CPU resource allocation ratio to 70% and the memory resource allocation ratio to 60%;

[0093] For the updated complete data set, the C4.5 decision tree algorithm is used to build a data change classification model to predict the data change trend within the next hour. According to the prediction results, if the data change speed accelerates, the data acquisition and calculation time cycle is adjusted to 2 minutes, and key indicators in the task execution process are obtained, such as data processing speed reaching 1000 / second and resource utilization reaching 80%, which are displayed in real time through a visual interface; if the data processing time exceeds the threshold of 20 seconds, the predefined alarm rules are triggered to notify the relevant responsible persons;

[0094] The optimized data calculation results are stored in the MySQL database, and order analysis reports and customer credit analysis reports are generated according to business needs and visualized through web pages.

[0095] Furthermore, in step S5, a real-time data processing engine is introduced to trigger corresponding data processing tasks by monitoring data change events. If data changes are found, the changed data is obtained to determine the data processing tasks to be executed;

[0096] According to the predefined data processing algorithm, the acquired real-time data is cleaned and noise data is removed. If the data quality does not meet the requirements, the data cleaning process is triggered to obtain clean data;

[0097] Adopt data conversion rules to convert the format and structure the cleaned data. By analyzing the data structure and field meaning, the data format is standardized according to the preset conversion template to determine whether the converted data meets the requirements of subsequent analysis.

[0098] Using event-driven architecture, the execution order of tasks is determined through task scheduling algorithm according to the priority of data processing tasks;

[0099] Obtain the dependencies of tasks, build a directed acyclic graph, and obtain the execution topology of tasks. When processing tasks in real time, use the stream computing framework to perform real-time filtering, conversion, and aggregation operations on the converted data. According to the characteristics of the data and business needs, select the appropriate computing function and window type. For different data processing scenarios, select the machine learning algorithm for real-time data analysis; if it is a classification problem, use the decision tree algorithm; if it is a prediction problem, use the linear regression algorithm; if it is a clustering problem, use the K-means algorithm;

[0100] Specifically, the Apache Flink real-time data processing engine is introduced, and the binlog of the MySQL database is monitored through the CDC (Change Data Capture) mechanism to capture data change events in real time. When the data is updated, the Flink job will automatically trigger the predefined data processing tasks, such as real-time cleaning of the newly added user logs, filtering out duplicate data using the Bloom filter algorithm, and verifying the field format through regular expressions to remove dirty data that does not meet the requirements. The cleaned data is structured according to the preset Avro format, the table structure is defined through RowTypeInfo, and the field mapping and type conversion are performed using MapFunction. Then, the task dependencies are defined in the form of a directed acyclic graph (DAG), the execution order is determined using the priority queue algorithm, and latency-sensitive real-time analysis tasks are prioritized. The Flink CEP library is applied to the converted data for complex event processing, the event pattern is defined through the Pattern API, the user behavior events that meet the conditions are filtered out in real time, and the incremental aggregation function ReduceFunction is used to perform statistical analysis on the events. For the user clustering scenario, the K-means clustering algorithm is selected, the user feature similarity is measured by the Euclidean distance, the cluster center is iteratively optimized, and the user clustering results are obtained.

[0101] In this step, it may also include: by obtaining key indicators of data processing, such as data processing speed, resource utilization, etc., displaying them in real time on a visual interface, if an indicator is found to be abnormal, such as the processing speed is lower than a threshold or the resource utilization is too high, triggering a predefined alarm rule to notify the relevant responsible person to handle it in time;

[0102] By establishing a data lineage relationship diagram, the entire process from data collection to output is recorded, including data source, processing logic, intermediate results and final output. Through data lineage analysis, the root cause of data quality problems can be determined, the data change process can be traced, and the traceability of data processing can be improved;

[0103] Adopting incremental data processing method, it processes new or changed data in real time to avoid repeated calculations. It determines whether the data needs to be reprocessed based on the data update time or version number to ensure the real-time and accuracy of the data processing results.

[0104] Specifically, during the analysis process, key indicators of data processing are monitored, such as the number of records processed per second (RecordsperSecond), latency, etc. When the processing speed is lower than 100 records per second or the 99th percentile latency exceeds 1 second, an alarm notification is triggered. Finally, a data lineage relationship graph is constructed through the process method of the FlinkDataStreamAPI to record each step of the data conversion and achieve end-to-end data traceability. For incremental data, Flink determines whether the data has been updated through the timestamp of the Kafka message or the update time of the MySQL table. It only processes new or changed data to avoid repeated calculations and improve real-time performance.

[0105] like Figure 2 As shown, the method further includes step S6, wherein step S6 specifically includes dynamically adjusting the task execution sequence and resource allocation according to the dependency relationship and resource consumption of the data processing tasks.

[0106] Furthermore, the process of dynamically adjusting the task execution order and resource allocation in step S6 is specifically to obtain the dependency and resource consumption of the data processing tasks, construct a task dependency graph and a resource consumption model, and if there is a dependency between the tasks, establish a directed edge in the task dependency graph to connect the related task nodes, adopt a heuristic algorithm, and generate the initial task execution order and resource allocation plan according to the task dependency graph and the resource consumption model, and obtain the optimized scheduling plan through iterative optimization;

[0107] Introducing deep reinforcement learning algorithms, the task scheduling problem is modeled as a Markov decision process. Through the interaction between the intelligent agent and the environment, the optimal task execution sequence and resource allocation strategy are dynamically learned. During the task execution process, the task progress and resource consumption are monitored in real time. If the task execution progress is lagging behind or the resource utilization rate is low, the dynamic adjustment mechanism is triggered to re-optimize the task execution sequence and resource allocation for independent data processing tasks.

[0108] Adopt parallel execution strategy, make full use of resources, improve task execution efficiency through parallel execution of tasks, execute tasks in series according to the optimized execution order for tasks with dependencies, determine the execution order of tasks according to the task dependency graph, and set task priorities according to the urgency and importance of tasks;

[0109] After setting the task priority in step S6, when there is resource competition, resources are allocated preferentially to high-priority tasks, a monitoring platform for task execution and resource consumption is built, task execution data is collected and analyzed in real time, and scheduling strategies and algorithm parameters are continuously optimized by evaluating the effect of the scheduling algorithm to achieve adaptive intelligent task scheduling;

[0110] Based on historical data and real-time monitoring data, the task execution order and resource allocation are dynamically adjusted. Using deep learning algorithms, the task execution status and resource consumption trends in the future are predicted, and resource scheduling and task optimization are performed in advance.

[0111] Specifically, when obtaining the dependency and resource consumption of data processing tasks, a directed acyclic graph (DAG) can be used to represent the dependency between tasks, where nodes represent tasks, edges represent dependencies, and edge weights can represent the amount of data transfer between tasks or the execution time of tasks.

[0112] By analyzing historical task execution data, a resource consumption model is established. For example, a linear regression model is used to predict the CPU, memory, I / O and other resources required for task execution based on the input data volume, computational complexity and other characteristics of the task. When using genetic algorithms to optimize task scheduling schemes, the task execution order and resource allocation scheme can be encoded into gene sequences. Through operations such as selection, crossover, and mutation, iterative optimization is performed to obtain a scheduling scheme with high fitness. For example, using binary encoding, each task corresponds to a gene bit, 0 means unscheduled, 1 means scheduled, and the optimal task scheduling order is searched through genetic algorithms;

[0113] When applying deep reinforcement learning algorithms, the task scheduling problem can be modeled as a Markov decision process, where the state is the current task execution progress and resource usage, the action is to select the next task to be executed and the allocated resources, and the reward is the weighted sum of task execution efficiency and resource utilization. By continuously interacting with the environment, the agent learns the optimal scheduling strategy. When triggering the dynamic adjustment mechanism, the thresholds of task execution progress and resource utilization can be set. For example, when the task execution progress is less than 50% and the resource utilization is less than 60%, rescheduling is triggered;

[0114] For independent tasks, you can split the task into multiple subtasks and assign them to different CPU cores or GPU devices for parallel execution, such as using the MapReduce framework for distributed computing. For tasks with dependencies, you can use a topological sorting algorithm, such as the Kahn algorithm or the DFS algorithm, to determine the order of task execution based on the task dependency graph.

[0115] When setting task priorities, you can calculate the priority score of the task based on factors such as the impact of the task on the business and the deadline. For example, if you use the weighted shortest job first (WSJF) algorithm, the priority score = business value / task duration. When monitoring task execution in real time, you can use tools such as ELK (Elasticsearch, Logstash, Kibana) to collect task execution logs, analyze task execution progress, resource consumption and other indicators in real time, and display the effect of the scheduling algorithm through the dashboard.

[0116] Based on historical data and real-time monitoring data, the LSTM network can be used to predict task execution and resource consumption trends in the future. For example, based on the task execution data of the past 24 hours, the task execution completion rate and resource utilization rate in the next hour can be predicted, and resource scheduling and task optimization can be performed in advance.

[0117] Example 2

[0118] like Figure 3 As shown, this embodiment provides an industrial Internet data processing system based on big data, which is used to implement an industrial Internet data processing method based on big data, including a visual process orchestration module, a data preprocessing module, a real-time data processing engine, a task scheduling and resource management module, and an intelligent decision support module;

[0119] The visual process arrangement module provides a user interface, allowing users to define data processing processes by dragging and dropping components; the visual process arrangement module includes a visual component library, from which users can select data processing components and set data types and processing conditions; the visual process arrangement module is responsible for generating task scheduling scripts and controlling the execution order of data processing components;

[0120] The data preprocessing module is responsible for performing initial screening and filtering of data according to the information configured by the user, establishing data screening conditions, performing data quality monitoring, and performing data cleaning and format conversion to ensure that only qualified data is further processed and that the data quality meets the requirements of subsequent processing;

[0121] The real-time data processing engine monitors data change events, triggers corresponding data processing tasks, and performs real-time data collection, cleaning, conversion and analysis;

[0122] The task scheduling and resource management module dynamically adjusts the task execution order and resource allocation according to the dependency relationship and resource consumption of data processing tasks, including the construction of task dependency graph, the establishment of resource consumption model, the application of deep reinforcement learning algorithm, and the monitoring of task execution and resource consumption, so as to realize intelligent task scheduling and resource optimization;

[0123] The intelligent decision support module uses deep learning algorithms to predict task execution status and resource consumption trends, performs resource scheduling and task optimization in advance, and provides decision support by analyzing historical data and real-time monitoring data, helping the system to adaptively adjust task execution order and resource allocation to improve overall processing efficiency and response speed.

[0124] In this embodiment, the visual process arrangement module provides an intuitive drag-and-drop user interface, where users can define data processing processes in a graphical manner. The interface displays icons of various data processing components, and users can build processes by dragging these icons to the workspace;

[0125] Build a component library that contains a variety of data processing operations, such as data cleaning, feature engineering, data integration, etc. Each component has a predefined algorithm and parameter setting interface, and users can configure these parameters as needed;

[0126] According to the user-defined process, task scheduling scripts are automatically generated. These scripts define the execution order of data processing components, use directed acyclic graphs (DAGs) to represent the dependencies between tasks, and adopt topological sorting algorithms to determine the execution order.

[0127] The data preprocessing module performs preliminary screening of the original data according to the screening conditions configured by the user, eliminates data that does not meet the conditions, monitors the accuracy, completeness and other indicators of the data in real time, ensures that the data quality meets the preset standards, cleans the screened data, including removing outliers, filling missing values, etc., and converts the data into a format suitable for analysis, such as converting dates from strings to date types.

[0128] The real-time data processing engine uses message queues or database change data capture (CDC) technology to monitor data change events, collect data in real time from various data sources such as sensors, databases, APIs, etc., clean and convert the real-time collected data to ensure data quality, and apply real-time analysis algorithms, such as stream processing algorithms, to perform instant analysis of data.

[0129] The task scheduling and resource management module constructs a task dependency graph to represent the dependency relationship between tasks, establishes a resource consumption model, predicts the resources required for task execution, applies a deep reinforcement learning algorithm, dynamically learns the optimal task execution sequence and resource allocation strategy, monitors task progress and resource consumption in real time, and adjusts task scheduling and resource allocation based on the monitoring results.

[0130] The intelligent decision support module uses deep learning algorithms, such as LSTM networks, to predict task execution and resource consumption trends, analyze historical data and real-time monitoring data, provide decision support, and help the system adaptively adjust the task execution order and resource allocation. According to the prediction results and real-time data, it automatically adjusts the task execution order and resource allocation to improve processing efficiency and response speed.

[0131] Users define the data processing process through the visual process orchestration module. The generated task scheduling script is used by the task scheduling and resource management module to control the execution of the data preprocessing module and the real-time data processing engine. The data processed by the data preprocessing module is further processed by the real-time data processing engine. The output of the real-time data processing engine is used for prediction and optimization by the intelligent decision support module. The task scheduling and resource management module dynamically adjusts the task and resource allocation according to the prediction results of the intelligent decision support module.

[0132] The specific embodiments of the invention are described in detail above, but they are only examples. The present invention is not limited to the specific embodiments described above. Those skilled in the art should understand that the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A method for processing industrial Internet data based on big data, characterized in that: The following steps are involved: S1. Design a visual data processing process orchestration tool that allows users to flexibly define the logic and steps of data processing. In the process of process orchestration, common data processing operations can be implemented through a predefined data processing operator library; S2. Through the visual configuration interface, the user is allowed to select the required data processing components by dragging and dropping, and set the corresponding data types and processing conditions to form a complete data processing flow chart; S3, obtaining the data type information configured by the user, performing initial condition filtering and judgment on the data based on the information, and screening out the original data that meets the conditions; S4. Determine whether it is necessary to dynamically calculate data over time based on whether the conditions change; S5. Introduce a real-time data processing engine to collect, clean, convert and analyze data in real time.

2. The method for processing industrial Internet data based on big data according to claim 1 is characterized in that: The step S1 of designing a visualized data processing flow arrangement tool specifically includes building a visualized component library containing common data processing operations according to a predefined data processing operator library, where each component corresponds to a data processing function; Design a visual drag-and-drop interface that allows users to select the required data processing components from the component library and drag them to the process orchestration area. In the process orchestration area, connect the various data processing components according to business logic through connecting lines; Design a data flow mechanism to ensure that data is transferred and processed between components in a predefined logical order, and the output of the previous component automatically becomes the input of the next component; Design a task scheduling mechanism to automatically generate task scheduling scripts based on the user-defined data processing flow to control the execution order and data flow of each data processing component; The real-time preview function is introduced to allow users to view the output results of each processing node in real time during the process of arranging the data processing flow, so as to adjust and optimize the processing logic in time.

3. The method for processing industrial Internet data based on big data according to claim 1 is characterized in that: The step S2 also includes automatically generating corresponding data processing logic code according to the data processing flow chart; The introduction of NLP technology allows users to describe their data processing requirements in natural language and convert natural language instructions into corresponding data processing flow configurations; The dependency relationship between tasks is represented by a directed acyclic graph, and the execution order of tasks is determined by the task scheduling algorithm.

4. The method for processing industrial Internet data based on big data according to claim 1 is characterized in that: The process of screening the original data that meets the conditions in step S3 is to obtain user configuration information, determine parameters such as data type, result identifier, data identifier and manually input value, establish the initial screening conditions of the original data according to the obtained user configuration information, and monitor the accuracy and completeness of the data in real time by setting data quality monitoring rules; Traverse the original data, make conditional judgments for each piece of data, and clean the data in the initial filtering result set according to predefined data quality rules to remove noise data; Adopt incremental data processing method to process new or changed data in real time to avoid repeated calculations; The cleaned data is formatted and structured according to the preset conversion rules, a data lineage diagram is established, the entire process from data collection to output is recorded, and the converted data subset that meets the conditions is used as the input for the next step of data processing.

5. The method for processing industrial Internet data based on big data according to claim 1 is characterized in that: In step S4, it is determined whether it is necessary to dynamically calculate data over time. If the conditions change, the configured data is dynamically obtained from the data warehouse according to the time value stored in the database, and the allocation of computing resources is automatically adjusted according to the complexity of data changes and computing requirements in combination with adaptive computing resource allocation; Specifically, according to the preset business rules, determine whether the triggering conditions of the condition change are met. If so, enter the real-time calculation process, otherwise maintain the current data status; Obtain configuration parameters related to the current time value from the database, determine the target data range and granularity that needs to be extracted from the data warehouse, adopt an incremental data synchronization strategy, dynamically obtain changed data from the data warehouse, and merge and update it with the existing local data to obtain an updated complete data set; By analyzing the complexity of data changes and the resource requirements of computing tasks, the allocation ratio of computing resources is dynamically adjusted. For the updated complete data set, a decision tree algorithm is used to build a classification model of data changes to predict the data change trend in the future. Based on the data change trend prediction results, the time period for data acquisition and calculation is dynamically adjusted.

6. The method for processing industrial Internet data based on big data according to claim 1 is characterized in that: In step S5, a real-time data processing engine is introduced to trigger corresponding data processing tasks by monitoring data change events. If data changes are found, the changed data is obtained to determine the data processing tasks to be executed; According to the predefined data processing algorithm, the acquired real-time data is cleaned and noise data is removed. If the data quality does not meet the requirements, the data cleaning process is triggered to obtain clean data; Adopt data conversion rules to convert the format and structure the cleaned data. By analyzing the data structure and field meaning, the data format is standardized according to the preset conversion template to determine whether the converted data meets the requirements of subsequent analysis. Using event-driven architecture, the execution order of tasks is determined through task scheduling algorithm according to the priority of data processing tasks; Obtain the dependencies of the tasks, build a directed acyclic graph, and obtain the execution topology of the tasks. When processing tasks in real time, use the stream computing framework to perform real-time filtering, conversion, and aggregation operations on the converted data. According to the characteristics of the data and business needs, select the appropriate computing function and window type. For different data processing scenarios, select machine learning algorithms for real-time data analysis.

7. The method for processing industrial Internet data based on big data according to claim 1 is characterized in that: The method further includes step S6, wherein step S6 specifically includes dynamically adjusting the task execution sequence and resource allocation according to the dependency relationship and resource consumption of the data processing tasks.

8. The method for processing industrial Internet data based on big data according to claim 7 is characterized in that: The process of dynamically adjusting the task execution order and resource allocation in step S6 is specifically to obtain the dependency and resource consumption of the data processing tasks, build a task dependency graph and a resource consumption model, and if there is a dependency between the tasks, establish a directed edge in the task dependency graph to connect the related task nodes, use a heuristic algorithm, and generate an initial task execution order and resource allocation plan based on the task dependency graph and the resource consumption model, and obtain an optimized scheduling plan through iterative optimization; Introducing deep reinforcement learning algorithms, the task scheduling problem is modeled as a Markov decision process. Through the interaction between the intelligent agent and the environment, the optimal task execution sequence and resource allocation strategy are dynamically learned. During the task execution process, the task progress and resource consumption are monitored in real time. If the task execution progress is lagging behind or the resource utilization rate is low, the dynamic adjustment mechanism is triggered to re-optimize the task execution sequence and resource allocation for independent data processing tasks. Adopt parallel execution strategy, make full use of resources, improve task execution efficiency through parallel execution of tasks, and execute tasks with dependencies in series according to the optimized execution order. Determine the order of task execution based on the task dependency graph, and set task priority based on the urgency and importance of the task.

9. The method for processing industrial Internet data based on big data according to claim 8 is characterized in that: After setting the task priority in step S6, when there is resource competition, resources are allocated preferentially to high-priority tasks, a monitoring platform for task execution and resource consumption is built, task execution data is collected and analyzed in real time, and scheduling strategies and algorithm parameters are continuously optimized by evaluating the effect of the scheduling algorithm to achieve adaptive intelligent task scheduling; Based on historical data and real-time monitoring data, the task execution order and resource allocation are dynamically adjusted. Using deep learning algorithms, the task execution status and resource consumption trends in the future are predicted, and resource scheduling and task optimization are performed in advance.

10. An industrial Internet data processing system based on big data, characterized in that: It includes visual process arrangement module, data preprocessing module, real-time data processing engine, task scheduling and resource management module and intelligent decision support module; The visual process arrangement module provides a user interface, allowing users to define data processing processes by dragging and dropping components; The visual process orchestration module includes a visual component library from which users can select data processing components and set data types and processing conditions; The visual process orchestration module is responsible for generating task scheduling scripts and controlling the execution order of data processing components; The data preprocessing module is responsible for performing initial screening and filtering of data according to the information configured by the user, establishing data screening conditions, performing data quality monitoring, and performing data cleaning and format conversion to ensure that only qualified data is further processed and that the data quality meets the requirements of subsequent processing; The real-time data processing engine monitors data change events, triggers corresponding data processing tasks, and performs real-time data collection, cleaning, conversion and analysis; The task scheduling and resource management module dynamically adjusts the task execution order and resource allocation according to the dependency relationship and resource consumption of data processing tasks, including the construction of task dependency graph, the establishment of resource consumption model, the application of deep reinforcement learning algorithm, and the monitoring of task execution and resource consumption, so as to realize intelligent task scheduling and resource optimization; The intelligent decision support module uses deep learning algorithms to predict task execution status and resource consumption trends, performs resource scheduling and task optimization in advance, and provides decision support by analyzing historical data and real-time monitoring data, helping the system to adaptively adjust task execution order and resource allocation to improve overall processing efficiency and response speed.

Images