Efficient data processing system based on cloud network fusion and implementation method thereof
Through an efficient data processing system based on cloud network convergence, using intelligent scheduling and distributed computing framework to optimize resource configuration and load balancing, the problem of inefficiency in traditional systems when processing large-scale data is solved, and efficient data processing and system performance are achieved. promote.
Patent Information
- Application Number
- CN202411449474.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-17
- Publication Date
- 2024-11-15
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Traditional data processing systems are limited by the allocation of network bandwidth and computing resources when processing large-scale data, resulting in inefficiency and increased latency, which affects user experience and system performance.
Adopt an efficient data processing system based on cloud network convergence, including data access unit, data preprocessing unit, cloud resource scheduling unit, data processing unit, data storage and output unit and monitoring unit, through intelligent scheduling algorithm and distributed computing framework ( Such as Spark or Flink, realize resource optimization configuration and load balancing, perform parallel processing and real-time monitoring.
It improves data processing efficiency, reduces data transmission delay, optimizes resource utilization, improves the system's elasticity and scalability, and ensures the continuous high-performance operation of the system.
Smart Images

Figure CN118964470A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of cloud computing and network communication technology, and in particular to an efficient data processing system based on cloud-network integration and an implementation method thereof. Background Art
[0002] With the advent of the big data era, the demand for data processing is growing. Traditional data processing systems are often limited by hardware resources and cannot meet the needs of large-scale data processing. Cloud-network integration technology provides a new solution for data processing. Through the elastic resources of cloud computing and the efficient transmission of network communications, data processing capabilities can be improved.
[0003] In existing technologies, traditional data processing systems are often limited by network bandwidth and the allocation of computing resources, resulting in low efficiency and increased latency when processing large-scale data, affecting user experience and system performance. Especially when operating data in the cloud, frequent data transmission and system response time become key factors affecting overall performance. Although there are some attempts in existing technologies to improve efficiency by optimizing network transmission protocols, improving computing resource scheduling, etc., it is difficult to achieve a significant improvement in the overall performance of the system. Summary of the invention
[0004] In view of the shortcomings of the prior art, the purpose of the present invention is to provide an efficient data processing system based on cloud-network integration and an implementation method thereof, which can improve data processing efficiency, reduce data transmission delay, optimize resource utilization, and enhance the elasticity and scalability of the system.
[0005] The above-mentioned object of the present invention is achieved through the following technical solutions: An efficient data processing system based on cloud-network integration includes a data access unit, a data preprocessing unit, a cloud resource scheduling unit, a data processing unit, a data storage and output unit, a monitoring unit, and an optimization unit that are sequentially connected in communication; The data access unit is used to receive data from different data sources in real time; The data preprocessing unit is used to perform cleaning, deduplication, and format conversion preprocessing operations on the received raw data to ensure data quality and consistency; The cloud resource scheduling unit dynamically allocates cloud computing resources, including computing nodes, storage space, and network bandwidth, according to the needs of data processing tasks, and achieves optimal resource configuration and load balancing through intelligent scheduling algorithms; The data processing unit uses Spark or Flink distributed computing framework to perform parallel processing on the preprocessed data; The data storage and output unit stores the processed data in a distributed file system or database, provides data query, analysis and visualization functions according to business requirements, and outputs the processing results to a specified target system or application; The data monitoring unit is used to monitor the operating status, resource utilization and performance indicators of the system in real time.
[0006] As a further technical solution of the present invention: the data preprocessing unit includes: Intelligent data identification and classification module uses natural language processing and machine learning technology to intelligently identify and classify the incoming data, and automatically divides the data into corresponding data sets according to the type, structure or semantic information of the data, so as to facilitate subsequent targeted processing; Dynamic data cleaning strategy module, through dynamic data cleaning strategy, adjusts cleaning rules according to the real-time quality of data; The incremental data preprocessing module adopts an incremental preprocessing method for continuously connected data streams, that is, only necessary preprocessing operations are performed on newly connected data to avoid repeated processing of the entire data set, thereby improving processing efficiency.
[0007] The multi-source data fusion module uses multi-source data fusion technology to comprehensively process data from different data sources. Through data alignment, correlation analysis and conflict resolution, it integrates information from multiple data sources into a unified data view to ensure data consistency and integrity. The adaptive data compression and decompression module uses adaptive data compression technology to reduce the transmission bandwidth and storage space occupied during the data access stage. During the preprocessing stage, it performs intelligent decompression according to processing requirements to ensure the availability and efficiency of data during the processing.
[0008] The visual data quality monitoring module provides a visual data quality monitoring interface that displays the quantity, quality indicators and preprocessing effects of the accessed data in real time. Through intuitive charts and alarm mechanisms, it helps users to promptly identify data problems and perform corresponding processing optimizations.
[0009] As a further technical solution of the present invention: in the dynamic data cleaning strategy module, a data quality assessment algorithm is used to perform real-time quality checks on the accessed data, including checking the integrity, accuracy, consistency and repeatability of the data, and dynamically judging the quality status of the data by real-time monitoring of data quality indicators, such as the proportion of missing values and the number of outliers; And build a library containing various cleaning rules, such as deduplication rules, format conversion rules, and missing value filling rules. According to the real-time data quality assessment results, the appropriate cleaning rules are automatically selected from the rule library for application.
[0010] As a further technical solution of the present invention: in the dynamic data cleaning strategy module, for the continuously accessed data stream, an incremental cleaning method is adopted, that is, only the newly accessed data is cleaned to avoid repeated processing of the cleaned data, and the status of the cleaned data is tracked and recorded by maintaining a cleaning status table or using timestamp technology to ensure the accuracy of incremental cleaning; Use machine learning or statistical methods to detect abnormal values in the data, such as outliers and mutation points, and automatically take corresponding processing measures according to the type and severity of the abnormal data, such as filtering, replacing or correcting the abnormal values.
[0011] As a further technical solution of the present invention: in the dynamic data cleaning strategy module, during the cleaning process, cleaning effect indicators are collected and displayed in real time, such as data comparison before and after cleaning and cleaning efficiency, and the parameters and rules of the cleaning strategy are dynamically adjusted according to the feedback results to optimize the cleaning effect; For large-scale data sets, parallelization and distributed processing technologies are used to accelerate the data cleaning process. By dividing the data into multiple subsets and assigning them to different processing nodes, parallel cleaning operations are achieved to improve overall processing efficiency.
[0012] As a further technical solution of the present invention: in the cloud resource scheduling unit, the intelligent scheduling algorithm includes resource demand prediction: using machine learning or deep learning models to analyze historical data processing tasks to predict the demand for computing nodes, storage space and network bandwidth for future tasks, and monitoring task queues and resource usage in real time, and dynamically adjusting the prediction model to improve accuracy; Multi-objective optimization: define multiple optimization goals, such as minimizing task completion time, maximizing resource utilization, and minimizing energy consumption, and use multi-objective optimization algorithms, such as genetic algorithms and particle swarm optimization, to find the best resource allocation solution that meets multiple goals; Load balancing strategy: Use load balancing algorithms to ensure that the load between computing nodes is evenly distributed, avoid overloading some nodes while other nodes are idle, consider the dependencies and priorities of tasks, and reasonably arrange the execution order of tasks and the allocated computing resources.
[0013] Dynamic resource adjustment: Dynamically adjust the allocated computing nodes, storage space, and network bandwidth according to the real-time execution of the task, and achieve elastic resource expansion and contraction to cope with sudden traffic or task peaks.
[0014] Fault recovery and fault tolerance mechanisms: Use fault detection and recovery strategies to timely reallocate tasks and resources when computing nodes or networks fail, and use data redundancy and backup mechanisms to ensure data integrity and availability when failures occur.
[0015] As a further technical solution of the present invention: the data processing unit includes: The data partitioning module uses the partitioning mechanism of Spark or Flink to divide the preprocessed data into multiple logical partitions, each of which can be processed independently in parallel; Task allocation module: In Spark, data is operated by defining RDD (Resilient Distributed Dataset) or DataFrame / DataSet API, and its built-in parallelization mechanism is used to automatically allocate data to multiple nodes in the cluster for processing; in Flink, data streams (DataStream) or batch data sets (DataSet) are created, and the degree of parallelism is specified to achieve parallel allocation of tasks; Computing logic module, write the corresponding computing logic, such as map, reduce, and join operations, to process the data in each partition. These operations are automatically executed in parallel on multiple nodes in the cluster.
[0016] As a further technical solution of the present invention: the data processing unit further includes: Resource allocation module, which reasonably allocates the resources of Spark or Flink cluster, such as CPU, memory, and storage, according to the scale of data and computational complexity to ensure efficient parallel processing; Partition strategy optimization module adjusts data partition strategy according to actual needs and data characteristics, such as using partition keys or custom partition functions to improve the efficiency and accuracy of data processing; The cache and persistence module uses Spark or Flink's cache mechanism to store frequently accessed data or intermediate results persistently, thus reducing unnecessary repeated calculations and network transmission overhead. Fault-tolerance mechanism module, which uses the fault-tolerance mechanisms provided by Spark or Flink, such as checkpoints and state backends, to ensure that data can be recovered and continue to be processed when a node fails or the network is abnormal; The monitoring and logging module integrates monitoring tools and logging systems to monitor the cluster status, performance indicators, and task execution in real time, so as to detect problems and make adjustments in a timely manner.
[0017] The implementation method of efficient data processing based on cloud-network integration specifically includes the following steps: Step 1: The data access unit receives data from different data sources in real time; Step 2: The data preprocessing unit performs cleaning, deduplication, and format conversion preprocessing operations on the received raw data to ensure data quality and consistency; Step 3: The cloud resource scheduling unit dynamically allocates cloud computing resources, including computing nodes, storage space, and network bandwidth, according to the needs of data processing tasks, and achieves optimal resource configuration and load balancing through intelligent scheduling algorithms; Step 4: The data processing unit uses Spark or Flink distributed computing framework to process the preprocessed data in parallel; Step 5: The data storage and output unit stores the processed data in a distributed file system or database, provides data query, analysis and visualization functions according to business requirements, and outputs the processing results to a specified target system or application; Step 6: The data monitoring unit monitors the system's operating status, resource utilization, and performance indicators in real time.
[0018] In summary, the present invention includes at least one of the following beneficial technical effects: 1. The present invention discloses an efficient data processing system based on cloud-network integration and its implementation method. This system realizes efficient utilization and dynamic expansion of data processing resources through cloud-network integration technology, reduces data transmission delay, and improves data processing efficiency and system stability. At the same time, through intelligent monitoring and optimization means, it ensures the continuous high-performance operation of the system.
[0019] 2. The system of the present invention has laid a solid foundation for efficient data processing through the close integration of the data access unit and the data preprocessing unit. The real-time receiving capability of the data access unit enables the system to seamlessly connect to various data sources, and both structured and unstructured data can be captured quickly and accurately by the system. The data preprocessing unit performs refined processing on these data, including operations such as cleaning, deduplication, and format conversion, thereby ensuring the quality and consistency of the data. The technical effect of this link is reflected in that it provides a clean and regular data foundation for subsequent data processing, avoiding processing errors and reduced efficiency due to data quality issues.
[0020] 3. The synergy between the cloud resource scheduling unit and the data processing unit of the present invention enables the system to perform well in processing large-scale data. The cloud resource scheduling unit can dynamically allocate cloud computing resources, such as computing nodes, storage space, and network bandwidth, according to the actual needs of the data processing tasks. This intelligent scheduling not only realizes the optimal configuration of resources, but also ensures load balancing, so that the system can still maintain high efficiency and stability when facing high-concurrency processing tasks. The data processing unit adopts the advanced Spark or Flink distributed computing framework to perform parallel processing of pre-processed data. This processing method makes full use of the advantages of distributed computing and greatly improves the speed and efficiency of data processing. The joint action of these two units enables the system to be able to handle massive amounts of data with ease and easily cope with various complex data processing scenarios.
[0021] 4. The perfect combination of the data storage and output unit and the monitoring unit of the present invention provides a strong guarantee for the stability and reliability of the system. The data storage and output unit stores the processed data securely in a distributed file system or database, and provides flexible data query, analysis and visualization functions according to business needs. This enables users to easily obtain the required data information and present it in an intuitive way, providing strong data support for decision support. At the same time, the unit can also accurately output the processing results to the specified target system or application, realizing seamless connection and efficient use of data. The monitoring unit monitors the operating status, resource utilization and performance indicators of the entire system in real time. Once an abnormal situation is found, the early warning mechanism can be triggered immediately to notify the management personnel to deal with it in time. This comprehensive monitoring mechanism ensures the stable operation and efficient performance of the system, and greatly reduces the risk of system failure and maintenance costs.
[0022] In summary, the efficient data processing system based on cloud-network integration achieves excellent technical results through the close cooperation and collaborative work of various units. It not only improves the efficiency and flexibility of data processing, but also ensures the stability and reliability of the system. This efficient and intelligent data processing method will win more competitive advantages and development opportunities for enterprises and organizations in the data age. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] Figure 1 Flow chart of the method of the present invention. DETAILED DESCRIPTION
[0024] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application; obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments, and all other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application without making creative work are within the scope of protection of the present application.
[0025] In the description of the present application, it should be noted that the terms "upper", "lower", "inner", "outer", "top / bottom" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific position, be constructed and operated in a specific position, and therefore cannot be understood as limiting the present application. In addition, the terms "first" and "second" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.
[0026] In the description of this application, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "provided with", "set / connected", "connected", etc. should be understood in a broad sense. For example, "connected" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.
[0027] Embodiment 1: The present invention discloses an efficient data processing system based on cloud-network integration, comprising a data access unit, a data preprocessing unit, a cloud resource scheduling unit, a data processing unit, a data storage and output unit, a monitoring unit and an optimization unit which are sequentially connected in communication; The data access unit is used to receive data from different data sources in real time; The data preprocessing unit is used to perform cleaning, deduplication, and format conversion preprocessing operations on the received raw data to ensure data quality and consistency; The cloud resource scheduling unit dynamically allocates cloud computing resources, including computing nodes, storage space, and network bandwidth, according to the needs of data processing tasks, and achieves optimal resource configuration and load balancing through intelligent scheduling algorithms; The data processing unit uses Spark or Flink distributed computing framework to process the preprocessed data in parallel; The data storage and output unit stores the processed data in a distributed file system or database, provides data query, analysis, and visualization functions according to business needs, and outputs the processing results to the specified target system or application; The data monitoring unit is used to monitor the system's operating status, resource utilization and performance indicators in real time.
[0028] The data access unit receives data from different data sources in real time, which specifically includes the following steps: Data source identification and configuration,First, the system needs to be able to identify and configure various types of data sources,,including but not limited to relational databases, NoSQL databases, API interfaces, IoT sensor data streams,,social media platforms, log files, etc. For each data source, configure the corresponding,connection parameters, such as URL, port, authentication information, etc.
[0029] Data access adapter development: For each data source type, develop dedicated data access adapters. These adapters are responsible for data interaction according to the data source's specific protocol and format. The adapter should have the ability to monitor data source changes in real time to ensure that newly generated data is captured in a timely manner.
[0030] Real-time data streams are established by using adapters to establish long connections or polling mechanisms with data sources to receive data streams in real time. For data sources that support push mechanisms (such as message queues such as Kafka and RabbitMQ), the adapter should be able to subscribe to the corresponding topics or queues to achieve real-time data reception.
[0031] Data format conversion and standardization: Since different data sources may provide different data formats, the adapter needs to perform format conversion while receiving the data, converting the data into a format that can be processed within the system. This step may involve parsing data formats such as JSON and XML, extracting key fields, and converting them into a unified data structure.
[0032] Data transmission and buffering: The received data is temporarily stored through an internal message queue or cache mechanism to ensure that the data is not lost before processing. Use efficient data transmission protocols (such as gRPC, HTTP / 2, etc.) to transfer data from the adapter to subsequent processing units.
[0033] Error handling and logging: During data access, a robust error handling mechanism should be implemented, including retry strategies, timeout control, etc., to ensure stable data reception. All access activities should be logged for subsequent tracking and debugging.
[0034] Extensibility and configuration management,Extensibility should be considered when designing the data access unit so that support for new data sources can be easily added in the future.Provide a configuration management interface or API to allow users or administrators to dynamically add, delete, or modify data source configurations.
[0035] Through the above steps, the data access unit can receive data from a variety of different data sources in real time and efficiently, providing a stable data flow for subsequent data processing and analysis.
[0036] In this embodiment, the data preprocessing unit includes: Intelligent data identification and classification module uses natural language processing and machine learning technology to intelligently identify and classify the incoming data, and automatically divides the data into corresponding data sets according to the type, structure or semantic information of the data, so as to facilitate subsequent targeted processing; Dynamic data cleaning strategy module, through dynamic data cleaning strategy, adjusts cleaning rules according to the real-time quality of data; The incremental data preprocessing module adopts an incremental preprocessing method for continuously connected data streams, that is, only necessary preprocessing operations are performed on newly connected data to avoid repeated processing of the entire data set, thereby improving processing efficiency.
[0037] The multi-source data fusion module uses multi-source data fusion technology to comprehensively process data from different data sources. Through data alignment, correlation analysis and conflict resolution, it integrates information from multiple data sources into a unified data view to ensure data consistency and integrity. The adaptive data compression and decompression module uses adaptive data compression technology to reduce the transmission bandwidth and storage space occupied during the data access stage. During the preprocessing stage, it performs intelligent decompression according to processing requirements to ensure the availability and efficiency of data during the processing.
[0038] The visual data quality monitoring module provides a visual data quality monitoring interface that displays the quantity, quality indicators and preprocessing effects of the accessed data in real time. Through intuitive charts and alarm mechanisms, it helps users to promptly identify data problems and perform corresponding processing optimizations.
[0039] In the dynamic data cleaning strategy module, the data quality assessment algorithm is used to perform real-time quality checks on the accessed data, including checking the integrity, accuracy, consistency and repeatability of the data. By real-time monitoring of data quality indicators, such as the proportion of missing values and the number of outliers, the quality status of the data can be dynamically judged. A library containing multiple cleaning rules is built, such as deduplication rules, format conversion rules, and missing value filling rules. According to the real-time data quality assessment results, appropriate cleaning rules are automatically selected from the rule library for application.
[0040] The specific technical content of the data quality assessment algorithm includes the following aspects: Integrity check: Use data profiling technology to scan the accessed data set and count the proportion of non-null values in each field, so as to identify which fields may have missing value problems. Set a threshold. When the proportion of missing values in a field exceeds the preset threshold, a warning is triggered or data filling operations are automatically performed (such as using statistical methods such as mean, median, mode, etc.).
[0041] Accuracy verification: Apply business rule verification to check whether the data value meets business requirements based on predefined business logic and data range, such as whether the date format is correct, whether the value is within a reasonable range, etc. Use machine learning algorithms, such as classifiers or regression models, to predict the data and compare it with the actual value to identify outliers or erroneous data.
[0042] Consistency check: Implement cross-field or cross-dataset comparison to ensure logical consistency between related data fields, for example, the attribute values of the same entity in different data tables should be the same. Use data version control technology to track data change history and ensure data consistency at different time points or between different systems.
[0043] Duplicate detection: Apply data deduplication algorithms, such as hash-based duplicate detection, to identify and remove duplicate records in the data set. For near-duplicate data, fuzzy matching algorithms (such as Levenshtein distance, Jaccard similarity, etc.) can be used to detect and process similar but not identical records.
[0044] Real-time monitoring and dynamic judgment: Establish a real-time data stream processing framework to continuously check the quality of the incoming data. Define key quality indicators (KQIs), such as the proportion of missing values and the number of outliers, and use dashboards or alarm systems to display the changes of these indicators in real time. Based on the preset quality thresholds and real-time monitoring results, dynamically judge the quality of the data and trigger the corresponding processing flow, such as data cleaning, data correction or data isolation.
[0045] The specific technical content of the data quality assessment algorithm covers multiple aspects, including integrity check, accuracy verification, consistency check, repeatability detection, real-time monitoring and dynamic judgment. The comprehensive application of these technical contents can effectively ensure the accuracy and reliability of access data, laying a solid foundation for subsequent data analysis and application.
[0046] In the dynamic data cleaning strategy module, an incremental cleaning method is adopted for continuously connected data streams, that is, only newly connected data is cleaned to avoid repeated processing of cleaned data. By maintaining a cleaning status table or using timestamp technology, the status of cleaned data is tracked and recorded to ensure the accuracy of incremental cleaning; machine learning or statistical methods are used to detect outliers in the data, such as outliers and mutation points, and corresponding processing measures are automatically taken according to the type and severity of the abnormal data, such as filtering, replacing or correcting the outliers.
[0047] In the dynamic data cleaning strategy module, during the cleaning process, cleaning effect indicators are collected and displayed in real time, such as data comparison before and after cleaning and cleaning efficiency. According to the feedback results, the parameters and rules of the cleaning strategy are dynamically adjusted to optimize the cleaning effect. For large-scale data sets, parallelization and distributed processing technologies are used to accelerate the data cleaning process. By dividing the data into multiple subsets and assigning them to different processing nodes, parallel cleaning operations are achieved to improve the overall processing efficiency.
[0048] In summary, the dynamic data cleaning strategy can flexibly respond to various data cleaning needs and improve data quality and processing efficiency by combining real-time data quality assessment, adaptive cleaning rule base, incremental cleaning technology, abnormal data detection and processing, and real-time feedback and adjustment of cleaning effects.
[0049] In the cloud resource scheduling unit, the intelligent scheduling algorithm includes resource demand prediction: using machine learning or deep learning models to analyze historical data processing tasks to predict future tasks' requirements for computing nodes, storage space, and network bandwidth, and monitor task queues and resource usage in real time, dynamically adjusting the prediction model to improve accuracy.
[0050] In the cloud resource scheduling unit, one of the cores of the intelligent scheduling algorithm is resource demand prediction, and machine learning or deep learning models play a key role in this process. These models can conduct in-depth analysis of historical data processing tasks to accurately predict the requirements of future tasks for various resources, including computing nodes, storage space, and network bandwidth. Machine learning models, such as random forests and support vector machines, can predict the resource requirements of new tasks by learning the characteristics of historical tasks (such as task type, data volume, processing time, etc.). These models can capture the complex relationship between task characteristics and resource requirements, so as to make relatively accurate predictions. Deep learning models, especially recurrent neural networks (RNNs) and long short-term memory networks (LSTMs), are more suitable for processing data with time series characteristics. They can learn the time dependency of historical task resource usage, so as to make more accurate predictions of future resource requirements. The power of deep learning models lies in their ability to automatically extract high-level feature representations, which makes them perform well in processing complex and nonlinear relationships. In order to improve the accuracy of predictions, these machine learning or deep learning models also need to be dynamically adjusted in combination with real-time monitoring of task queues and resource usage. For example, when it is found that there is a large deviation between actual resource usage and prediction, the model can relearn and adjust its parameters to adapt to the new task characteristics or resource usage patterns. In general, machine learning or deep learning models play an important role in resource demand prediction in cloud resource scheduling. They can help the system allocate resources more intelligently and improve resource utilization and processing efficiency.
[0051] Multi-objective optimization: define multiple optimization goals, such as minimizing task completion time, maximizing resource utilization, and minimizing energy consumption, and use multi-objective optimization algorithms, such as genetic algorithms and particle swarm optimization, to find the best resource allocation solution that meets multiple goals.
[0052] Among them, the genetic algorithm iteratively searches for the optimal solution through three basic operations: selection, crossover, and mutation. Selection: select excellent individuals to enter the next generation based on their fitness. The fitness function is usually designed based on multiple goals of the problem. Crossover: randomly select two individuals as parents, and generate new offspring through crossover operations, which inherit the excellent characteristics of the parents. Mutation: Random mutation of newly generated offspring with a certain probability to increase the diversity of the population and avoid falling into the local optimum. Genetic algorithms have strong global search capabilities and are suitable for complex multi-objective optimization problems, but may require longer computing time and a larger population size to find a satisfactory solution.
[0053] In particle swarm optimization, each particle represents a potential solution to the problem, with two properties: position and speed. Particles update their speed and position based on their own historical best position and the historical best position of the group. Update strategy: The speed and position update formula of particles usually contains inertia weight, individual learning factor and social learning factor to balance global search and local search capabilities; the particle swarm optimization algorithm is simple and easy to implement, converges quickly, and is suitable for multi-objective optimization problems in continuous space; but it may easily fall into local optimality, and it is necessary to enhance the global search capability by adjusting parameters and introducing mutation strategies.
[0054] Load balancing strategy: Use load balancing algorithms to ensure that the load between computing nodes is evenly distributed, avoid overloading some nodes while other nodes are idle, consider the dependencies and priorities of tasks, and reasonably arrange the execution order of tasks and the allocated computing resources.
[0055] Dynamic resource adjustment: Dynamically adjust the allocated computing nodes, storage space, and network bandwidth according to the real-time execution of the task, and achieve elastic resource expansion and contraction to cope with sudden traffic or task peaks.
[0056] Fault recovery and fault tolerance mechanisms: Use fault detection and recovery strategies to timely reallocate tasks and resources when computing nodes or networks fail, and use data redundancy and backup mechanisms to ensure data integrity and availability when failures occur.
[0057] In this embodiment, the data processing unit includes: The data partitioning module uses the partitioning mechanism of Spark or Flink to divide the preprocessed data into multiple logical partitions, each of which can be processed independently in parallel; Task allocation module: In Spark, data is operated by defining RDD (Resilient Distributed Dataset) or DataFrame / DataSet API, and its built-in parallelization mechanism is used to automatically allocate data to multiple nodes in the cluster for processing; in Flink, data streams (DataStream) or batch data sets (DataSet) are created, and the degree of parallelism is specified to achieve parallel allocation of tasks; Computing logic module, write the corresponding computing logic, such as map, reduce, and join operations, to process the data in each partition. These operations are automatically executed in parallel on multiple nodes in the cluster.
[0058] Resource allocation module, which reasonably allocates the resources of Spark or Flink cluster, such as CPU, memory, and storage, according to the scale of data and computational complexity to ensure efficient parallel processing; Partition strategy optimization module adjusts data partition strategy according to actual needs and data characteristics, such as using partition keys or custom partition functions to improve the efficiency and accuracy of data processing; The cache and persistence module uses Spark or Flink's cache mechanism to store frequently accessed data or intermediate results persistently, thus reducing unnecessary repeated calculations and network transmission overhead. Fault-tolerance mechanism module, which uses the fault-tolerance mechanisms provided by Spark or Flink, such as checkpoints and state backends, to ensure that data can be recovered and continue to be processed when a node fails or the network is abnormal; The monitoring and logging module integrates monitoring tools and logging systems to monitor the cluster status, performance indicators, and task execution in real time, so as to detect problems and make adjustments in a timely manner.
[0059] In this embodiment, the data storage and output unit includes the following core modules: data storage module, data query and analysis module, data visualization module, and data output module. Each module has its own unique function, and they work together to achieve efficient storage, query, analysis and output of data.
[0060] The data storage module is responsible for storing the processed data safely and reliably in a distributed file system or database. It can intelligently select the appropriate storage strategy based on factors such as data type, size, and access frequency to ensure efficient storage and fast access to data.
[0061] The data query and analysis module is used to provide powerful data query and analysis functions, supporting multiple query languages such as SQL and NoSQL, as well as advanced analysis algorithms such as data mining and machine learning. Users can easily obtain the required data through this module and conduct in-depth analysis and mining.
[0062] The data visualization module is used to display data in intuitive forms such as charts, images, and animations to help users better understand the data and analysis results. This module supports a variety of visualization tools and libraries, such as Tableau, D3.js, etc., to meet users' different visualization needs.
[0063] The data output module is responsible for outputting the processing results to the specified target system or application. This module can output the results in various ways such as API interface, file transfer, data stream, etc. according to business needs to ensure that the data can flow smoothly to the next link.
[0064] Working principle: The entire data storage and output unit realizes the functions of data storage, query, analysis and output through the cooperation between various modules. When there is new data to be stored, the data storage module will be responsible for storing it safely and reliably in the distributed file system or database; when the user needs to query or analyze the data, the data query and analysis module will provide the corresponding functions; when the user needs to view the visualization of the data, the data visualization module will display the analysis results in an intuitive form; finally, when the processing results need to be output to the specified target system or application, the data output module will be responsible for completing this task. Throughout the process, efficient data transmission and interaction mechanisms are used between various modules to ensure the smooth flow and correct processing of data.
[0065] In the data monitoring unit, it is a key component to ensure the stable operation of the system, including the alarm and notification module, the visualization module, and the storage and archiving module. The purpose of the alarm and notification module is: when the system performance exceeds the preset threshold or detects abnormal behavior, the module will trigger the alarm mechanism and promptly notify the administrator or relevant personnel through email, SMS, instant messaging, etc. This ensures that the problem is responded to and handled in a timely manner. The visualization module visualizes the processed data in the form of charts, curves, dashboards, etc., so that the administrator can intuitively understand the operation status and performance of the system. This module supports custom views and reports to meet different monitoring needs. The storage and archiving module is responsible for compressing, encrypting and archiving historical data for subsequent data analysis, auditing and tracing. This module can ensure the integrity and security of data while optimizing the efficiency of storage space.
[0066] Embodiment 2: Reference Figure 1 , the implementation method of efficient data processing based on cloud network integration specifically includes the following steps: Step 1: The data access unit receives data from different data sources in real time; Step 2: The data preprocessing unit performs cleaning, deduplication, and format conversion preprocessing operations on the received raw data to ensure data quality and consistency; Step 3: The cloud resource scheduling unit dynamically allocates cloud computing resources, including computing nodes, storage space, and network bandwidth, according to the needs of data processing tasks, and achieves optimal resource configuration and load balancing through intelligent scheduling algorithms; Step 4: The data processing unit uses Spark or Flink distributed computing framework to process the preprocessed data in parallel; Step 5: The data storage and output unit stores the processed data in a distributed file system or database, provides data query, analysis and visualization functions according to business requirements, and outputs the processing results to a specified target system or application; Step 6: The data monitoring unit monitors the system's operating status, resource utilization, and performance indicators in real time.
[0067] The implementation principle of the present invention is as follows: The present invention discloses an efficient data processing system based on cloud-network integration and its implementation method. This system realizes efficient utilization and dynamic expansion of data processing resources through cloud-network integration technology, reduces data transmission delay, and improves data processing efficiency and system stability. At the same time, through intelligent monitoring and optimization means, the continuous high-performance operation of the system is ensured.
[0068] The embodiments of this specific implementation method are all preferred embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Therefore, any equivalent changes made based on the structure, shape, and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. An efficient data processing system based on cloud-network integration, characterized by: It includes a data access unit, a data preprocessing unit, a cloud resource scheduling unit, a data processing unit, a data storage and output unit, a monitoring unit and an optimization unit which are sequentially connected in communication; The data access unit is used to receive data from different data sources in real time; The data preprocessing unit is used to perform cleaning, deduplication, and format conversion preprocessing operations on the received raw data to ensure data quality and consistency; The cloud resource scheduling unit dynamically allocates cloud computing resources, including computing nodes, storage space, and network bandwidth, according to the needs of data processing tasks, and achieves optimal resource configuration and load balancing through intelligent scheduling algorithms; The data processing unit uses Spark or Flink distributed computing framework to perform parallel processing on the preprocessed data; The data storage and output unit stores the processed data in a distributed file system or database, provides data query, analysis and visualization functions according to business requirements, and outputs the processing results to a specified target system or application; The data monitoring unit is used to monitor the operating status, resource utilization and performance indicators of the system in real time.
2. The efficient data processing system based on cloud-network integration according to claim 1 is characterized in that: The data preprocessing unit comprises: Intelligent data identification and classification module uses natural language processing and machine learning technology to intelligently identify and classify the incoming data, and automatically divides the data into corresponding data sets according to the type, structure or semantic information of the data, so as to facilitate subsequent targeted processing; Dynamic data cleaning strategy module, through dynamic data cleaning strategy, adjusts cleaning rules according to the real-time quality of data; The incremental data preprocessing module adopts an incremental preprocessing method for continuously accessed data streams, that is, only necessary preprocessing operations are performed on newly accessed data to avoid repeated processing of the entire data set, thereby improving processing efficiency; The multi-source data fusion module uses multi-source data fusion technology to comprehensively process data from different data sources. Through data alignment, correlation analysis and conflict resolution, it integrates information from multiple data sources into a unified data view to ensure data consistency and integrity. Adaptive data compression and decompression module: In the data access stage, adaptive data compression technology is used to reduce the transmission bandwidth and storage space occupation. In the pre-processing stage, intelligent decompression is performed according to the processing requirements to ensure the availability and efficiency of data during the processing; The visual data quality monitoring module provides a visual data quality monitoring interface that displays the quantity, quality indicators and preprocessing effects of the accessed data in real time. Through intuitive charts and alarm mechanisms, it helps users to promptly identify data problems and perform corresponding processing optimizations.
3. The efficient data processing system based on cloud-network integration according to claim 2 is characterized in that: In the dynamic data cleaning strategy module, a data quality assessment algorithm is used to perform real-time quality checks on the accessed data, including checking the integrity, accuracy, consistency and repeatability of the data, and dynamically judging the quality status of the data by real-time monitoring of data quality indicators, such as the proportion of missing values and the number of outliers; And build a library containing various cleaning rules, such as deduplication rules, format conversion rules, and missing value filling rules. According to the real-time data quality assessment results, the appropriate cleaning rules are automatically selected from the rule library for application.
4. The efficient data processing system based on cloud-network integration according to claim 2 is characterized in that: In the dynamic data cleaning strategy module, for continuously accessed data streams, an incremental cleaning method is adopted, that is, only the newly accessed data is cleaned to avoid repeated processing of the cleaned data. By maintaining a cleaning status table or using timestamp technology, the status of the cleaned data is tracked and recorded to ensure the accuracy of incremental cleaning; Use machine learning or statistical methods to detect abnormal values in the data, such as outliers and mutation points, and automatically take corresponding processing measures according to the type and severity of the abnormal data, such as filtering, replacing or correcting the abnormal values.
5. The efficient data processing system based on cloud-network integration according to claim 2 is characterized in that: In the dynamic data cleaning strategy module, during the cleaning process, cleaning effect indicators are collected and displayed in real time, such as data comparison before and after cleaning and cleaning efficiency. According to the feedback results, the parameters and rules of the cleaning strategy are dynamically adjusted to optimize the cleaning effect; For large-scale data sets, parallelization and distributed processing technologies are used to accelerate the data cleaning process. By dividing the data into multiple subsets and assigning them to different processing nodes, parallel cleaning operations are achieved to improve overall processing efficiency.
6. The efficient data processing system based on cloud-network integration according to claim 1 is characterized in that: In the cloud resource scheduling unit, the intelligent scheduling algorithm includes resource demand prediction: using machine learning or deep learning models to analyze historical data processing tasks to predict the demand for computing nodes, storage space and network bandwidth of future tasks, and monitoring task queues and resource usage in real time, and dynamically adjusting the prediction model to improve accuracy; Multi-objective optimization: define multiple optimization goals, such as minimizing task completion time, maximizing resource utilization, and minimizing energy consumption, and use multi-objective optimization algorithms, such as genetic algorithms and particle swarm optimization, to find the best resource allocation solution that meets multiple goals; Load balancing strategy: Use load balancing algorithms to ensure that the load between computing nodes is evenly distributed, avoid overloading some nodes while other nodes are idle, consider the dependencies and priorities of tasks, and reasonably arrange the execution order of tasks and the allocated computing resources; Dynamic resource adjustment: Dynamically adjust the allocated computing nodes, storage space, and network bandwidth according to the real-time execution of tasks, and realize elastic resource expansion and contraction to cope with sudden traffic or task peaks; Fault recovery and fault tolerance mechanisms: Use fault detection and recovery strategies to timely reallocate tasks and resources when computing nodes or networks fail, and use data redundancy and backup mechanisms to ensure data integrity and availability when failures occur.
7. The efficient data processing system based on cloud-network integration according to claim 1 is characterized in that: The data processing unit comprises: The data partitioning module uses the partitioning mechanism of Spark or Flink to divide the preprocessed data into multiple logical partitions, each of which can be processed independently in parallel; Task allocation module: In Spark, data is operated by defining RDD (Resilient Distributed Dataset) or DataFrame / DataSet API, and its built-in parallelization mechanism is used to automatically allocate data to multiple nodes in the cluster for processing; in Flink, data streams (DataStream) or batch data sets (DataSet) are created, and the degree of parallelism is specified to achieve parallel allocation of tasks; Computing logic module, write the corresponding computing logic, such as map, reduce, and join operations, to process the data in each partition. These operations are automatically executed in parallel on multiple nodes in the cluster.
8. The efficient data processing system based on cloud-network integration according to claim 1 is characterized in that: The data processing unit also includes: Resource allocation module, which reasonably allocates the resources of Spark or Flink cluster, such as CPU, memory, and storage, according to the scale of data and computational complexity to ensure efficient parallel processing; Partition strategy optimization module adjusts data partition strategy according to actual needs and data characteristics, such as using partition keys or custom partition functions to improve the efficiency and accuracy of data processing; The cache and persistence module uses Spark or Flink's cache mechanism to store frequently accessed data or intermediate results persistently, thus reducing unnecessary repeated calculations and network transmission overhead. Fault-tolerance mechanism module, which uses the fault-tolerance mechanisms provided by Spark or Flink, such as checkpoints and state backends, to ensure that data can be recovered and continue to be processed when a node fails or the network is abnormal; The monitoring and logging module integrates monitoring tools and logging systems to monitor the cluster status, performance indicators, and task execution in real time, so as to detect problems and make adjustments in a timely manner.
9. A method for implementing efficient data processing based on cloud-network integration, characterized in that: The specific steps include: Step 1: The data access unit receives data from different data sources in real time; Step 2: The data preprocessing unit performs cleaning, deduplication, and format conversion preprocessing operations on the received raw data to ensure data quality and consistency; Step 3: The cloud resource scheduling unit dynamically allocates cloud computing resources, including computing nodes, storage space, and network bandwidth, according to the needs of data processing tasks, and achieves optimal resource configuration and load balancing through intelligent scheduling algorithms; Step 4: The data processing unit uses Spark or Flink distributed computing framework to process the preprocessed data in parallel; Step 5: The data storage and output unit stores the processed data in a distributed file system or database, provides data query, analysis and visualization functions according to business requirements, and outputs the processing results to a specified target system or application; Step 6: The data monitoring unit monitors the system's operating status, resource utilization, and performance indicators in real time.
Citation Information
Patent Citations
Data analysis visualization method, system and device based on big data artificial intelligence
CN112883100A
Multi-source heterogeneous public security big data intelligent docking engine system
CN113468163A
Industrial park management system driven by cloud computing
CN117891606A
Cloud intelligent maintenance center system
CN117992307A
Cloud computing scheduling system and cloud computing scheduling method
CN118747121A
Cited By
Multivariable real-time measurement and state test method for data center
CN119782187A
Intelligent cloud platform for Internet of Things data service
CN119922189A
Natural resource data cleaning and preprocessing method and system based on large model
CN120011086A
A natural resource data cleaning and preprocessing method and system based on large models
CN120011086B
Self-adaptive dynamic data cleaning method for real-time streaming data
CN120045848A
