Data processing framework construction system and device

Abstract

The embodiment of the invention provides a data processing framework construction system and device, and the method comprises the steps: receiving data source information through a data processing end, carrying out the extraction and classification operation of the data source information, and determining corresponding metadata; and the task server performs task arrangement on the metadata, determines a corresponding task execution command, performs command optimization operation on the task execution command according to a set reinforcement learning algorithm, determines an optimal task execution command, and performs supervision operation on the execution condition of the optimal task execution command according to a set task supervision model. Determining a corresponding task execution result; and the result server performs classification and grading operation on the task execution result according to a preset classification hierarchical structure, determines a corresponding data index and outputs the data index to the user side to complete data management, and the data management method and device can improve the efficiency and accuracy of data management.

Description

Technical Field

[0001] The present application relates to the field of data processing, and specifically to a data processing framework construction system and device. Background Art

[0002] In today's era of rapid information development, data has become an important asset for corporate decision-making and business operations. With the rapid increase in data types and data volume, traditional data management and processing methods face many challenges, including low efficiency in identifying data source diversity and low data processing efficiency. Therefore, it is particularly important to build an efficient, flexible and accurate data processing framework.

[0003] In terms of data source diversity, modern enterprise IT environments have a wide variety of data sources, including relational databases (such as MySQL, Oracle), non-relational databases (such as MongoDB), and unstructured data sources (such as file storage, object storage, etc.). These data sources each have their own specific storage structure and access methods. Traditional data integration tools often lack flexibility and adaptability and cannot effectively handle the formats and structures of different data sources. This leads to high time costs for data integration and the easy creation of data islands, making it impossible for departments to share and utilize data.

[0004] In terms of data processing efficiency, as the amount of data continues to grow, many traditional data processing systems rely on batch processing mode and cannot meet the needs of real-time data analysis. Especially in the case of high concurrency, the system may have bottlenecks, resulting in processing delays and waste of resources.

[0005] Therefore, modern data processing urgently needs to build a data processing framework, which can improve the efficiency and accuracy of data management through automated data source identification and management, metadata extraction and comparison, and parallel task scheduling. Summary of the invention

[0006] In response to the problems in the prior art, the present application provides a data processing framework construction system and device, which can improve the efficiency and accuracy of data management.

[0007] In order to solve at least one of the above problems, the present application provides the following technical solutions:

[0008] In a first aspect, the present application provides a data processing framework construction system, comprising:

[0009] A data processing end is used to receive data source information, perform extraction and classification operations on the data source information, determine corresponding classified data information, perform metadata extraction and analysis operations on the classified data information according to preset metadata extraction rules and preset hash algorithms, and determine corresponding metadata;

[0010] A task server, for receiving the metadata sent by the data processing end, performing task scheduling on the metadata, determining corresponding task execution commands, performing command optimization operations on the task execution commands according to a set reinforcement learning algorithm, determining optimal task execution commands, and performing supervision operations on the execution of the optimal task execution commands according to a set task supervision model, and determining corresponding task execution results, wherein the reinforcement learning algorithm is used to adjust the task execution order and the number of concurrency;

[0011] The result server is used to receive the task execution result sent by the task server, classify and grade the task execution result according to a preset classification hierarchy structure, determine the corresponding data classification directory, extract keywords from the data classification directory according to a preset keyword extraction algorithm, determine the corresponding data tag, perform data index construction operation according to the data classification directory and the data tag, determine the corresponding data index, and output the data index to the user end to complete data management.

[0012] Further, the extracting and classifying operations on the data source information to determine the corresponding classified data information includes:

[0013] According to a preset scanning rule, a scanning operation is performed on the data source information to determine corresponding available data source information;

[0014] The available data source information is classified according to a preset K-Means algorithm to determine corresponding classified data information, wherein the classification operation is used to identify the data source type, and the classified data information includes structured data and unstructured data.

[0015] Furthermore, performing metadata extraction and analysis operations on the classified data information according to preset metadata extraction rules and preset hash algorithms to determine corresponding metadata includes:

[0016] Performing a metadata extraction operation according to a preset metadata extraction rule adapted to the classified data information to obtain first metadata, and performing a hash analysis operation on the first metadata according to a preset hash algorithm to determine a corresponding first hash value,

[0017] The first Hash value is compared with a preset historical Hash value, and if there is a change, a self-increment operation is performed on the first metadata to determine the corresponding metadata.

[0018] Furthermore, the metadata is task-arranged to determine the corresponding task execution command, including:

[0019] Performing a data node setting operation on the metadata according to a preset directed acyclic graph to determine corresponding dependency relationships;

[0020] The task scheduling operation is performed according to the dependency relationship and the preset scheduler to determine the corresponding task execution command.

[0021] Furthermore, before performing a command optimization operation on the task execution command according to the set reinforcement learning algorithm to determine the optimal task execution command, the method includes:

[0022] Collect historical task execution data, and determine a corresponding reinforcement learning environment based on the historical execution data;

[0023] The reinforcement learning environment is trained for command execution according to a preset Q-learning algorithm to determine a corresponding set reinforcement learning algorithm.

[0024] Furthermore, the command execution training is performed on the reinforcement learning environment according to the preset Q-learning algorithm to determine the corresponding set reinforcement learning algorithm, including:

[0025] Performing command execution training on the reinforcement learning environment according to a preset Q-learning algorithm, determining a corresponding training result, and evaluating the training result according to a preset reward function to determine a corresponding Q value;

[0026] The parameters of the preset Q-learning algorithm are updated according to the optimal Q value to determine the corresponding set reinforcement learning algorithm.

[0027] Furthermore, before the execution of the optimal task execution command is supervised according to the set task supervision model to determine the corresponding task execution result, the method further includes:

[0028] Collect historical task execution result data, perform model training on a preset random forest model according to the historical task execution result data, and determine corresponding training results;

[0029] The loss function is determined according to the training results and the preset validation set, and the parameters of the preset random forest model are updated according to the back propagation algorithm using the minimum loss function to determine the corresponding set task supervision model.

[0030] Further, the monitoring operation on the execution of the optimal task execution command according to the set task monitoring model to determine the corresponding task execution result includes:

[0031] Performing a supervision operation on the execution of the optimal task execution command according to a set task supervision model to determine a corresponding prediction result, wherein the prediction result is success or failure;

[0032] If the prediction result is failure, a parameter adjustment operation is performed on the optimal execution command according to the task supervision model, and a task execution operation is performed according to the parameters after the parameter adjustment operation to determine a corresponding task execution result.

[0033] In a second aspect, the present application provides a data processing framework construction device, comprising:

[0034] A data processing module is used to receive data source information according to the data processing end, extract and classify the data source information, determine corresponding classified data information, perform metadata extraction and analysis operations on the classified data information according to preset metadata extraction rules and preset hash algorithms, and determine corresponding metadata;

[0035] A task service module, configured to receive the metadata from a task service end, perform task scheduling on the metadata, determine corresponding task execution commands, perform command optimization operations on the task execution commands according to a set reinforcement learning algorithm, determine optimal task execution commands, perform supervision operations on the execution of the optimal task execution commands according to a set task supervision model, and determine corresponding task execution results, wherein the reinforcement learning algorithm is used to adjust the task execution order and the number of concurrency;

[0036] The result service module is used to receive the task execution result from the result service end, classify and grade the task execution result according to a preset classification hierarchy structure, determine the corresponding data classification directory, extract keywords from the data classification directory according to a preset keyword extraction algorithm, determine the corresponding data tag, perform data index construction operations according to the data classification directory and the data tag, determine the corresponding data index, and output the data index to the user end to complete data management.

[0037] In a third aspect, the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of constructing a data processing framework system when executing the program.

[0038] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of constructing the data processing framework system.

[0039] In a fifth aspect, the present application provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of building a data processing framework system.

[0040] It can be seen from the above technical scheme that the present application provides a data processing framework construction system and device, which receives data source information through a data processing end, extracts and classifies the data source information, and determines the corresponding metadata; the task service end performs task scheduling on the metadata, determines the corresponding task execution command, performs command optimization operations on the task execution command according to a set reinforcement learning algorithm, determines the optimal task execution command, supervises the execution of the optimal task execution command according to a set task supervision model, and determines the corresponding task execution result; the result service end classifies and grades the task execution result according to a preset classification hierarchy structure, determines the corresponding data index, and outputs the data index to the user end to complete data management, thereby improving the efficiency and accuracy of data management. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0042] Figure 1 This is one of the flow diagrams of the data processing framework construction system in the embodiment of the present application;

[0043] Figure 2 The second flowchart of the data processing framework construction system in the embodiment of the present application;

[0044] Figure 3 The third flowchart of the data processing framework construction system in the embodiment of the present application;

[0045] Figure 4 This is a fourth flowchart of a data processing framework construction system in an embodiment of the present application;

[0046] Figure 5 This is a fifth flowchart of a data processing framework construction system in an embodiment of the present application;

[0047] Figure 6 This is a sixth flowchart of a data processing framework construction system in an embodiment of the present application;

[0048] Figure 7 7 is a flowchart of a data processing framework construction system according to an embodiment of the present application;

[0049] Figure 8 This is a flowchart of a data processing framework construction system in an embodiment of the present application.

[0050] Fig. 9 A structural diagram of a data processing framework construction device in an embodiment of the present application;

[0051] Fig.10 It is a schematic diagram of the structure of an electronic device in an embodiment of the present application.

[0052] Reference numerals:

[0053] Electronic device 9600, central processing unit 9100, memory 9140, communication module 9110, input unit 9120, audio processor 9130, display 9160, power supply 9170, buffer memory 9141, application / function storage unit 9142, data storage unit 9143, driver program storage unit 9144, antenna 9111, speaker 9131, microphone 9132. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0055] The acquisition, storage, use, and processing of data in the technical solution of this application comply with the relevant provisions of national laws and regulations.

[0056] Considering the current era of rapid development of informatization, with the sharp increase in data types and data volumes, traditional data management faces the problems of low efficiency and low accuracy. The present application provides a data processing framework construction system and device, which receives data source information through a data processing end, extracts and classifies the data source information, and determines the corresponding metadata; the task service end performs task scheduling on the metadata, determines the corresponding task execution command, performs command optimization operations on the task execution command according to a set reinforcement learning algorithm, determines the optimal task execution command, and supervises the execution of the optimal task execution command according to a set task supervision model to determine the corresponding task execution result; the result service end classifies and grades the task execution result according to a preset classification hierarchy structure, determines the corresponding data index, and outputs the data index to the user end to complete data management, thereby improving the efficiency and accuracy of data management.

[0057] In order to improve the efficiency and accuracy of data management, the present application provides an embodiment of a data processing framework construction system, see Figure 1 , the data processing framework construction system specifically includes the following contents:

[0058] Step S101: The data processing end is used to receive data source information, extract and classify the data source information, determine corresponding classified data information, perform metadata extraction and analysis operations on the classified data information according to preset metadata extraction rules and preset hash algorithms, and determine corresponding metadata;

[0059] Optionally, in this embodiment, the purpose of this step is to automatically identify and manage multiple data sources, improve data management efficiency, and automatically extract and compare metadata to timely reflect data changes and generate historical versions.

[0060] Optionally, in this embodiment, the data processing end has two steps, one is to scan and classify different data sources, and the other is to analyze and extract the extracted data.

[0061] Specifically, with respect to the first point, when faced with multiple data sources, multiple data sources are scanned and identified.

[0062] First, set the nmap scanning strategy and use the -p parameter to specify the MySQL (usually 3306), Minio (usually 9000) and FTP (usually 21) ports for scanning. After the scan is completed, parse the nmap output and extract the data source information available in the data source, including IP address and service type.

[0063] Secondly, use pre-trained K-Means or decision tree algorithms to cluster and automatically classify the scan results. Based on the classification results, use the corresponding SDK or JDBC driver, provide the connection string and authentication information, and connect through code.

[0064] Specifically, when training the model, the features of the known data source (such as port, service version, etc.) can be used as training samples.

[0065] It is understandable that the K-Means algorithm can be used for classification to accurately identify the type of data source by analyzing the characteristics of the scanned data source. This identification helps to extract metadata more efficiently. For example, for structured databases (such as MySQL) and unstructured storage (such as Minio), the algorithm can select the appropriate metadata extraction method for different data source types.

[0066] Specifically, with respect to the second point, according to the classified data information, the most suitable metadata extraction rule is selected to perform metadata extraction.

[0067] For MySQL, use the SQL queries SHOW TABLES and DESCRIBE to extract table structure information.

[0068] For Minio, use the API of its SDK to obtain metadata of buckets and objects, including size, creation time, etc.

[0069] For FTP, use an FTP library (such as Apache Commons Net) to execute the LIST command to obtain file information.

[0070] After the metadata is extracted, the hash value is calculated for the metadata and compared with the previous hash value. If there is a change, the version is incremented and the old data is archived (for example, using Minio or FTP for backup).

[0071] At the same time, the data source provides a metadata query interface to the outside world, which can provide metadata incrementally (according to version changes); it also provides a data access interface to the outside world to obtain the content of the data source (database table records or file content).

[0072] It can be understood that after the above steps, in the data processing framework, the data source end has first solved the problem that traditional methods are difficult to automatically identify and manage multiple types of data sources and that manual management of metadata cannot reflect data changes in a timely manner. Through the data source end, these different port data sources can be automatically discovered and connected, and changes in data structures can be tracked, thereby improving the efficiency and accuracy of data management in extracting data and reflecting data changes.

[0073] Step S102: The task server is used to receive the metadata sent by the data processing end, perform task scheduling on the metadata, determine the corresponding task execution command, perform command optimization operation on the task execution command according to the set reinforcement learning algorithm, determine the optimal task execution command, and perform supervision operation on the execution of the optimal task execution command according to the set task supervision model to determine the corresponding task execution result, wherein the reinforcement learning algorithm is used to adjust the task execution order and the number of concurrency;

[0074] Optionally, in this embodiment, the purpose of this step is to improve the parallel processing capability and execution efficiency of task execution.

[0075] Optionally, the technical means adopted in this step is to adopt the optimal task scheduling and failure prediction mechanism to improve the task execution efficiency and reliability.

[0076] Optionally, the metadata sent by the data processing end is received, tasks are arranged for the metadata, and corresponding task execution commands are determined.

[0077] Specifically, Apache Airflow is used for task orchestration and scheduling. First, the basic parameters of DAG (directed acyclic graph) are defined, including ID, scheduling interval, default parameters, etc. Then, corresponding task nodes (Operators) are created for data extraction, processing, and storage tasks. Finally, the dependencies between tasks are set through operators to ensure that tasks are executed in sequence.

[0078] After determining the task nodes and dependencies as above, set the scheduling interval (such as daily, hourly) in the DAG definition. Use the scheduling expression, Cron expression or Airflow's predefined scheduling symbols. Obtain the task execution command that can be used for execution through task scheduling.

[0079] For example, suppose we have a data pipeline with the following tasks:

[0080] Data Extraction: Extract daily sales data from API.

[0081] Data processing: Cleaning and transformation of extracted data.

[0082] Data loading: Load the processed data into the database.

[0083] First, define DAG and task nodes.

[0084] 1. Data extraction task (Extract):

[0085] Task Description: Extract daily sales data from the specified API.

[0086] Time interval: Daily execution.

[0087] 2. Data processing task (Transform):

[0088] Task Description: Clean and transform the extracted data.

[0089] Interval: Daily (depends on the completion of the extraction task).

[0090] 3. Data loading task (Load):

[0091] Task Description: Load the processed data into the database.

[0092] Interval: Daily execution (depending on the completion of processing tasks).

[0093] Second, dependency settings.

[0094] Make sure tasks are executed in order, extract first, process second, and load third.

[0095] Finally, define the time interval.

[0096] The task will be executed once every day at midnight (00:00).

[0097] The final effect of task execution is:

[0098] Extraction task: Extract sales data from API every day.

[0099] Processing tasks: Clean and transform the extracted data on a daily basis.

[0100] Loading task: Load the processed data into the PostgreSQL database every day.

[0101] Through task scheduling, data is regularly extracted from different data sources (such as order databases, user databases), data analysis and report generation are performed, and these data processing processes are automated to ensure timely data updates and analysis. In this way, the task execution process is effectively defined and the task execution commands are determined, laying a solid foundation for subsequent optimal scheduling methods and failure prediction.

[0102] Optionally, after determining the task execution command, a command optimization operation is performed on the task execution command according to a set reinforcement learning algorithm to determine an optimal task execution command.

[0103] The purpose of this step is to optimize the task scheduling strategy using reinforcement learning algorithms, dynamically adjust the scheduling plan according to task execution and resource usage, and improve task execution efficiency and resource utilization.

[0104] Specifically, the reinforcement learning algorithm adopts the Q-learning algorithm.

[0105] First, collect the task execution data of each task, including execution time (including start time, end time and total execution time), failure rate (number of successes, number of failures, calculation failure rate), task dependencies, resource usage and other information. Calculate the expected execution time of each task (taking the failure rate into account). For example, expected execution time = execution time × (1 + failure rate).

[0106] It is understandable that historical task execution data refers to the record of system status and its changes over a period of time. By analyzing historical data, we can understand the behavior patterns and dynamic characteristics of the system. It also lays the foundation for defining the environment space through historical data in the future.

[0107] Then, based on the collected task execution data, a reinforcement learning environment of a reinforcement model is defined, including environment state, environment action, and reward function. Among them, the environment state can include the current task queue, system resource usage, execution time and failure rate of executed tasks, etc. The environment action can be to select the next task to be executed and adjust the number of concurrency (such as increase or decrease). The reward function can set the reward to the negative value of the overall execution time of the current time step, or give different rewards according to the task execution results (such as success, failure).

[0108] It is understandable that the focus of the Q-learning algorithm will be different depending on the setting of the reward function. The reward function used for optimal resource scheduling is preferably set to the execution time. If it is used to adjust the success and failure of task execution results, the reward function is set according to the success and failure results.

[0109] Next, define the Q-learning algorithm and execute it in the reinforcement learning environment.

[0110] The first step is to define the Q value of the Q-learning algorithm and create an initial Q table with rows representing states, columns representing actions, and an initial value of 0.

[0111] In the second step, we use the ε-greedy strategy to select actions: we randomly select actions with a probability of ε, and select the action with the highest Q value with a probability of 1-ε.

[0112] It can be understood that using the ε-greedy strategy can balance the relationship between exploration (trying new actions) and exploitation (selecting the currently known best action).

[0113] The third step is to update the Q value according to the reward obtained after executing the action, and learn the optimal strategy by iteratively updating the Q value table (or Q value network).

[0114] Finally, in the simulation execution of multiple task scheduling, actions are selected according to the current state, actions are executed and rewards are obtained, the Q value is updated to transfer to the next state, and the Q value table is obtained. The optimal strategy is extracted from the trained Q table, and the Q-learning algorithm corresponding to the optimal strategy is applied to actual task scheduling.

[0115] For example, suppose we are on an e-commerce platform and have multiple tasks to process, such as order verification, inventory checking, payment processing, and shipping. Each task has different execution time and resource requirements. We hope to optimize the execution order and concurrency of these tasks through Q-learning to improve the overall order processing efficiency.

[0116] 1. Task definition

[0117] Task 1: Order verification, verify whether the order information submitted by the user is complete and valid. (Time: 2 minutes, resources: 1 CPU); Task 2: Inventory check, check whether the inventory of the ordered goods is sufficient (Time: 3 minutes, resources: 1 CPU); Task 3: Payment processing, process user payment and confirm that the funds are in place (Time: 5 minutes, resources: 2 CPUs); Task 4: Delivery, arrange logistics delivery according to order information (Time: 4 minutes, resources: 1 CPU).

[0118] 2. Environmental status and actions

[0119] State: We can define the state as the currently executing task, available resources (number of CPUs), task queue, etc. For example, the state can be expressed as:

[0120] Current tasks, number of available CPUs, and completed tasks.

[0121] Action: can be defined as selecting the next task to be executed or changing the order in which tasks are to be executed. For example: perform order verification, perform inventory check, perform payment processing, perform shipment.

[0122] 3. Reward

[0123] Rewards can be defined based on task completion time, resource utilization, and customer satisfaction. For example:

[0124] Give a positive reward (e.g. +10) for completing a task.

[0125] If the task times out or has low resource utilization, a negative reward (e.g. -5) is given.

[0126] 4. Q-learning algorithm

[0127] Using the Q-learning algorithm, we can use the following steps to optimize the execution order and concurrency of tasks:

[0128] Initialize the Q-value table: Initialize the Q-values ​​of all state-action pairs to zero.

[0129] Select action: Select an action (perform the next task) according to the ε-greedy strategy.

[0130] Perform an action: Execute the selected task and observe the new state and reward.

[0131] Update Q value: Iterate each training and update the optimal Q value.

[0132] Through multiple iterations, Q-learning will learn the best task execution order and concurrency number. For example:

[0133] If payment processing is found to take a long time and consumes more resources, the algorithm may learn to prioritize inventory checks and order verification, using CPU resources to process them in parallel, before executing payment processing and shipping.

[0134] It can be understood that this step shows how to use reinforcement learning (Q-learning) to optimize the execution order and concurrency of tasks.

[0135] Optionally, a supervision operation is performed on the execution status of the optimal task execution command according to a set task supervision model to determine a corresponding task execution result.

[0136] Optionally, the purpose of this step is to implement a supervised learning model (such as random forest) to predict the possibility of task failure and automatically retry or adjust task execution parameters based on the prediction results.

[0137] Specifically, the supervised learning model selects the random forest model.

[0138] The process of training a random forest is as follows:

[0139] First, the historical task execution result data is collected, including: task ID, execution time, input parameters (such as configuration, resource usage, etc.), context information (such as system load, network status, etc.), task status (success / failure) and other fields.

[0140] Then, the data is preprocessed, including but not limited to, cleaning the data: removing missing values ​​and outliers. Feature engineering: extracting features, such as average response time, resource utilization, historical failure rate, etc. during task execution. Label encoding: converting the task status (success / failure) into a binary label (0 / 1).

[0141] Next, the dataset was divided into a training set (70%), a validation set (15%), and a test set (15%). Random forest was used as the main prediction model. The training set was used to train the random forest, and hyperparameters (such as the number of trees, maximum depth, etc.) were adjusted to optimize model performance. The cross-validation method was used to evaluate the accuracy and robustness of the model. The validation set was used to evaluate the model performance, considering indicators such as precision, recall, and F1 score. The loss function was calculated using the model training results and the validation set, and the optimal loss function was obtained through iterative training. Then, the reverse update algorithm was used to backpropagate the parameters corresponding to the optimal loss function to each functional layer of the random forest model to obtain a trained task supervision model.

[0142] Optionally, after obtaining the above-mentioned trained task supervision model, the model is used to predict the success rate of the optimal task execution command obtained by the Q-learning algorithm, and high-risk tasks are automatically retried or resource allocation is adjusted according to the predicted success rate, thereby improving the success rate and efficiency of the overall order processing.

[0143] For example, let's take the order processing of the above e-commerce platform as an example.

[0144] Suppose the random forest model predicts the following failure probability in a certain task execution:

[0145] Order verification: 10% failure probability

[0146] Inventory check: 25% chance of failure

[0147] Payment processing: 5% failure probability

[0148] Delivery: Failure probability 15%

[0149] Detecting a high probability of inventory check failure, the system decides:

[0150] 1. Automatic retry: Retry the task immediately and adjust the number of concurrent requests or increase resource allocation (such as increasing CPU / memory).

[0151] 2. Data check: Before retrying, the system checks the accuracy of the inventory data to ensure that there are no problems with the data during the next execution.

[0152] The above example shows how to combine Q-learning to optimize the order and concurrency of task execution in an e-commerce platform, and use the random forest model to predict the possibility of task failure to achieve a more efficient order processing process. This method can dynamically adjust the task execution strategy in a real-time environment to ensure the stability and reliability of the system.

[0153] Step S103: The result server is used to receive the task execution result sent by the task server, classify and grade the task execution result according to a preset classification hierarchy structure, determine the corresponding data classification directory, extract keywords from the data classification directory according to a preset keyword extraction algorithm, determine the corresponding data tag, perform data index construction operations according to the data classification directory and the data tag, determine the corresponding data index, and output the data index to the user end to complete data management.

[0154] Optionally, in this embodiment, the purpose of this step is to classify and grade the results to support fast retrieval.

[0155] Optionally, in this embodiment, the result server receives the task execution results from the task server, and each result may include multiple categories, multiple tags and a grade.

[0156] Optionally, in this embodiment, a classification hierarchy structure (such as a tree structure) is preset to classify and grade the task execution results. The classification is organized into a tree structure through the TreeCode algorithm, and each data resource saves the TreeCode of the classification leaf node. Grading is to perform a grading operation on the task execution results to determine the classification directory to which they belong.

[0157] Optionally, in this embodiment, keyword extraction uses a TF / IDF algorithm to extract keywords from the data classification directory, and determines corresponding data tags through keyword extraction.

[0158] Optionally, a data index is constructed based on the classification directory and data tags. The corresponding data index is determined and generated to facilitate rapid retrieval of data resources.

[0159] Finally, the constructed data index is output to the user end to complete data management.

[0160] For example, let’s take the e-commerce platform as an example.

[0161] First, the result server classifies the order data according to the preset classification hierarchy (such as order status classification, payment method classification, etc.). For example, the order status can be classified into "paid" and "unpaid", and the payment method can be classified into "credit card" and "PayPal".

[0162] Then, a keyword extraction algorithm is used to extract keywords from the category directory and generate data labels. For example, keyword labels such as "paid order" and "credit card payment" are extracted from the categories "paid" and "credit card".

[0163] Next, data indexing is performed based on the classification directory (such as "paid" and "credit card") and data tags (such as "paid order" and "credit card payment") to generate a reverse index structure that can be quickly retrieved.

[0164] Finally, the data index is output to the user end, which can be queried and accessed through the API interface. It supports multiple output formats (such as JSON, XML) and multiple modes (such as pull and push).

[0165] In addition, the API service can also provide incremental metadata queries after data indexing, obtain the latest metadata based on version changes, and generate signatures through API Key and Secret Key to ensure communication security.

[0166] By receiving task execution results, performing classification and grading, and extracting keywords, building a data index and outputting it to the user end, this technical solution provides an efficient data management and retrieval mechanism to ensure that users can quickly and accurately obtain the required data.

[0167] This example demonstrates how this embodiment constructs a data processing framework in which the modules are closely linked, interconnected, and highly coordinated. Efficient and accurate data processing is achieved through multi-terminal identification of data sources, automatic analysis and extraction of metadata, parallel scheduling of task processing, and classified and graded presentation and indexing of processing results.

[0168] From the above description, it can be seen that the data processing framework construction system provided in the embodiment of the present application can receive data source information through the data processing end, extract and classify the data source information, and determine the corresponding metadata; the task service end performs task scheduling on the metadata, determines the corresponding task execution command, performs command optimization operations on the task execution command according to the set reinforcement learning algorithm, determines the optimal task execution command, supervises the execution of the optimal task execution command according to the set task supervision model, and determines the corresponding task execution result; the result service end classifies and grades the task execution result according to the preset classification hierarchy structure, determines the corresponding data index, and outputs the data index to the user end to complete data management, thereby improving the efficiency and accuracy of data management.

[0169] In one embodiment of the data processing framework construction system of the present application, see Figure 2 , and can also include the following:

[0170] Step S201: Scan the data source information according to a preset scanning rule to determine corresponding available data source information;

[0171] Step S202: performing a classification operation on the available data source information according to a preset K-Means algorithm to determine corresponding classified data information, wherein the classification operation is used to identify the data source type, and the classified data information includes structured data and unstructured data.

[0172] Optionally, in this embodiment, preset scanning rules, set nmap scanning strategy, use -p parameter to specify MySQL (usually 3306), Minio (usually 9000) and FTP (usually 21) ports for scanning. After the scan is completed, parse the output of nmap to extract the available data source information in the data source, including IP address and service type.

[0173] Optionally, in this embodiment, a K-Means algorithm is preset, and features of known data sources (such as ports, service versions, etc.) can be used as training samples when training the model.

[0174] It is understandable that the K-Means algorithm can be used for classification to accurately identify the type of data source by analyzing the characteristics of the scanned data source. This identification helps to extract metadata more efficiently. For example, for structured databases (such as MySQL) and unstructured storage (such as Minio), the algorithm can select the appropriate metadata extraction method for different data source types.

[0175] Through step S202, this embodiment obtains a data source classification structure, which lays a solid foundation for subsequently selecting different metadata extraction rules according to data sources with different structures, and can improve the efficiency and accuracy of data classification.

[0176] In one embodiment of the data processing framework construction system of the present application, see Figure 3 , and can also include the following:

[0177] Step S301: performing a metadata extraction operation according to a preset metadata extraction rule adapted to the classified data information to obtain first metadata, and performing a hash analysis operation on the first metadata according to a preset hash algorithm to determine a corresponding first hash value;

[0178] Step S302: Compare the first Hash value with a preset historical Hash value. If there is a change, perform a self-increment operation on the first metadata to determine the corresponding metadata.

[0179] Optionally, in this embodiment, metadata extraction is performed on the data, and metadata extraction rules are selected specifically according to the data classification information obtained in the previous step.

[0180] For MySQL, use the SQL queries SHOW TABLES and DESCRIBE to extract table structure information.

[0181] For Minio, use the API of its SDK to obtain metadata of buckets and objects, including size, creation time, etc.

[0182] For FTP, use an FTP library (such as Apache Commons Net) to execute the LIST command to obtain file information.

[0183] After the metadata is extracted, the hash value is calculated for the metadata and compared with the previous hash value. If there is a change, the version is incremented and the old data is archived (for example, using Minio or FTP for backup).

[0184] At the same time, the data source provides a metadata query interface to the outside world, which can provide metadata incrementally (according to version changes); it also provides a data access interface to the outside world to obtain the content of the data source (database table records or file content).

[0185] It can be understood that after the above steps, in the data processing framework, the data source end has first solved the problem that traditional methods are difficult to automatically identify and manage multiple types of data sources and that manual management of metadata cannot reflect data changes in a timely manner. Through the data source end, these different port data sources can be automatically discovered and connected, and changes in data structures can be tracked, thereby improving the efficiency and accuracy of data management in extracting data and reflecting data changes.

[0186] Through step S302, this embodiment realizes connecting different port data sources, thereby improving the efficiency and accuracy of data management in extracting data and reflecting data changes.

[0187] In one embodiment of the data processing framework construction system of the present application, see Figure 4 , and can also include the following:

[0188] Step S401: performing a data node setting operation on the metadata according to a preset directed acyclic graph to determine corresponding dependency relationships;

[0189] Step S402: Perform task scheduling operations according to the dependency relationship and a preset scheduler to determine corresponding task execution commands.

[0190] Optionally, in this embodiment, Apache Airflow is used for task scheduling and scheduling. First, the basic parameters of DAG (directed acyclic graph) are defined, including ID, scheduling interval, default parameters, etc. Then, corresponding task nodes (Operators) are created for data extraction, processing and storage tasks. Finally, the dependencies between tasks are set through operators to ensure that tasks are executed in sequence.

[0191] After determining the task nodes and dependencies as above, set the scheduling interval (such as daily, hourly) in the DAG definition. Use the scheduling expression, Cron expression or Airflow's predefined scheduling symbols. Obtain the task execution command that can be used for execution through task scheduling.

[0192] For example, suppose we have a data pipeline with the following tasks:

[0193] Data Extraction: Extract daily sales data from API.

[0194] Data processing: Cleaning and transformation of extracted data.

[0195] Data loading: Load the processed data into the database.

[0196] First, define DAG and task nodes.

[0197] 1. Data extraction task (Extract):

[0198] Task Description: Extract daily sales data from the specified API.

[0199] Time interval: Daily execution.

[0200] 2. Data processing task (Transform):

[0201] Task Description: Clean and transform the extracted data.

[0202] Interval: Daily (depends on the completion of the extraction task).

[0203] 3. Data loading task (Load):

[0204] Task Description: Load the processed data into the database.

[0205] Interval: Daily execution (depending on the completion of processing tasks).

[0206] Second, dependency settings.

[0207] Make sure tasks are executed in order, extract first, process second, and load third.

[0208] Finally, define the time interval.

[0209] The task will be executed once every day at midnight (00:00).

[0210] The final effect of task execution is:

[0211] Extraction task: Extract sales data from API every day.

[0212] Processing tasks: Clean and transform the extracted data on a daily basis.

[0213] Loading task: Load the processed data into the PostgreSQL database every day.

[0214] Through task orchestration, data is regularly extracted from different data sources (such as order database, user database), data analysis and report generation are performed, and these data processing processes are automated to ensure timely data updates and analysis.

[0215] Through step S402, this embodiment successfully extracts data from different data sources (such as order database, user database) regularly through task scheduling, effectively defines the task execution process, determines the task execution command, and lays a solid foundation for subsequent search for the optimal scheduling method and failure prediction.

[0216] In one embodiment of the data processing framework construction system of the present application, see Figure 5 , and can also include the following:

[0217] Step S501: collecting historical task execution data, and determining a corresponding reinforcement learning environment according to the historical execution data;

[0218] Step S502: Perform command execution training on the reinforcement learning environment according to a preset Q-learning algorithm to determine a corresponding set reinforcement learning algorithm.

[0219] Optionally, in this embodiment, a reinforcement learning algorithm is used to optimize the task scheduling strategy, and the scheduling plan is dynamically adjusted according to the task execution status and resource usage to improve the task execution efficiency and resource utilization.

[0220] Optionally, in this embodiment, the task execution data includes execution time (including start time, end time and total execution time), failure rate (number of successes, number of failures, calculation failure rate), task dependencies, resource usage and other information. Calculate the expected execution time of each task (taking the failure rate into account). For example, expected execution time = execution time × (1 + failure rate).

[0221] Optionally, in this embodiment, the reinforcement learning environment of the reinforcement model includes environment state, environment action, and reward function. Among them, the environment state may include the current task queue, system resource usage, execution time and failure rate of executed tasks, etc. The environment action may be to select the next task to be executed and adjust the number of concurrency (such as increase or decrease). The reward function may set the reward to the negative value of the overall execution time of the current time step, or give different rewards according to the task execution result (such as success, failure).

[0222] Next, define the Q-learning algorithm and execute it in the reinforcement learning environment.

[0223] Finally, in the simulation execution of multiple task scheduling, actions are selected according to the current state, actions are executed and rewards are obtained, the Q value is updated to transfer to the next state, and the Q value table is obtained. The optimal strategy is extracted from the trained Q table, and the Q-learning algorithm corresponding to the optimal strategy is applied to actual task scheduling.

[0224] Through step S502, this embodiment successfully obtains a reinforcement learning algorithm, which can improve task scheduling efficiency.

[0225] In one embodiment of the data processing framework construction system of the present application, see Figure 6 , and can also include the following:

[0226] Step S601: performing command execution training on the reinforcement learning environment according to a preset Q-learning algorithm, determining a corresponding training result, evaluating the training result according to a preset reward function, and determining a corresponding Q value;

[0227] Step S602: updating the parameters of the preset Q-learning algorithm according to the optimal Q value, and determining the corresponding set reinforcement learning algorithm.

[0228] Optionally, in this embodiment, a Q-learning algorithm is defined, and the defined Q-learning algorithm is executed in a reinforcement learning environment.

[0229] The first step is to define the Q value of the Q-learning algorithm and create an initial Q table with rows representing states, columns representing actions, and an initial value of 0.

[0230] In the second step, we use the ε-greedy strategy to select actions: we randomly select actions with a probability of ε, and select the action with the highest Q value with a probability of 1-ε.

[0231] It can be understood that using the ε-greedy strategy can balance the relationship between exploration (trying new actions) and exploitation (selecting the currently known best action).

[0232] The third step is to update the Q value according to the reward obtained after executing the action, and learn the optimal strategy by iteratively updating the Q value table (or Q value network).

[0233] Finally, in the simulation execution of multiple task scheduling, actions are selected according to the current state, actions are executed and rewards are obtained, the Q value is updated to transfer to the next state, and the Q value table is obtained. The optimal strategy is extracted from the trained Q table, and the Q-learning algorithm corresponding to the optimal strategy is applied to actual task scheduling.

[0234] For example, suppose we are on an e-commerce platform and have multiple tasks to process, such as order verification, inventory checking, payment processing, and shipping. Each task has different execution time and resource requirements. We hope to optimize the execution order and concurrency of these tasks through Q-learning to improve the overall order processing efficiency.

[0235] 1. Task definition

[0236] Task 1: Order verification, verify whether the order information submitted by the user is complete and valid. (Time: 2 minutes, resources: 1 CPU); Task 2: Inventory check, check whether the inventory of the ordered goods is sufficient (Time: 3 minutes, resources: 1 CPU); Task 3: Payment processing, process user payment and confirm that the funds are in place (Time: 5 minutes, resources: 2 CPUs); Task 4: Delivery, arrange logistics delivery according to order information (Time: 4 minutes, resources: 1 CPU)

[0237] 2. Environmental status and actions

[0238] State: We can define the state as the currently executing task, available resources (number of CPUs), task queue, etc. For example, the state can be expressed as:

[0239] Current tasks, number of available CPUs, and completed tasks

[0240] Action: can be defined as selecting the next task to be executed or changing the order in which tasks are to be executed. For example: perform order verification, perform inventory check, perform payment processing, perform shipment.

[0241] 3. Reward

[0242] Rewards can be defined based on task completion time, resource utilization, and customer satisfaction. For example:

[0243] Give a positive reward (e.g. +10) for completing a task.

[0244] If the task times out or has low resource utilization, a negative reward (e.g. -5) is given.

[0245] 4. Q-learning algorithm

[0246] Using the Q-learning algorithm, we can use the following steps to optimize the execution order and concurrency of tasks:

[0247] Initialize the Q-value table: Initialize the Q-values ​​of all state-action pairs to zero.

[0248] Select action: Select an action (perform the next task) according to the ε-greedy strategy.

[0249] Perform an action: Execute the selected task and observe the new state and reward.

[0250] Update Q value: Iterate each training and update the optimal Q value.

[0251] Through multiple iterations, Q-learning will learn the best task execution order and concurrency number. For example:

[0252] If payment processing is found to take a long time and consumes more resources, the algorithm may learn to prioritize inventory checks and order verification, using CPU resources to process them in parallel, before executing payment processing and shipping.

[0253] It can be understood that this step shows how to use reinforcement learning (Q-learning) to optimize the execution order and concurrency of tasks.

[0254] Through step S602, this embodiment successfully obtains a trained reinforcement learning algorithm, which can optimize the execution order and concurrency of tasks.

[0255] In one embodiment of the data processing framework construction system of the present application, see Figure 7 , and can also include the following:

[0256] Step S701: Collect historical task execution result data, perform model training on a preset random forest model according to the historical task execution result data, and determine corresponding training results;

[0257] Step S702: Determine the loss function according to the training results and the preset validation set, update the parameters of the preset random forest model with the minimum loss function according to the back propagation algorithm, and determine the corresponding set task supervision model.

[0258] Optionally, in this embodiment, the supervised learning model selects a random forest model.

[0259] The process of training a random forest is as follows:

[0260] First, the historical task execution result data is collected, including: task ID, execution time, input parameters (such as configuration, resource usage, etc.), context information (such as system load, network status, etc.), task status (success / failure) and other fields.

[0261] Then, the data is preprocessed, including but not limited to, cleaning the data: removing missing values ​​and outliers. Feature engineering: extracting features, such as average response time, resource utilization, historical failure rate, etc. during task execution. Label encoding: converting the task status (success / failure) into a binary label (0 / 1).

[0262] Next, the dataset was divided into a training set (70%), a validation set (15%), and a test set (15%). Random forest was used as the main prediction model. The training set was used to train the random forest, and hyperparameters (such as the number of trees, maximum depth, etc.) were adjusted to optimize model performance. The cross-validation method was used to evaluate the accuracy and robustness of the model. The validation set was used to evaluate the model performance, considering indicators such as precision, recall, and F1 score. The loss function was calculated using the model training results and the validation set, and the optimal loss function was obtained through iterative training. Then, the reverse update algorithm was used to backpropagate the parameters corresponding to the optimal loss function to each functional layer of the random forest model to obtain a trained task supervision model.

[0263] Through step S702, this embodiment successfully obtains the task supervision model, and can adjust resource allocation or reprocess the order according to the prediction result of the order, thereby improving the success rate and efficiency of the overall order processing.

[0264] In one embodiment of the data processing framework construction system of the present application, see Figure 8 , and can also include the following:

[0265] Step S801: performing a supervision operation on the execution of the optimal task execution command according to a set task supervision model to determine a corresponding prediction result, wherein the prediction result is success or failure;

[0266] Step S802: If the prediction result is failure, a parameter adjustment operation is performed on the optimal execution command according to the task supervision model, and a task execution operation is performed according to the parameters after the parameter adjustment operation to determine a corresponding task execution result.

[0267] Optionally, in this embodiment, after obtaining the above-mentioned trained task supervision model, the model is used to predict the success rate of the optimal task execution command obtained by the Q-learning algorithm, and automatically retry high-risk tasks or adjust resource allocation based on the predicted success rate, thereby improving the success rate and efficiency of the overall order processing.

[0268] For example, let's take the order processing of the above e-commerce platform as an example.

[0269] Suppose the random forest model predicts the following failure probability in a certain task execution:

[0270] Order verification: 10% failure probability

[0271] Inventory check: 25% chance of failure

[0272] Payment processing: 5% failure probability

[0273] Delivery: Failure probability 15%

[0274] Detecting a high probability of inventory check failure, the system decides:

[0275] 1. Automatic retry: Retry the task immediately and adjust the number of concurrent requests or increase resource allocation (such as increasing CPU / memory).

[0276] 2. Data check: Before retrying, the system checks the accuracy of the inventory data to ensure that there are no problems with the data during the next execution.

[0277] Through step S802, this embodiment successfully implements automatic retry or adjustment when the order task failure rate is high according to the task supervision model, thereby improving the success rate and efficiency of the overall order processing.

[0278] In order to improve the efficiency and accuracy of data management, the present application provides an embodiment of a data processing framework construction device for implementing all or part of the content of the data processing framework construction system, see Fig. 9 , the data processing framework construction device specifically includes the following contents:

[0279] The data processing module 10 is used for the data processing end, and is used to receive data source information, extract and classify the data source information, determine corresponding classified data information, perform metadata extraction and analysis operations on the classified data information according to preset metadata extraction rules and preset hash algorithms, and determine corresponding metadata;

[0280] The task service module 20 is used for the task service end, and is used to receive the metadata sent by the data processing end, perform task scheduling on the metadata, determine the corresponding task execution command, perform command optimization operation on the task execution command according to the set reinforcement learning algorithm, determine the optimal task execution command, and perform supervision operation on the execution of the optimal task execution command according to the set task supervision model to determine the corresponding task execution result, wherein the reinforcement learning algorithm is used to adjust the task execution order and the number of concurrency;

[0281] The result service module 30 is used for the result server, and is used to receive the task execution result sent by the task server, classify and grade the task execution result according to a preset classification hierarchy structure, determine the corresponding data classification directory, extract keywords from the data classification directory according to a preset keyword extraction algorithm, determine the corresponding data tag, perform data index construction operation according to the data classification directory and the data tag, determine the corresponding data index, and output the data index to the user end to complete data management.

[0282] From the above description, it can be seen that the data processing framework construction device provided in the embodiment of the present application can receive data source information through the data processing end, extract and classify the data source information, and determine the corresponding metadata; the task service end performs task scheduling on the metadata, determines the corresponding task execution command, performs command optimization operation on the task execution command according to the set reinforcement learning algorithm, determines the optimal task execution command, supervises the execution of the optimal task execution command according to the set task supervision model, and determines the corresponding task execution result; the result service end classifies and grades the task execution result according to the preset classification hierarchy structure, determines the corresponding data index, and outputs the data index to the user end to complete data management, thereby improving the efficiency and accuracy of data management.

[0283] From the hardware level, in order to improve the efficiency and accuracy of data management, the present application provides an embodiment of an electronic device for implementing all or part of the content in the data processing framework construction system, and the electronic device specifically includes the following content:

[0284] Processor, memory, communication interface and bus; wherein the processor, memory and communication interface communicate with each other through the bus; the communication interface is used to realize information transmission between the data processing framework construction system and the core business system, user terminal and related database and other related devices; the logic controller can be a desktop computer, a tablet computer and a mobile terminal, etc., and the present embodiment is not limited thereto. In the present embodiment, the logic controller can be implemented with reference to the embodiment of the data processing framework construction system and the embodiment of the data processing framework construction system in the embodiment, and the contents thereof are incorporated herein, and the repeated parts are not repeated.

[0285] It is understandable that the user terminal may include a smart phone, a tablet electronic device, a network set-top box, a portable computer, a desktop computer, a personal digital assistant (PDA), a vehicle-mounted device, a smart wearable device, etc. Among them, the smart wearable device may include smart glasses, a smart watch, a smart bracelet, etc.

[0286] In practical applications, part of the data processing framework construction system can be executed on the electronic device side as described above, or all operations can be completed in the client device. The specific selection can be based on the processing capability of the client device and the limitations of the user's usage scenario. This application does not limit this. If all operations are completed in the client device, the client device may also include a processor.

[0287] Fig.10FIG. 9 is a schematic block diagram of the system structure of the electronic device 9600 according to an embodiment of the present application. Fig.10 As shown, the electronic device 9600 may include a central processor 9100 and a memory 9140; the memory 9140 is coupled to the central processor 9100. It is worth noting that Fig.10 is exemplary; other types of structures may also be used to supplement or replace this structure to implement telecommunication functions or other functions.

[0288] In one embodiment, the data processing framework construction system function may be integrated into the central processing unit 9100. The central processing unit 9100 may be configured to implement steps S101 to S103.

[0289] In another embodiment, the data processing framework construction system can be configured separately from the central processing unit 9100. For example, the data processing framework construction system can be configured as a chip connected to the central processing unit 9100, and the functions of the data processing framework construction system are realized through the control of the central processing unit.

[0290] like Fig.10 As shown, the electronic device 9600 may also include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is worth noting that the electronic device 9600 does not necessarily have to include Fig.10 In addition, the electronic device 9600 may also include Fig.10 For components not shown, reference may be made to the prior art.

[0291] like Fig.10 As shown, the central processing unit 9100 is sometimes also referred to as a controller or an operation control, and may include a microprocessor or other processor device and / or logic device. The central processing unit 9100 receives input and controls the operation of various components of the electronic device 9600.

[0292] The memory 9140 may be, for example, one or more of a cache, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory or other suitable devices. The above-mentioned information related to the failure may be stored, and a program for executing the relevant information may also be stored. The CPU 9100 may execute the program stored in the memory 9140 to implement information storage or processing, etc.

[0293] The input unit 9120 provides input to the central processing unit 9100. The input unit 9120 is, for example, a key or a touch input device. The power supply 9170 is used to provide power to the electronic device 9600. The display 9160 is used to display display objects such as images and texts. The display may be, for example, an LCD display, but is not limited thereto.

[0294] The memory 9140 may be a solid-state memory, such as a read-only memory (ROM), a random access memory (RAM), a SIM card, etc. It may also be a memory that saves information even when the power is off, can be selectively erased, and is provided with more data, examples of which are sometimes referred to as EPROMs, etc. The memory 9140 may also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application / function storage unit 9142, which is used to store application programs and function programs or processes for executing the operation of the electronic device 9600 through the central processor 9100.

[0295] The memory 9140 may also include a data storage unit 9143 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 may include various drivers for communication functions of the electronic device and / or for executing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0296] The communication module 9110 is a transmitter / receiver that sends and receives signals via the antenna 9111. The communication module 9110 is coupled to the central processor 9100 to provide input signals and receive output signals, which may be the same as the case of a conventional mobile communication terminal.

[0297] Based on different communication technologies, multiple communication modules 9110 may be provided in the same electronic device, such as a cellular network module, a Bluetooth module and / or a wireless local area network module, etc. The communication module 9110 is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide an audio output via the speaker 9131 and receive an audio input from the microphone 9132, thereby realizing a common telecommunication function. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. In addition, the audio processor 9130 is also coupled to the central processor 9100, so that recording can be performed on the machine through the microphone 9132, and the sound stored on the machine can be played through the speaker 9131.

[0298] An embodiment of the present application also provides a computer-readable storage medium capable of implementing all the steps in the data processing framework construction system in the above-mentioned embodiments in which the execution subject is a server or a client. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements all the steps in the data processing framework construction system in the above-mentioned embodiments in which the execution subject is a server or a client.

[0299] The embodiments of the present application also provide a computer program product that can implement all the steps in the data processing framework construction system in the above embodiments with the execution subject being a server or a client. When the computer program / instructions are executed by a processor, the steps of the data processing framework construction system are implemented.

[0300] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A data processing framework construction system, characterized in that: It includes data processing, task server and result server, including: A data processing end is used to receive data source information, perform extraction and classification operations on the data source information, determine corresponding classified data information, perform metadata extraction and analysis operations on the classified data information according to preset metadata extraction rules and preset hash algorithms, and determine corresponding metadata; A task server, for receiving the metadata sent by the data processing end, performing task scheduling on the metadata, determining corresponding task execution commands, performing command optimization operations on the task execution commands according to a set reinforcement learning algorithm, determining optimal task execution commands, and performing supervision operations on the execution of the optimal task execution commands according to a set task supervision model, and determining corresponding task execution results, wherein the reinforcement learning algorithm is used to adjust the task execution order and the number of concurrency; The result server is used to receive the task execution result sent by the task server, classify and grade the task execution result according to a preset classification hierarchy structure, determine the corresponding data classification directory, extract keywords from the data classification directory according to a preset keyword extraction algorithm, determine the corresponding data tag, perform data index construction operation according to the data classification directory and the data tag, determine the corresponding data index, and output the data index to the user end to complete data management.

2. The data processing framework construction system according to claim 1, characterized in that: The extracting and classifying operations on the data source information to determine corresponding classified data information include: According to a preset scanning rule, a scanning operation is performed on the data source information to determine corresponding available data source information; The available data source information is classified according to a preset K-Means algorithm to determine corresponding classified data information, wherein the classification operation is used to identify the data source type, and the classified data information includes structured data and unstructured data.

3. The data processing framework construction system according to claim 1, characterized in that: The performing metadata extraction and analysis operations on the classified data information according to the preset metadata extraction rules and the preset hash algorithm to determine the corresponding metadata includes: Performing a metadata extraction operation according to a preset metadata extraction rule adapted to the classified data information to obtain first metadata, and performing a hash analysis operation on the first metadata according to a preset hash algorithm to determine a corresponding first hash value; The first Hash value is compared with a preset historical Hash value, and if there is a change, a self-increment operation is performed on the first metadata to determine the corresponding metadata.

4. The data processing framework construction system according to claim 1, characterized in that: The step of performing task arrangement on the metadata and determining corresponding task execution commands includes: Performing a data node setting operation on the metadata according to a preset directed acyclic graph to determine corresponding dependency relationships; The task scheduling operation is performed according to the dependency relationship and the preset scheduler to determine the corresponding task execution command.

5. The data processing framework construction system according to claim 1, characterized in that: Before performing a command optimization operation on the task execution command according to the set reinforcement learning algorithm to determine the optimal task execution command, the method includes: Collect historical task execution data, and determine a corresponding reinforcement learning environment based on the historical execution data; The reinforcement learning environment is trained for command execution according to a preset Q-learning algorithm to determine a corresponding set reinforcement learning algorithm.

6. The data processing framework construction system according to claim 5, characterized in that: The step of performing command execution training on the reinforcement learning environment according to the preset Q-learning algorithm to determine the corresponding set reinforcement learning algorithm includes: Performing command execution training on the reinforcement learning environment according to a preset Q-learning algorithm, determining a corresponding training result, and evaluating the training result according to a preset reward function to determine a corresponding Q value; The parameters of the preset Q-learning algorithm are updated according to the optimal Q value to determine the corresponding set reinforcement learning algorithm.

7. The data processing framework construction system according to claim 1, characterized in that: Before the execution of the optimal task execution command is supervised according to the set task supervision model to determine the corresponding task execution result, the method includes: Collect historical task execution result data, perform model training on a preset random forest model according to the historical task execution result data, and determine corresponding training results; The loss function is determined according to the training results and the preset validation set, and the parameters of the preset random forest model are updated according to the back propagation algorithm using the minimum loss function to determine the corresponding set task supervision model.

8. The data processing framework construction system according to claim 1, characterized in that: The step of supervising the execution of the optimal task execution command according to the set task supervision model to determine the corresponding task execution result includes: Performing a supervision operation on the execution of the optimal task execution command according to a set task supervision model to determine a corresponding prediction result, wherein the prediction result is success or failure; If the prediction result is failure, a parameter adjustment operation is performed on the optimal execution command according to the task supervision model, and a task execution operation is performed according to the parameters after the parameter adjustment operation to determine a corresponding task execution result.

9. A data processing framework construction device, characterized in that: The device comprises: A data processing module is used to receive data source information according to the data processing end, extract and classify the data source information, determine corresponding classified data information, perform metadata extraction and analysis operations on the classified data information according to preset metadata extraction rules and preset hash algorithms, and determine corresponding metadata; A task service module, configured to receive the metadata from a task service end, perform task scheduling on the metadata, determine corresponding task execution commands, perform command optimization operations on the task execution commands according to a set reinforcement learning algorithm, determine optimal task execution commands, perform supervision operations on the execution of the optimal task execution commands according to a set task supervision model, and determine corresponding task execution results, wherein the reinforcement learning algorithm is used to adjust the task execution order and the number of concurrency; The result service module is used to receive the task execution result from the result service end, classify and grade the task execution result according to a preset classification hierarchy structure, determine the corresponding data classification directory, extract keywords from the data classification directory according to a preset keyword extraction algorithm, determine the corresponding data tag, perform data index construction operations according to the data classification directory and the data tag, determine the corresponding data index, and output the data index to the user end to complete data management.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the data processing framework construction system according to any one of claims 1 to 7 are implemented.

Images