A method and system for intelligent optimization of data collection tasks in a big data system
By automatically analyzing the input and output characteristics and dependencies of the data acquisition task node, combining topological sorting and heuristic algorithms, the data acquisition task configuration is optimized, and the problems of manual orchestration and high computing complexity in the existing technology are solved, achieving high efficiency and intelligence data acquisition task is executed.
Patent Information
- Application Number
- CN202411217959.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-02
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-09-02
AI Technical Summary
The existing data acquisition technology relies on manual orchestration and scheduling, lacks automated intelligent analysis tools, resulting in low execution efficiency, high computational complexity of genetic algorithms, and increases hardware resource consumption and time cost in large-scale tasks and node environments, dynamic feedback adjustment strategy Not flexible enough.
By intelligently analyzing the input and output characteristics of task nodes, automatically identify complex dependencies between task nodes, re-optimize task configuration, combine topological sorting and heuristic algorithms to identify parallel opportunities, optimize task execution sequences, and reduce manual intervention and resource consumption.
It significantly improves the accuracy and efficiency of task recognition, reduces task configuration errors caused by manual errors, improves the execution efficiency of big data acquisition tasks, and meets the needs of efficient and intelligent data acquisition in modern big data environments.
Smart Images

Figure CN118733283B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of data processing, and in particular to a method and system for intelligently optimizing data collection tasks in a big data system. Background Art
[0002] Currently, the main challenges facing data collection include the wide distribution of data sources, the diversity of data types, and the high requirements for real-time and efficiency. Although there are a variety of data collection solutions and ETL (Extract, Transform, Load) tools on the market, these tools allow users to manually orchestrate collection tasks and complete data collection through scheduling execution. However, these tools usually require manual analysis and orchestration of dependencies between tasks, and lack automated intelligent analysis tools to identify opportunities for parallel execution, resulting in low execution efficiency.
[0003] Existing data collection technologies mainly rely on manual orchestration and scheduling tasks, which may include data extraction, transformation and loading. Although some ETL tools provide parallel execution functions to improve execution efficiency, these functions usually require users to have in-depth technical knowledge and manual configuration capabilities.
[0004] For example, a Chinese patent with authorization announcement number CN117272838B discloses a data collection optimization method for a government big data platform. A data collection optimization method for a government big data platform includes: obtaining the collection node Ci and the CPU computing power Qi corresponding to the collection node Ci; setting the collection task Tj and the collection task data volume Nj corresponding to the collection task Tj; based on the collection node Ci, the CPU computing power Qi, the collection task Tj and the collection task data volume Nj, the data collection task is optimized and simulated through an improved genetic algorithm, and a data collection plan is output. The invention simulates the problem of allocating collection tasks to collection nodes through an improved genetic algorithm, outputs the data collection plan with the highest collection efficiency, and performs social network data collection based on the data collection plan with the highest collection efficiency, so that data on social networks can be collected in a timely manner.
[0005] For example, a Chinese patent with authorization announcement number CN117573382B discloses a data collection task scheduling method and device, which includes: determining the fitness of each node to perform each collection task according to the node feature information and the task feature information; using a decision maker to determine a first task scheduling strategy for randomly assigning the M collection tasks to the N nodes, and determining a first task efficiency score of the first task scheduling strategy according to the task feature information; calculating a first global value of the first task scheduling strategy based on the first task efficiency score and the global value calculation function of the decision maker; performing multiple iterations and updates until the global value converges to obtain a target task scheduling strategy. This solution can more intelligently and efficiently select data sources and task allocation strategies to meet the diverse requirements of power grid operations, adapt to the complexity of large-scale data collection, adapt to dynamically changing data needs, and help improve resource utilization efficiency.
[0006] The defects and shortcomings of the above patents are: 1) Although genetic algorithms are efficient, they have high computational complexity, especially in large-scale tasks and node environments, which may increase hardware resource consumption and time costs. At the same time, the optimization feedback mechanism does not elaborate on the iterative update content. The dynamic feedback adjustment strategy of the genetic algorithm may not be flexible enough in real-time response to changes in tasks and node capabilities; 2) It relies on complex algorithms and calculations, and has high requirements for algorithm design and parameter optimization, which may increase the difficulty of implementation. At the same time, the process of iterating until the global value converges may require multiple attempts, affecting the initial deployment efficiency and response speed. In addition, although dynamic adaptation is emphasized, the adaptability of strategy adjustments in the event of extreme data fluctuations or the addition of new data sources remains to be verified. Summary of the invention
[0007] In view of the shortcomings of the prior art, the present invention proposes a method and system for intelligent optimization of data acquisition tasks in a big data system. By intelligently analyzing the input and output characteristics of task nodes, the complex dependencies between task nodes are automatically identified, and the task configuration is re-optimized on this basis. At the same time, parallel opportunities are intelligently explored. The core of the present invention is to reduce manual intervention, reduce resource consumption, and improve the overall processing speed by optimizing the task execution sequence.
[0008] To achieve the above object, the present invention provides the following technical solutions:
[0009] A method for intelligent optimization of data collection tasks in a big data system includes the following specific steps:
[0010] Get data collection tasks;
[0011] According to the acquired data collection tasks, analyzing and collecting the node set of each data collection task;
[0012] Analyzing characteristics of input data and output data of each node in the node set;
[0013] According to the characteristic analysis results of node input data and output data, the dependency relationship between nodes is analyzed and the dependency matrix between nodes is constructed;
[0014] According to the dependency matrix between the nodes, identifying optimizable node configuration information and potential parallel execution sets and sequential execution sets between the nodes;
[0015] According to the identified parallel execution set and sequential execution set, the configuration execution order of the acquisition tasks is reordered and optimized.
[0016] Specifically, the node set for analyzing and collecting each data collection task includes:
[0017] Set the data collection task set to T z , data collection task T T z , T = {t1, t2, ..., t i ,...,t n}, where n represents the number of nodes in the data collection task T, t i Represents the i-th node in the data collection task T.
[0018] Specifically, the analyzing the characteristics of the input data and output data of each node in the node set includes:
[0019] Set node t i The input data and output data sets are X and Y respectively, X={x1,x2,...,x p},Y={y1,y2,...,y q}, where x p Represents node t i The pth input data, y q Represents node t i The qth output data of ;
[0020] Extract nodes t separately i The characteristics of the input and output data sets, including mean, variance, skewness and kurtosis, node t i The mean, variance, skewness, and kurtosis of the input data set are , , and , node t i The mean, variance, skewness, and kurtosis of the output data set are , , and ;
[0021] Using the clustering model to cluster the input data and output data sets of node t i and extract the clustering features of the input data and output data sets of node t i ; and ;
[0022] According to the characteristics of the input data and output data sets of the i-th node, construct the feature vectors of the input data and output data sets of the i-th node. The specific formula is: F iX = [ m X , s X 2 , t X , k X , i X ] , F iY = [ m Y , s Y 2 , t Y , k Y , i Y ] , where F iX represents the feature vector of the input data set of the i-th node, and F iY represents the feature vector of the output data set of the i-th node.
[0023] Specifically, analyzing the dependency relationship between nodes and constructing the dependency matrix between nodes includes:
[0024] Calculating the similarity between the feature vector F iX of the input data set of the i-th node and the feature vector F jY of the output data set of the j-th node. The calculation formula is: , where S ij represents the similarity between the feature vector F iX of the input data set of the i-th node and the feature vector F jY of the output data set of the j-th node, F jY represents the feature vector of the output data set of the j-th node, represents the vector modulus function;
[0025] Setting the similarity threshold as S yz , and analyzing the dependency relationship between the i-th node and the j-th node. The specific formula is: , where D ij represents the dependency relationship between the i-th node and the j-th node. When D ij = 1, it means that the output of the j-th node must depend on the input of the i-th node. When D ij = 0, it means that the output of the j-th node does not depend on the input of the i-th node;
[0026] Constructing the dependency matrix between nodes. The specific formula is: D = [D ij , 0 < i ≤ n, 0 < j ≤ n, and i and j are integers.
[0027] Specifically, the identifying of optimizable node configuration information and potential parallel execution sets and sequential execution sets between nodes includes:
[0028] Set the parallel execution set to M, which consists of all task nodes without predecessor nodes, specifically expressed as: M={t j ∣∀t i ∈T, j≠i, D ij =0}, ∀ means existence, ∈ means belonging to, t j represents the jth node in the data collection task;
[0029] Set the forward execution set to N, which consists of nodes with execution dependencies, specifically expressed as: N={t j ∣∀t i ∈T, j≠i, D ij =1}.
[0030] Specifically, the reordering and optimizing the configuration execution order of the acquisition tasks includes:
[0031] According to the identified parallel execution set M and sequential execution set N, the configuration execution order of the collection tasks is reordered and optimized. The specific formula is: f(T)={(M1,N1),(M2,N2),...,(M e ,N f )}, where f(T) represents the relationship between parallel execution and sequential execution between data acquisition task nodes, e and f are positive integers, and satisfy e+f=n, M e To execute the e-th task node in the set M in parallel, N f To sequentially execute the fth task node in the set N.
[0032] A big data system data collection task intelligent optimization system, used to implement a big data system data collection task intelligent optimization method, including: a task acquisition module, a task analysis module, a feature extraction module, a dependency analysis module, an optimization recognition module and a task optimization module;
[0033] The task acquisition module is used to acquire data collection tasks;
[0034] The task analysis module is used to analyze and collect the node set of each data collection task according to the acquired data collection task;
[0035] The feature extraction module is used to analyze the features of input data and output data of each node in the node set;
[0036] The dependency analysis module is used to analyze the dependency relationship between nodes and construct a dependency matrix between nodes based on the characteristic analysis results of the node input data and output data;
[0037] The optimization identification module is used to identify the node configuration information that can be optimized, and the potential parallel execution set and sequential execution set between the nodes according to the dependency matrix between the nodes;
[0038] The task optimization module is used to reorder and optimize the configuration execution order of the acquisition tasks according to the identified parallel execution set and sequential execution set.
[0039] Specifically, the dependency analysis module includes: a node dependency relationship analysis unit and a dependency matrix construction unit;
[0040] The node dependency analysis unit is used to calculate the similarity between the input data of the first node and the output data of the second node, and determine the dependency between the input data of the first node and the output data of the second node according to the similarity threshold;
[0041] The dependency matrix construction unit is used to construct a dependency matrix between nodes according to the analyzed dependency relationship between the input data of the first node and the output data of the second node.
[0042] Specifically, the optimization recognition module includes: a parallel execution recognition unit and a forward execution recognition unit;
[0043] The parallel execution identification unit is used to determine the nodes for parallel execution and construct a parallel execution set;
[0044] The forward execution identification unit is used to determine the forward execution nodes and construct a forward execution set.
[0045] An electronic device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the processor implements the steps of a method for intelligent optimization of data collection tasks in a big data system.
[0046] A computer-readable storage medium stores computer instructions, which, when executed, execute the steps of a method for intelligent optimization of data collection tasks in a big data system.
[0047] Compared with the prior art, the present invention has the following beneficial effects:
[0048] 1. The present invention proposes an intelligent optimization method for data collection tasks in a big data system. Through automated analysis of task node input and output characteristics, the accuracy and efficiency of task identification are significantly improved. This automated processing reduces dependence on human intervention, thereby reducing task configuration errors caused by human errors.
[0049] 2. The present invention proposes an intelligent optimization method for data collection tasks in a big data system, which combines topological sorting and heuristic algorithms to identify potential parallel opportunities. By making full use of these parallel opportunities, the execution efficiency of big data collection tasks can be significantly improved.
[0050] 3. The present invention proposes an intelligent optimization method for data collection tasks in a big data system, which meets the demand for efficient and intelligent data collection methods in a modern big data environment. BRIEF DESCRIPTION OF THE DRAWINGS
[0051] Figure 1 A flow chart of an intelligent optimization method for data collection tasks in a big data system provided by the present invention;
[0052] Figure 2 This is a diagram of the architecture of an intelligent optimization system for data collection tasks in a big data system provided by the present invention;
[0053] Figure 3 A schematic diagram of an electronic device provided by the present invention;
[0054] Figure 4 A schematic diagram of the structure of a computer-readable storage medium provided by the present invention. DETAILED DESCRIPTION
[0055] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, in the description of the present invention, it should be noted that the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside" and the like indicate the orientation or position relationship based on the orientation or position relationship shown in the accompanying drawings, which is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "No. 1", "No. 2" and "No. 3" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. The present invention is further explained below in conjunction with specific embodiments.
[0056] Example 1
[0057] See also Figure 1 , an embodiment provided by the present invention: a method for intelligent optimization of data collection tasks in a big data system, comprising the following specific steps:
[0058] Step S1: Obtain data collection tasks;
[0059] The input phase of the acquisition task involves receiving information about the data acquisition task configuration from the user or the automated system;
[0060] Step S2: Analyze and collect the node set of each data collection task according to the acquired data collection task;
[0061] Each task consists of multiple nodes, forming a node set: T={t1,t2,...,t i ,...,t n}, this formula defines the set of task nodes, where T represents the set of nodes in a data collection task, n represents the total number of nodes in the data collection task, and t i represents the i-th node in the data collection task, the i-th node t i Contains data information such as the node's expected receiving parameters, expected output results, and any specific processing requirements;
[0062] Step S3: analyzing the characteristics of input data and output data of each node in the node set;
[0063] The specific steps of step S3 include:
[0064] Step S301: Set node t i The input data and output data sets are X and Y respectively, X={x1,x2,...,x p},Y={y1,y2,...,y q}, where x p Represents node t i The pth input data, y q Represents node t i The qth output data of ;
[0065] Step S302: Extract nodes t respectively i The characteristics of the input data and output data set include: mean, variance, skewness and kurtosis, etc. The specific formula is:
[0066] , ,
[0067] in, Represents node t i The average value of the input data set, Represents node t i The jth input data of Represents node t i The average value of the output data set, Represents node t i The jth output data of
[0068] , ,
[0069] in, Represents node t i The variance of the input data set, Represents node t i The variance of the output data set,
[0070] , ,
[0071] in, Represents node t i The skewness of the input data set, Represents node t i The skewness of the output data set,
[0072] , ,
[0073] in, Represents node t i The kurtosis of the input data set, Represents node t i The kurtosis of the output data set;
[0074] Step S303: Use the clustering model to cluster nodes i The input data and output data sets are clustered to extract node t i Clustering features of the input and output data sets and ;
[0075] Step S304: construct the input data and output data set feature vector of the i-th node according to the features of the input data and output data set of the i-th node. The specific formula is:
[0076] F iX = [ m X , s X 2 , t X , k X , i X ] ,
[0077] F iY = [ m Y , s Y 2 , t Y , k Y , i Y ] , where F iX Represents the feature vector of the input data set of the i-th node, F iY The feature vector representing the output data set of the i-th node.
[0078] Step S4: Analyze the dependency relationship between nodes and construct a dependency matrix between nodes based on the feature analysis results of the node input data and output data;
[0079] The specific steps of step S4 are:
[0080] Step S401: Calculate the feature vector F of the i-th node input data set iXThe similarity with the feature vector F of the output data set of the j-th node jY is calculated by the formula:
[0081] ,
[0082] where S ij represents the feature vector F of the input data set of the i-th node iX and the similarity with the feature vector F of the output data set of the j-th node jY , F jY represents the feature vector of the output data set of the j-th node, represents the vector norm function;
[0083] Step S402: Set the similarity threshold to S yz , and analyze the dependency relationship between the i-th node and the j-th node. The specific formula is:
[0084] ,
[0085] where D ij represents the dependency relationship between the i-th node and the j-th node. When D ij = 1, it means that the output of the j-th node must depend on the input of the i-th node. When D ij = 0, it means that the output of the j-th node does not depend on the input of the i-th node;
[0086] Step S403: Construct the dependency matrix between nodes. The specific formula is: D = [D ij , 0 < i ≤ n, 0 < j ≤ n, and i and j are integers.
[0087] The dependency matrix D is an n×n matrix;
[0088] Step S5: According to the dependency matrix between the nodes, identify the node configuration information that can be optimized, as well as the potential parallel execution set and sequential execution set between the nodes;
[0089] The specific steps of Step S5 are:
[0090] Step S501: Set the parallel execution set to M. The parallel execution set M consists of all task nodes without predecessor nodes, and is specifically represented as: M = {t j ∣∀t i ∈ T, j ≠ i, D ij = 0}, ∀ represents existence, ∈ represents belonging, and t j represents the j-th node in the data acquisition task;
[0091] If the task node t jDoes not depend on any other node t in the node set T i (ie t j If there is no input parameter or the input parameter does not depend on the output parameter of other nodes in the set T), it can be executed in parallel with any node;
[0092] Step S502: Set the forward execution set to N. The forward execution set N consists of nodes with execution dependencies, specifically expressed as: N={t j ∣∀t i ∈T, j≠i, D ij =1};
[0093] If the task node t j Depends on t in the node set T i (ie t j The input parameter depends on t i t i and t j Must be executed in order, that is, must wait for t i After the node is executed, t j Node to execute.
[0094] Step S6: reordering the configuration execution order of the optimized acquisition tasks according to the identified parallel execution set and sequential execution set.
[0095] The specific steps of step S6 are:
[0096] According to the identified parallel execution set M and sequential execution set N, the configuration execution order of the collection tasks is reordered and optimized. The specific formula is: f(T)={(M1,N1),(M2,N2),...,(M e ,N f )}, where f(T) represents the relationship between parallel execution and sequential execution between data acquisition task nodes, e and f are positive integers, and satisfy e+f=n, M e To execute the e-th task node in the set M in parallel, N f For the fth task node in the sequential execution set N, the parallel execution task set M can start executing at the same time, while the sequential execution task set N needs to wait for its dependent tasks to complete before it can start executing.
[0097] Example 2
[0098] See also Figure 2 ,Another embodiment provided by the present invention: A big data system data acquisition task intelligent optimization system, comprising: a task acquisition module, a task analysis module, a feature extraction module, a dependency analysis module, an optimization identification module and a task optimization module;
[0099] The task acquisition module is used to acquire data collection tasks;
[0100] The task analysis module is used to analyze and collect the node set of each data collection task according to the acquired data collection task;
[0101] The feature extraction module is used to analyze the features of input data and output data of each node in the node set;
[0102] The dependency analysis module is used to analyze the dependency relationship between nodes and construct a dependency matrix between nodes based on the characteristic analysis results of the node input data and output data;
[0103] The optimization identification module is used to identify the node configuration information that can be optimized, and the potential parallel execution set and sequential execution set between the nodes according to the dependency matrix between the nodes;
[0104] The task optimization module is used to reorder and optimize the configuration execution order of the acquisition tasks according to the identified parallel execution set and sequential execution set.
[0105] The dependency analysis module includes: a node dependency relationship analysis unit and a dependency matrix construction unit;
[0106] The node dependency analysis unit is used to calculate the similarity between the input data of the first node and the output data of the second node, and determine the dependency between the input data of the first node and the output data of the second node according to the similarity threshold;
[0107] The dependency matrix construction unit is used to construct a dependency matrix between nodes according to the analyzed dependency relationship between the input data of the first node and the output data of the second node.
[0108] The optimization recognition module includes: a parallel execution recognition unit and a forward execution recognition unit;
[0109] The parallel execution identification unit is used to determine the nodes for parallel execution and construct a parallel execution set;
[0110] The forward execution identification unit is used to determine the forward execution nodes and construct a forward execution set.
[0111] Example 3
[0112] Figure 3 1 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Figure 3As shown, the present application also provides an electronic device 500. The electronic device 500 may include one or more processors and one or more memories. The memories store computer readable codes, and when the computer readable codes are run by one or more processors, a method for intelligent optimization of data collection tasks in a big data system may be executed.
[0113] The method or system according to the embodiment of the present application can also be used by Figure 3 The electronic device architecture shown in FIG. Figure 3 As shown, the electronic device 500 may include a bus 501, one or more CPUs 502, a ROM 503, a RAM 504, a communication port 505 connected to a network, an input / output component 506, a hard disk 507, and the like. The storage device in the electronic device 500, such as the ROM 503 or the hard disk 507, may store a method for intelligently optimizing data acquisition tasks of a big data system provided in the present application. A method for intelligently optimizing data acquisition tasks of a big data system may, for example, include: acquiring data acquisition tasks, analyzing and collecting node sets of each data acquisition task according to the acquired data acquisition tasks, analyzing the characteristics of the input data and output data of each node in the node set, analyzing the dependencies between nodes according to the characteristic analysis results of the node input data and output data, constructing a dependency matrix between nodes, identifying optimizable node configuration information and potential parallel execution sets and sequential execution sets between nodes according to the dependency matrix between nodes, and reordering the configuration execution order of the optimized acquisition tasks according to the identified parallel execution sets and sequential execution sets. Further, the electronic device 500 may also include a user interface 508. Of course, Figure 3 The architecture shown is only exemplary and can be omitted according to actual needs when implementing different devices. Figure 3 One or more components of an electronic device are shown.
[0114] Example 4
[0115] Figure 4 Schematic diagram of a computer-readable storage medium structure provided by an embodiment of the present application. Figure 4 As shown, the present application discloses a computer-readable storage medium 600. Computer-readable instructions are stored on the computer-readable storage medium 600. When the computer-readable instructions are executed by the processor, a method for intelligent optimization of data acquisition tasks in a big data system according to an embodiment of the present application described with reference to the above figures can be executed. The storage medium 600 includes, but is not limited to, for example, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and cache memory (cache). Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc.
[0116] It should be understood that the methods, apparatuses, and devices of the present application can be implemented in many ways. For example, the methods, apparatuses, and devices of the present application can be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is only for illustration, and the steps of the method of the present application are not limited to the order specifically described above unless otherwise specified. In addition, in some embodiments, the present application can also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present application. Therefore, the present application also covers a recording medium storing a program for executing the method according to the present application.
[0117] In addition, the parts of the above-mentioned technical solutions provided in the embodiments of the present application that are consistent with the implementation principles of the corresponding technical solutions in the prior art are not described in detail to avoid excessive redundancy.
[0118] The specific implementation modes as described above further describe the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation mode of the present invention and is not intended to limit the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A method for intelligent optimization of data collection tasks in a big data system, characterized in that: include: Get data collection tasks; According to the acquired data collection tasks, analyzing and collecting the node set of each data collection task; Analyzing characteristics of input data and output data of each node in the node set; According to the characteristic analysis results of node input data and output data, the dependency relationship between nodes is analyzed and the dependency matrix between nodes is constructed; According to the dependency matrix between the nodes, identifying optimizable node configuration information and potential parallel execution sets and sequential execution sets between the nodes; Reorder and optimize the configuration execution order of the acquisition tasks according to the identified parallel execution set and sequential execution set; The analyzing the characteristics of the input data and the output data of each node in the node set includes: Set node t i The input data and output data sets are X and Y respectively, X={x1,x2,...,x p },Y={y1,y2,...,y q }, where x p Represents node t i The pth input data, y q Represents node t i The qth output data of ; Extract nodes t separately i The characteristics of the input and output data sets, including mean, variance, skewness and kurtosis, node t i The mean, variance, skewness, and kurtosis of the input data set are , , and , node t i The mean, variance, skewness, and kurtosis of the output data set are , , and ; Using clustering model to classify node t i The input data and output data sets are clustered to extract node t i Clustering features of the input and output data sets and ; According to the characteristics of the input data and output data set of the i-th node, the input data and output data set feature vector of the i-th node is constructed. The specific formula is: , , where F iX Represents the feature vector of the input data set of the i-th node, F iY The feature vector representing the output data set of the i-th node; The analyzing the dependency relationship between nodes and constructing the dependency matrix between nodes includes: Calculate the feature vector F of the i-th node input data set iX and the output data set feature vector F of the jth node jY The similarity is calculated as: , where S ij The feature vector F representing the input data set of the i-th node iX and the output data set feature vector F of the jth node jY The similarity, F jY represents the output data set feature vector of the jth node, represents the vector modulo function; Set the similarity threshold to S yz , analyze the dependency relationship between the i-th node and the j-th node, the specific formula is: , where D ij represents the dependency relationship between the i-th node and the j-th node. When D ij =1, indicating that the output of the jth node must depend on the input of the ith node. ij =0, indicating that the output of the jth node does not depend on the input of the ith node; Construct a dependency matrix between nodes. The specific formula is: D = [D ij , where 0 < i ≤ n, 0 < j ≤ n, and i and j are integers.
2. The intelligent optimization method for data collection tasks of a big data system according to claim 1, characterized in that: The node set for analyzing and collecting each data collection task includes: Set the data collection task set to T z , data collection task T T z , T = {t1, t2, ..., t i ,...,t n }, where n represents the number of nodes in the data collection task T, t i Represents the i-th node in the data collection task T.
3. The intelligent optimization method for data collection tasks of a big data system according to claim 1, characterized in that: The identifying of optimizable node configuration information and potential parallel execution sets and sequential execution sets between nodes includes: Set the parallel execution set to M, which consists of all task nodes without predecessor nodes, specifically expressed as: M={t j ∣∀t i ∈T, j≠i, D ij =0}, ∀ means existence, ∈ means belonging to, t j represents the jth node in the data collection task; Set the forward execution set to N, which consists of nodes with execution dependencies, specifically expressed as: N={t j ∣∀t i ∈T, j≠i, D ij =1}.
4. The method for intelligent optimization of data collection tasks in a big data system according to claim 1, characterized in that: The reordering and optimizing the configuration execution order of the acquisition tasks includes: According to the identified parallel execution set M and sequential execution set N, the configuration execution order of the collection tasks is reordered and optimized. The specific formula is: f(T)={(M1,N1),(M2,N2),...,(M e ,N f )}, where f(T) represents the relationship between parallel execution and sequential execution between data acquisition task nodes, e and f are positive integers, and satisfy e+f=n, M e To execute the e-th task node in the set M in parallel, N f To sequentially execute the fth task node in the set N.
5. A big data system data collection task intelligent optimization system, used to implement a big data system data collection task intelligent optimization method according to any one of claims 1 to 4, characterized in that: include: Task acquisition module, task analysis module, feature extraction module, dependency analysis module, optimization identification module and task optimization module; The task acquisition module is used to acquire data collection tasks; The task analysis module is used to analyze and collect the node set of each data collection task according to the acquired data collection task; The feature extraction module is used to analyze the features of input data and output data of each node in the node set; The dependency analysis module is used to analyze the dependency relationship between nodes and construct a dependency matrix between nodes based on the characteristic analysis results of the node input data and output data; The optimization identification module is used to identify the node configuration information that can be optimized, and the potential parallel execution set and sequential execution set between the nodes according to the dependency matrix between the nodes; The task optimization module is used to reorder and optimize the configuration execution order of the acquisition tasks according to the identified parallel execution set and sequential execution set.
6. The intelligent optimization system for data collection tasks of a big data system according to claim 5, characterized in that: The dependency analysis module includes: a node dependency relationship analysis unit and a dependency matrix construction unit; The node dependency analysis unit is used to calculate the similarity between the input data of the first node and the output data of the second node, and determine the dependency between the input data of the first node and the output data of the second node according to the similarity threshold; The dependency matrix construction unit is used to construct a dependency matrix between nodes according to the analyzed dependency relationship between the input data of the first node and the output data of the second node.
7. The intelligent optimization system for data collection tasks of a big data system according to claim 6, characterized in that: The optimization recognition module includes: a parallel execution recognition unit and a forward execution recognition unit; The parallel execution identification unit is used to determine the nodes for parallel execution and construct a parallel execution set; The forward execution identification unit is used to determine the forward execution nodes and construct a forward execution set.
8. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of a method for intelligent optimization of data acquisition tasks in a big data system according to any one of claims 1 to 4 are implemented.
9. A computer-readable storage medium, characterized in that: Computer instructions are stored thereon, and when the computer instructions are executed, the steps of a method for intelligent optimization of data acquisition tasks in a big data system described in any one of claims 1-4 are executed.
