A big data management system and method based on distributed architecture

Abstract

The present invention provides a big data management system based on a distributed architecture, including a data acquisition component, a target setting component, a data processing component and a database. Data processing targets and data processing trigger conditions are set up, data processing tasks are generated, a task table is constructed for each data processing task, and the priority of the data processing tasks is periodically updated; the delay and energy consumption of different data processing tasks at different task execution nodes are calculated to obtain the processing cost, the optimal task execution node is selected to execute the corresponding data processing task, and the data processing sub-line is monitored. The present invention solves the problems in the prior art of low data management efficiency, slow data response speed, serious data redundancy due to huge data volume, and easy server congestion when facing high concurrent data, affecting the user experience.

Description

Technical Field

[0001] The present invention relates to the fields of big data and computers, and in particular to a big data management system and method based on a distributed architecture. Background Art

[0002] Big data is a data entity composed of data that is distributed and stored in the disk space of each single node in the cluster and can be processed in a distributed manner. Due to the huge amount and complex structure of big data, extracting valuable information from big data and achieving efficient management and fast indexing of big data requires reliable big data architecture support.

[0003] my country's patent application number: CN201810932452, publication date: 2019.02.12, discloses a big data information management system, the system includes: a big data collection unit, used to collect website big data; a content filtering module, used to perform content review on website big data; a cloud server, used to store website big data that has passed the review; an information push platform, used to call the website big data stored in the cloud server to push media content; a terminal, used to receive media content pushed by the information publishing platform.

[0004] However, in the process of implementing the technical solution of the invention in the embodiments of the present application, the inventors of the present application found that the above-mentioned technology has at least the following technical problems: the existing big data management systems and methods have low data management efficiency and slow data response speed. The huge amount of data leads to serious data redundancy, and when facing high-concurrency data, it is easy to cause server congestion, affecting the user experience. Summary of the invention

[0005] The present invention provides a big data management system and method based on a distributed architecture, which solves the problems of low data management efficiency and slow data response speed in existing big data management systems and methods, serious data redundancy due to huge data volume, and easy server congestion when facing high concurrent data, affecting the user experience. The invention solves the problems of low data management efficiency and slow response speed in traditional distributed architecture big data management systems.

[0006] The present invention provides a big data management system and method based on a distributed architecture, which specifically includes the following technical solutions:

[0007] A big data management system based on distributed architecture includes the following parts:

[0008] Data collection components, target setting components, data processing components and databases;

[0009] The data collection component is used to collect relevant data and store it in the database, and the data collection component is connected to the database by means of data transmission;

[0010] The target establishment component is used to establish a data processing target and send it to the data processing component. The target establishment component is connected to the data processing component by means of data transmission;

[0011] The data processing component includes a task generation unit, a rule setting unit, a task allocation unit, a task execution unit, a task monitoring unit and a thread management unit; the task allocation unit constructs a task table for each data processing task, updates the priority of each periodic data processing task, calculates the delay and energy consumption of different data processing tasks at different task execution nodes, obtains the processing cost of the data processing task at different task execution nodes, and screens out the task execution node that is executed first among the task execution nodes that meet the processing cost threshold, and sends it to the task execution unit; the task execution unit allocates the data processing task to the corresponding task execution node for data processing, obtains the execution status and execution result, and realizes the distributed call support for the data processing task; the task monitoring unit passes the data processing task or the data processing trigger condition into the data processing thread, and obtains the real-time progress of data processing according to the current data processing task or the data processing trigger condition; the thread management unit is used to match the thread performance parameters, determine the data processing sub-thread that meets the performance requirements after matching the thread performance parameters from the first set thread, capture the qualified data processing sub-thread, interrupt the unqualified data processing sub-program in time, and notify the task allocation unit, and the thread management unit sends the real-time progress of data processing to the task monitoring unit.

[0012] The units in the data processing component are connected by data transmission, and the data processing component stores the generated data into the database by data transmission;

[0013] The database is used to store big data and data generated in the data management process, and send the big data to the data processing component through data transmission.

[0014] A big data management method based on a distributed architecture comprises the following steps:

[0015] S1. Establish data processing goals and data processing trigger conditions, generate data processing tasks, build a task table for each data processing task, and periodically update the priority of data processing tasks;

[0016] S2. Calculate the delay and energy consumption of different data processing tasks at different task execution nodes, obtain the processing cost, select the optimal task execution node to execute the corresponding data processing task, and monitor the data processing sub-line.

[0017] Further, the step S1 comprises:

[0018] The task allocation unit constructs a task table for each data processing task and updates the priority of the data processing task in each cycle.

[0019] Further, the step S2 comprises:

[0020] The task allocation unit calculates the delay and energy consumption of different data processing tasks at different task execution nodes, obtains the processing cost of the data processing tasks at different task execution nodes, and selects the task execution node that is executed first among the task execution nodes that meet the processing cost threshold.

[0021] Further, the step S2 comprises:

[0022] The thread management unit matches thread performance parameters, determines a data processing sub-thread that meets the performance requirements after matching thread performance parameters from a first set thread, captures qualified data processing sub-threads, and promptly interrupts unqualified data processing sub-routines.

[0023] The present invention has at least the following technical effects or advantages:

[0024] 1. The task allocation unit builds a task table for each data processing task, and uniformly processes data through the task mechanism to adapt to a variety of data processing businesses. In addition, through the construction and management of the task table, the data response speed can be effectively improved, and the priority of the data processing task can be updated to improve the update efficiency. At the same time, it also avoids the impact of frequent updates on the performance of the task execution node and reduces data redundancy;

[0025] 2. Calculate the delay and energy consumption of different data processing tasks at different task execution nodes to obtain the processing cost and assign appropriate task execution nodes to the data processing tasks. When facing high-concurrency big data, it can effectively handle the congestion caused by server cache reconstruction, thereby improving the high availability of the big data system and the effectiveness of data management. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a structural diagram of a big data management system based on a distributed architecture according to the present invention;

[0027] Figure 2 This is a flow chart of a big data management method based on a distributed architecture described in the present invention. DETAILED DESCRIPTION

[0028] The embodiments of the present application provide a big data management system and method based on a distributed architecture, thereby solving the problems in the prior art of low data management efficiency, slow data response speed, serious data redundancy due to huge data volume, and easy server congestion when facing high-concurrency data, which affects the user experience.

[0029] The technical solution in the embodiment of the present application is to solve the above problems, and the overall idea is as follows:

[0030] The task allocation unit constructs a task table for each data processing task, and uniformly processes data through the task mechanism to adapt to a variety of data processing businesses. Moreover, through the construction and management of the task table, it can effectively improve the data response speed, and update the priority of the data processing task, thereby improving the update efficiency. At the same time, it also avoids the impact of frequent updates on the performance of the task execution node and reduces data redundancy. It calculates the delay and energy consumption of different data processing tasks at different task execution nodes to obtain the processing cost, and allocates appropriate task execution nodes to the data processing tasks. When facing high-concurrency big data, it can effectively handle the congestion caused by the server during cache reconstruction, thereby improving the high availability of the big data system and the effectiveness of data management.

[0031] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0032] See attached Figure 1 The big data management system based on distributed architecture described in the present invention includes the following parts:

[0033] A data collection component 10 , a target setting component 20 , a data processing component 30 and a database 40 .

[0034] The data acquisition component 10 is used to collect relevant data and store it in the database 40. The data acquisition component 10 is connected to the database 40 by means of data transmission;

[0035] The target setting component 20 is used to set up a data processing target and send it to the data processing component 30. The target setting component 20 is connected to the data processing component 30 by means of data transmission;

[0036] The data processing component 30 includes a task generation unit 301, a rule setting unit 302, a task allocation unit 303, a task execution unit 304, a task monitoring unit 305 and a thread management unit 306. The task generation unit 301 is used to take a data processing trigger condition or a combination of multiple data processing trigger conditions as a data processing task, and transmit the data processing task to the rule setting unit 302; the rule setting unit 302 is used to obtain the attribute information of the data processing task, and add the required built-in functional components or algorithm operators to the data processing task according to the attribute information and preset functional rules. The rule setting unit 302 transmits the processed data processing task to the task allocation unit 303 and the task execution unit 304; the task allocation unit 303 constructs a task table for each data processing task, updates the priority of the data processing task in each cycle, calculates the delay and energy consumption of different data processing tasks at different task execution nodes, and obtains the data processing task. The processing costs of different task execution nodes are screened out, and the first task execution node that is executed among the task execution nodes that meet the processing cost threshold is sent to the task execution unit 304; the task execution unit 304 assigns the data processing task to the corresponding task execution node for data processing, obtains the execution status and execution results, and realizes the distributed call support for the data processing task; the task monitoring unit 305 passes the data processing task or data processing trigger condition into the data processing thread, and obtains the real-time progress of data processing according to the current data processing task or data processing trigger condition; the thread management unit 306 is used to match the thread performance parameters, and after matching the thread performance parameters from the first set thread, determines the data processing sub-thread that meets the performance requirements, captures qualified data processing sub-threads, promptly interrupts unqualified data processing subroutines, and notifies the task allocation unit 303, and the thread management unit 306 sends the real-time progress of data processing to the task monitoring unit 305. The units in the data processing component 30 are connected by data transmission, and the data processing component 30 stores the generated data in the database 40 by data transmission;

[0037] The database 40 is used to store big data and data generated in the data management process, and send the big data to the data processing component 30 through data transmission.

[0038] See attached Figure 2 The big data management method based on distributed architecture described in the present invention comprises the following steps:

[0039] S1. Establish data processing goals and data processing trigger conditions, generate data processing tasks, build a task table for each data processing task, and periodically update the priority of data processing tasks;

[0040] S11. The data acquisition component 10 collects data and stores it in the database 40. The target setting component 20 sets a data processing target and sends it to the data processing component 30. The data processing component 30 processes the data according to the data processing target. The data processing target includes at least one data processing trigger condition, and the data processing trigger condition triggers the task execution unit 304 to process the data. The data processing trigger condition includes data cleaning, data integration, data reduction, data transformation, feature extraction, cluster analysis, data mining and other data processing methods.

[0041] As a specific embodiment, feature extraction is used as a data processing trigger condition. When the task information contains the current data processing trigger condition, data features are extracted after data acquisition. As a specific embodiment, cluster analysis is used as a data processing trigger condition. When the task information contains the current data processing trigger condition, cluster centers are found, cluster analysis is performed, and cluster results are output after data acquisition.

[0042] S12. The task generation unit 301 takes a data processing trigger condition or a combination of multiple data processing trigger conditions as a data processing task, and the rule setting unit 302 obtains the attribute information of the data processing task, and adds the required built-in functional components or algorithm operators to the data processing task according to the attribute information and the preset functional rules. The attribute information of the data processing task refers to the functional attributes of the data processing trigger conditions, that is, the data processing methods corresponding to different data processing trigger conditions; the preset functional rules are the specific algorithm contents for different data processing methods.

[0043] S13. The task allocation unit 303 constructs a task table for each data processing task, and the expression of the task table is: in, represents the priority of the i-th data processing task in the u-th cycle, v ij It represents the transmission rate ratio of the i-th data processing task to the j-th task execution node. and They represent the maximum transmission rate and minimum transmission rate of the i-th data processing task to all task execution nodes, and b ij represents the server bandwidth ratio of the jth task execution node allocated to the ith data processing task, β ij represents the resource ratio of the jth task execution node allocated to the ith data processing task, d i Indicates the size of the task. represents the maximum allowable delay of the i-th data processing task.

[0044] The priority of the data processing task is updated as the cycle changes. The specific calculation formula of the priority is:

[0045]

[0046] Among them, θ represents the priority coefficient, which is used to determine the proportion of the priority of the current cycle data processing task and the priority of the historical cycle data processing task, δ represents the correlation coefficient, count i represents the number of times the i-th data processing task is created in the historical cycle, t represents the current time, t0 represents the creation time of the data processing task, suc i Indicates the number of times the i-th data processing task is successfully executed in the historical cycle, Indicates the priority of the i-th data processing task in the previous cycle. If the i-th data processing task is created in the current cycle, this item is 0.

[0047] The beneficial effects of step S1 are as follows: the task allocation unit constructs a task table for each data processing task, performs data processing in a unified manner through a task mechanism, and adapts to a variety of data processing services. Moreover, through the construction and management of the task table, the data response speed can be effectively improved, and the priority of the data processing task can be updated, thereby improving the update efficiency. At the same time, it also avoids the impact of frequent updates on the performance of the task execution node and reduces data redundancy.

[0048] S2. Calculate the delay and energy consumption of different data processing tasks on different task execution nodes, obtain the processing cost, select the optimal task execution node to execute the corresponding data processing task, monitor the data processing sub-threads, and ensure the effectiveness of data processing.

[0049] S21. The task allocation unit 303 calculates the delay and energy consumption of different data processing tasks at different task execution nodes. The specific calculation formulas of the delay and energy consumption are as follows:

[0050]

[0051] E ij =p ij ×T ij

[0052] Among them, T ij represents the delay of the i-th data processing task at the j-th task execution node, Z represents the number of CPU cycles required to process unit data, and They represent the uplink rate and downlink rate of the i-th data processing task transmitted to the j-th task execution node, respectively, and f j represents the data processing capability of the jth task execution node, Indicates the amount of data returned by the jth task execution node as a result of the task execution. ijrepresents the energy consumption of the i-th data processing task at the j-th task execution node, and pij represents the power of the j-th task execution node in processing the i-th data processing task.

[0053] The task allocation unit 303 calculates the processing cost of the data processing task at different task execution nodes according to the delay and energy consumption of the data processing task at different task execution nodes:

[0054] C ij =αT ij +(1-α)E ij

[0055] Among them, α and 1-α represent the weights of time cost and energy cost respectively. The task allocation unit 303 selects the task execution nodes that meet the preset processing cost threshold, and determines the execution order of the data processing tasks in the selected task execution nodes according to the current priority of the data processing tasks, obtains the first task execution node that meets the processing cost threshold, and sends the finally selected task execution node to the task execution unit 304.

[0056] S22. The task execution unit 304 assigns the data processing task to the corresponding task execution node for data processing, and obtains the execution status and execution result. The task execution node includes a processor, a server and other devices with data processing capabilities.

[0057] During the data processing process performed by the task execution unit 304, the task monitoring unit 305 can pass the data processing task or data processing trigger condition into the data processing thread to obtain the real-time progress of data processing according to the current data processing task or data processing trigger condition. The real-time progress of data processing represents the data processing result obtained according to the current data processing task or data processing trigger condition. The data processing thread is a set of data processing sub-threads that meet the performance requirements determined by the first setting thread after matching the thread performance parameters, and the thread management unit 306 matches the thread performance parameters; the first setting thread includes at least one data processing sub-thread; the thread performance parameters are set according to the actual situation.

[0058] The thread management unit 306 captures qualified data processing sub-threads from the first set thread, wherein the qualified data processing sub-threads refer to data processing sub-threads whose quantitative comparison result between the data volume of the big data processed by the current data processing sub-thread and the data volume obtained after processing meets the set requirements. The unqualified data processing sub-threads are interrupted in time, and the task allocation unit 303 is notified to reselect the task execution node for data processing.

[0059] The beneficial effects of step S2 are: calculating the delay and energy consumption of different data processing tasks at different task execution nodes to obtain the processing cost, allocating appropriate task execution nodes to the data processing tasks, and effectively handling the congestion caused by server cache reconstruction when facing high-concurrency big data, thereby improving the high availability of the big data system and the effectiveness of data management.

[0060] In summary, the big data management system and method based on distributed architecture described in the present invention are completed.

[0061] Effect research:

[0062] The technical solution of the present invention can effectively solve the problems of low data management efficiency and slow data response speed in existing big data management systems and methods. The huge amount of data leads to serious data redundancy, and when facing high concurrent data, it is easy to cause server congestion, affecting the user experience. In addition, the above system or method has undergone a series of effect surveys and has been verified to ultimately solve the problems of low data management efficiency and slow response speed in traditional distributed architecture big data management systems.

[0063] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0064] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0065] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0066] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A big data management system based on a distributed architecture, characterized in that: Includes the following sections: Data collection components, target setting components, data processing components and databases; The data collection component is used to collect relevant data and store it in the database, and the data collection component is connected to the database by means of data transmission; The target establishment component is used to establish a data processing target and send it to the data processing component. The target establishment component is connected to the data processing component by means of data transmission; The data processing component includes a task generation unit, a rule setting unit, a task allocation unit, a task execution unit, a task monitoring unit and a thread management unit; the task allocation unit constructs a task table for each data processing task, updates the priority of each periodic data processing task, calculates the delay and energy consumption of different data processing tasks at different task execution nodes, obtains the processing cost of the data processing task at different task execution nodes, and screens out the task execution node that is executed first among the task execution nodes that meet the processing cost threshold, and sends it to the task execution unit; the task execution unit allocates the data processing task to the corresponding task execution node for data processing, obtains the execution status and execution result, and realizes the distributed call support for the data processing task; the task monitoring unit passes the data processing task or the data processing trigger condition into the data processing thread, and obtains the real-time progress of data processing according to the current data processing task or the data processing trigger condition; the thread management unit is used to match the thread performance parameters, determine the data processing sub-thread that meets the performance requirements after matching the thread performance parameters from the first set thread, capture qualified data processing sub-threads, interrupt unqualified data processing subroutines in time, and notify the task allocation unit, and the thread management unit sends the real-time progress of data processing to the task monitoring unit; The units in the data processing component are connected by data transmission, and the data processing component stores the generated data into the database by data transmission; The database is used to store big data and data generated in the data management process, and send the big data to the data processing component through data transmission.

2. A big data management method based on a distributed architecture, applied to a big data management system based on a distributed architecture as claimed in claim 1, characterized in that: The following steps are involved: S1. Establish data processing targets and data processing trigger conditions, generate data processing tasks, and the task allocation unit constructs a task table for each data processing task, and periodically updates the priority of the data processing task. The specific calculation formula of the priority is: in, represents the priority of the i-th data processing task in the u-th cycle; θ represents the priority coefficient; δ represents the correlation coefficient; count i represents the number of times the i-th data processing task is created in the historical cycle; t represents the current time; t0 represents the creation time of the data processing task; suc i Indicates the number of times the i-th data processing task is successfully executed in the historical cycle; Indicates the priority of the i-th data processing task in the previous cycle; S2. The task allocation unit calculates the delay and energy consumption of different data processing tasks at different task execution nodes. The specific calculation formulas of the delay and energy consumption are as follows: AND ij =p ij ×T ij Among them, T ij represents the delay of the i-th data processing task at the j-th task execution node; v ij represents the transmission rate ratio of the i-th data processing task to the j-th task execution node; d i Indicates the size of the task; Z indicates the number of CPU cycles required to process unit data; and They represent the uplink rate and downlink rate of the i-th data processing task transmitted to the j-th task execution node respectively; f represents the data processing capacity of the j-th task execution node; Indicates the amount of data returned by the jth task execution node as a result of the task execution; E ij represents the energy consumption of the i-th data processing task at the j-th task execution node; p ij represents the power of the jth task execution node processing the ith data processing task; The processing cost of data processing tasks at different task execution nodes is calculated based on the delay and energy consumption of data processing tasks at different task execution nodes: C ij =αT ij +(1-α)E ij Among them, α and 1-α represent the weights of time cost and energy cost respectively. Select the optimal task execution node to execute the corresponding data processing task and monitor the data processing sub-line.

3. A big data management method based on a distributed architecture as claimed in claim 2, characterized in that: The step S2 comprises: The task allocation unit calculates the delay and energy consumption of different data processing tasks at different task execution nodes, obtains the processing cost of the data processing tasks at different task execution nodes, and selects the task execution node that is executed first among the task execution nodes that meet the processing cost threshold.

4. The big data management method based on distributed architecture as claimed in claim 2, characterized in that: The step S2 comprises: The thread management unit matches thread performance parameters, determines a data processing sub-thread that meets the performance requirements after matching thread performance parameters from a first set thread, captures qualified data processing sub-threads, and promptly interrupts unqualified data processing sub-routines.

Images