Data transmission system based on cloud storage

Abstract

The invention discloses a data transmission system based on cloud storage, which comprises a cloud storage module, a data fragmentation transmission module, a cloud node real-time load index calculation module and a task scheduling module, and relates to the technical field of data transmission, and the load index is calculated through a CPU utilization rate, a cloud node memory utilization rate, a network bandwidth occupancy rate and a disk IO utilization rate. The task scheduling module calculates the task quantity of fragmented data in the current cloud node queue and the current processing data quantity, the one-sidedness of single index evaluation is avoided, the task scheduling module combines the fragmented data quantity and the complexity score to calculate the processing cost, then the adaptation degree is determined through the real-time load index of the cloud node, and the transmission tasks are allocated through the adaptation degree; according to the method, the task load of fragmented data processed by each cloud node is matched with the resource capacity of the cloud node, the parallel processing efficiency and the resource utilization rate are maximized, and the transmission efficiency of cross-database transmission from the MySQL database to the MongoDB database is improved through dynamic adaptation of the transmission tasks and the cloud nodes.

Description

Technical Field

[0001] The present invention relates to the technical field of data transmission, and in particular to a data transmission system based on cloud storage. Background Art

[0002] With the deepening of enterprise digital transformation, the demand for cross-database data transmission and migration is increasing. Especially in scenarios such as hybrid cloud architecture and multi-database collaborative analysis, how to efficiently and reliably transfer data from the relational database MySQL to the non-relational database MongoDB has become a key technical issue.

[0003] The existing data is transferred from the relational database MySQL to the non-relational database MongoDB, which has the following problems:

[0004] Currently, MySQL data is fixedly sharded by table and row ranges, and allocated to fixed cloud nodes manually or through simple rules. This lacks dynamic perception of the real-time node load and data characteristics, and is prone to the phenomenon of "strong nodes being idle when processing simple tasks, and weak nodes being overloaded when processing complex tasks." This reduces resource utilization. In addition, the load of traditional cloud nodes only monitors a single indicator of CPU or memory, ignoring the impact of network bandwidth and disk IO, which reduces the accuracy of cloud node load calculations. Summary of the Invention

[0005] In order to solve the technical problems existing in the background technology, the present invention proposes a data transmission system based on cloud storage.

[0006] The present invention proposes a cloud storage-based data transmission system, comprising:

[0007] Cloud storage module: used to store data in MySQL and MongoDB databases;

[0008] Data sharding transmission module: shards MySQL data by table, row, or column to generate multiple shards, distributes the multiple shards to multiple cloud nodes, and multiple cloud nodes transmit the shards to the MongoDB database in parallel;

[0009] Real-time load perception module: used to collect resource status parameters of each cloud node in real time, and the obtained resource status parameters form a multi-dimensional load vector;

[0010] Resource status parameters include CPU utilization, cloud node memory utilization, network bandwidth utilization, disk IO utilization, as well as the number of tasks for sharded data in the cloud node queue and the current amount of data being processed;

[0011] Cloud node real-time load index calculation module: obtains the multi-dimensional load vector of the real-time load perception module and obtains the cloud node real-time load index through the multi-dimensional load vector;

[0012] Task scheduling module: calculates the complexity score of the sharded data, and then calculates the processing cost of the sharded task based on the complexity score and the data volume of the sharded data;

[0013] According to the real-time load index of each cloud node calculated by the cloud node real-time load index calculation module, the adaptability of the shard data and the cloud node is obtained by combining the real-time load index of each cloud node with the processing cost of the shard task, and the shard data is allocated to the cloud node with the largest adaptability.

[0014] Preferably, in the cloud node real-time load index calculation module:

[0015] Obtain the multi-dimensional load vector of the current cloud node in the real-time load perception module;

[0016] In the multi-dimensional load vector:

[0017] Assume that CPU utilization is F1, cloud node memory utilization is F2, network bandwidth utilization is F3, and disk IO utilization is F4;

[0018] Normalize the number of tasks of the shard data in the current cloud node queue and the current processing data size in the current cloud node queue to obtain the normalized value F5 of the number of tasks and the normalized value F6 of the current processing data size;

[0019] The calculation formula for the current cloud node load index is:

[0020] Load index=α·F1+β·F2+γ·F3+δ·F4+εF5·+η·F6;

[0021] Among them, α, β, γ, δ, ε, and η are the weights of F1, F2, F3, F4, F5, and F6, respectively.

[0022] Preferably, in the task scheduling module, the complexity score of the shard data is calculated as follows:

[0023] Calculate the table join depth, non-scalar field ratio, and index / constraint characteristics of each shard data through statistical analysis.

[0024] The logical complexity of data cleaning rules is converted into quantitative indicators through quantitative analysis to obtain the complexity characteristics of sharded data;

[0025] The complexity features of the sharded data are normalized, and then the entropy weight method is used to calculate the weight of each complexity feature, and the complexity score of the sharded data is obtained by weighted summation.

[0026] Preferably, in the task scheduling module, the load index is set to L j , the maximum amount of data that can be processed by the cloud node is C j , then the remaining processing capacity of the cloud node R j for:

[0027] R j =C j ·(1-L j );

[0028] Assume the amount of shard data is S i , the complexity score of the sharded data is K i , the processing cost of the shard task is T i , T i The calculation formula is:

[0029] T i =S i (1+K i );

[0030] Calculate the compatibility M of each shard data with any cloud node i，j :

[0031]

[0032] Among them, λ is the sensitivity coefficient.

[0033] Preferably, in the task scheduling module, all the shard data are sorted by T i Sort in descending order, press T i Calculate the compatibility M of the shard data with any cloud node in descending order i，j .

[0034] Preferably, in the data shard transmission module, if the data size of the shard data exceeds a set threshold, the portion of the shard data exceeding the threshold is split into sub-shards.

[0035] Preferably, it also includes:

[0036] Dynamic feedback adjustment module: When the cloud node load index exceeds the warning threshold, the task migration process is triggered. The task migration process is as follows:

[0037] Among the shard data not processed by the cloud node, take T i The shard data with the lowest value is dispatched to the cloud node with the lowest current load index.

[0038] A data transmission method based on cloud storage, comprising the following steps:

[0039] S1. Shard MySQL data by table, row, or column to generate multiple shards. Distribute the shards to multiple cloud nodes, which then transfer the shards to the MongoDB database in parallel.

[0040] S2 is used to collect resource status parameters of each cloud node in real time, and the obtained resource status parameters form a multi-dimensional load vector;

[0041] Resource status parameters include CPU utilization, cloud node memory utilization, network bandwidth utilization, disk IO utilization, as well as the number of tasks for sharded data in the cloud node queue and the current amount of data being processed;

[0042] S3, obtain the real-time load index of the cloud node through the multi-dimensional load vector obtained in S2;

[0043] S4. Obtain the complexity score of the sharded data through calculation, and calculate the processing cost of the sharded task by combining the complexity score with the data volume of the sharded data.

[0044] According to the real-time load index of each cloud node calculated by the cloud node real-time load index calculation module, the adaptability of the shard data and the cloud node is obtained by combining the real-time load index of each cloud node with the processing cost of the shard task, and the shard data is allocated to the cloud node with the largest adaptability.

[0045] The cloud storage-based data transmission system proposed in the present invention has the following beneficial technical effects:

[0046] By sharding MySQL data by table, row or column, multiple shard data are generated and distributed to multiple cloud nodes for parallel transmission, breaking through the traditional single-node processing bottleneck and improving the transmission efficiency of large-scale data transmission; the load index is calculated through CPU utilization, cloud node memory utilization, network bandwidth occupancy, disk IO utilization, as well as the number of shard data tasks in the current cloud node queue and the current processing data volume, avoiding the one-sidedness of single indicator evaluation. The task scheduling module calculates the processing cost based on the shard data volume and complexity score, and then determines the adaptability through the real-time load index of the cloud node. The transmission tasks are allocated based on the adaptability, so that the shard data task volume processed by each cloud node matches its own resource capabilities, achieving maximum parallel processing efficiency and resource utilization, and improving the transmission efficiency of cross-database transmission from MySQL database to MongoDB database through dynamic adaptation of transmission tasks and cloud nodes.

[0047] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a principle block diagram of the system of the present invention;

[0049] Figure 2 Flowchart of the method of the present invention. DETAILED DESCRIPTION

[0050] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar symbols throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention, and are not to be construed as limiting the present invention.

[0051] like Figure 1 The data transmission system based on cloud storage shown in the figure includes a cloud storage module, a data sharding transmission module, a cloud node real-time load index calculation module, and a task scheduling module:

[0052] Cloud storage module: used to store data in MySQL and MongoDB databases;

[0053] Data sharding transmission module: shards MySQL data by table, row, or column to generate multiple shards, distributes the multiple shards to multiple cloud nodes, and multiple cloud nodes transmit the shards to the MongoDB database in parallel;

[0054] Real-time load perception module: used to collect resource status parameters of each cloud node in real time, and the obtained resource status parameters form a multi-dimensional load vector;

[0055] Resource status parameters include CPU utilization, cloud node memory utilization, network bandwidth utilization, disk IO utilization, the number of tasks for sharded data in the current cloud node queue, and the current amount of processed data in the current cloud node queue;

[0056] The current processing data volume refers to the total volume of all sharded data in the current cloud node queue;

[0057] Cloud node real-time load index calculation module: obtains the multi-dimensional load vector of the real-time load perception module and obtains the cloud node real-time load index through the multi-dimensional load vector;

[0058] In an optional embodiment, in the cloud node real-time load index calculation module:

[0059] Obtain the multi-dimensional load vector of the current cloud node in the real-time load perception module;

[0060] In the multi-dimensional load vector:

[0061] Assume that CPU utilization is F1, cloud node memory utilization is F2, network bandwidth utilization is F3, and disk IO utilization is F4;

[0062] Normalize the number of tasks of the shard data in the current cloud node queue and the current processing data size in the current cloud node queue to obtain the normalized value F5 of the number of tasks and the normalized value F6 of the current processing data size;

[0063] The calculation formula for the current cloud node load index is:

[0064] Load index=α·F1+β·F2+γ·F3+δ·F4+εF5·+η·F6;

[0065] Among them, α, β, γ, δ, ε, and η are the weights of F1, F2, F3, F4, F5, and F6 respectively. The values of α, β, γ, δ, ε, and η are adaptively adjusted by the machine learning algorithm based on historical transmission data;

[0066] The load index is used to evaluate the current load pressure of the cloud node. The load index is calculated based on CPU utilization, cloud node memory utilization, network bandwidth utilization, disk IO utilization, the number of tasks for sharded data in the current cloud node queue, and the current amount of data being processed. This avoids the one-sidedness of single-indicator evaluation and improves the accuracy of negative cloud node load judgment.

[0067] Task scheduling module: calculates the complexity score of the sharded data, and then calculates the processing cost of the sharded task based on the complexity score and the data volume of the sharded data;

[0068] According to the real-time load index of each cloud node calculated by the cloud node real-time load index calculation module, the adaptability of the shard data and the cloud node is obtained by combining the real-time load index of each cloud node with the processing cost of the shard task, and the shard data is allocated to the cloud node with the largest adaptability.

[0069] By dynamically adjusting the sharded data processing tasks of each cloud node, load balancing is achieved, so that the amount of sharded tasks processed by each node matches its own resource capabilities, ultimately maximizing parallel processing efficiency and resource utilization.

[0070] The above content, by sharding MySQL data by table, row or column, generates multiple shard data and distributes them to multiple cloud nodes for parallel transmission, breaking through the traditional single-node processing bottleneck and improving the transmission efficiency during large-scale data transmission; the task scheduling module combines the shard data volume and complexity score to calculate the processing cost, and then determines the adaptability through the real-time load index of the cloud node. The transmission tasks are allocated based on the adaptability, so that the shard data task volume processed by each cloud node matches its own resource capabilities, achieving the maximization of parallel processing efficiency and resource utilization, and improving the transmission efficiency of cross-database transmission from MySQL database to MongoDB database through dynamic adaptation of transmission tasks and cloud nodes.

[0071] In an optional embodiment, in the task scheduling module, the complexity score of the shard data is calculated as follows:

[0072] Calculate the table join depth, non-scalar field ratio, and index / constraint characteristics of each shard data through statistical analysis.

[0073] The logical complexity of data cleaning rules is converted into quantitative indicators through quantitative analysis to obtain the complexity characteristics of sharded data;

[0074] The complexity features of the sharded data are normalized, and then the entropy weight method is used to calculate the weight of each complexity feature, and the complexity score of the sharded data is obtained by weighted summation.

[0075] Among them, the table association depth is obtained by counting the foreign key association levels;

[0076] The proportion of non-scalar fields is obtained by dividing the number of non-scalar fields by the total number of fields.

[0077] Index / constraint features are obtained by counting the index / constraint types and quantities and assigning scores;

[0078] The logical complexity of data cleaning rules includes the number of rules and nesting depth;

[0079] The goal is to calculate the shard data complexity score through statistical analysis and quantitative methods, and convert implicit processing costs such as table association depth, non-scalar field ratio, number of index constraints, and cleaning rule complexity into quantitative indicators. This avoids the problem of "complex tasks dragging down nodes" caused by traditional solutions that only allocate data based on data volume. The entropy weight method is used to dynamically calculate the complexity feature weights, and the evaluation model is automatically adjusted based on historical data, so that the complexity score is continuously optimized as the business scenario changes.

[0080] In an optional embodiment, in the task scheduling module, the load index is set to L j , the maximum amount of data that can be processed by the cloud node is C j , then the remaining processing capacity of the cloud node R j for:

[0081] R j =C j ·(1-L j );

[0082] Assume the amount of shard data is S i , the complexity score of the sharded data is K i , the processing cost of the shard task is T i , T i The calculation formula is:

[0083] T i =S i (1+K i );

[0084] Calculate the compatibility M of each shard data with any cloud node i，j :

[0085]

[0086] Among them, λ is the sensitivity coefficient.

[0087] By R j =C j ·(1-L j ) Quantify the remaining processing capacity of the cloud node combined with the shard processing cost T i =S i (1+K i ), due to T i ≤R j Ensure that the task allocation does not exceed the available resources of the node, avoid single cloud node memory overflow or CPU overload, improve the stability of operation, and use exp(-λ·L j ) Strengthen the priority of low-load nodes so that new tasks are allocated to idle cloud nodes first, reducing the task accumulation of high-load cloud nodes.

[0088] In an optional embodiment, in the task scheduling module, all the slice data are sorted by T i Sort in descending order, press T i Calculate the compatibility M of the shard data with any cloud node in descending order i，j ;

[0089] Then distribute the shard data to M i，j The largest cloud node.

[0090] By giving priority to tasks with the highest processing cost among the sharded tasks, tasks with lower processing cost among the sharded tasks are prevented from occupying high-quality cloud node resources. Because the optimal cloud nodes are allocated at the beginning, transmission efficiency is improved, which is especially suitable for mixed load scenarios.

[0091] In an optional embodiment, in the data shard transmission module, if the data size of the shard data exceeds a set threshold, the portion of the shard data exceeding the threshold is split into sub-shards, thereby reducing the occurrence of memory overflow in a single cloud node;

[0092] Dynamic feedback adjustment module: When the cloud node load index exceeds the warning threshold, the task migration process is triggered. The task migration process is as follows:

[0093] Among the shard data not processed by the cloud node, take T i The shard data with the lowest value is dispatched to the cloud node with the lowest current load index;

[0094] When the cloud node load index exceeds the warning threshold, take T i The shard data with the lowest value is dispatched to the cloud node with the lowest current load index, thereby quickly reducing the pressure on overloaded cloud nodes and preventing local bottlenecks from spreading to global performance degradation.

[0095] like Figure 2 A cloud storage-based data transmission method is shown, comprising the following steps:

[0096] S1. Shard MySQL data by table, row, or column to generate multiple shards. Distribute the shards to multiple cloud nodes, which then transfer the shards to the MongoDB database in parallel.

[0097] S2 is used to collect resource status parameters of each cloud node in real time, and the obtained resource status parameters form a multi-dimensional load vector;

[0098] Resource status parameters include CPU utilization, cloud node memory utilization, network bandwidth utilization, disk IO utilization, as well as the number of tasks for sharded data in the cloud node queue and the current amount of data being processed;

[0099] S3, obtain the real-time load index of the cloud node through the multi-dimensional load vector obtained in S2;

[0100] S4. Obtain the complexity score of the sharded data through calculation, and calculate the processing cost of the sharded task by combining the complexity score with the data volume of the sharded data.

[0101] According to the real-time load index of each cloud node calculated by the cloud node real-time load index calculation module, the adaptability of the shard data and the cloud node is obtained by combining the real-time load index of each cloud node with the processing cost of the shard task, and the shard data is allocated to the cloud node with the largest adaptability.

[0102] Meanwhile, the contents not described in detail in this specification belong to the prior art known to those skilled in the art.

[0103] In the embodiments provided by the present invention, it should be understood that the disclosed systems or methods can be implemented in other ways. For example, the embodiments of the invention described above are merely illustrative. For example, the division of modules is only a logical function division, and other division methods may be used in actual implementation.

[0104] Modules described as separate components may or may not be physically separate, and components shown as modules may or may not be physical modules, and may be located in one place or distributed across multiple network modules. Some or all of these modules may be selected to achieve the objectives of this embodiment based on actual needs.

[0105] In addition, the functional modules in various embodiments of the present invention may be integrated into a single processing module, each module may exist physically separately, or two or more modules may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or hardware plus software functional modules.

[0106] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the basic characteristics of the present invention.

[0107] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A data transmission system based on cloud storage, characterized in that: include: Cloud storage module: used to store data in MySQL and MongoDB databases; Data sharding transmission module: shards MySQL data by table, row, or column to generate multiple shards, distributes the multiple shards to multiple cloud nodes, and multiple cloud nodes transmit the shards to the MongoDB database in parallel; Resource status parameters include CPU utilization, cloud node memory utilization, network bandwidth utilization, disk IO utilization, as well as the number of tasks for sharded data in the cloud node queue and the current amount of data being processed; Cloud node real-time load index calculation module: obtains the multi-dimensional load vector of the real-time load perception module and obtains the cloud node real-time load index through the multi-dimensional load vector; Task scheduling module: calculates the complexity score of the sharded data, and then calculates the processing cost of the sharded task based on the complexity score and the data volume of the sharded data; According to the real-time load index of each cloud node calculated by the cloud node real-time load index calculation module, the adaptability of the shard data and the cloud node is obtained by combining the real-time load index of each cloud node with the processing cost of the shard task, and the shard data is allocated to the cloud node with the largest adaptability.

2. The cloud storage-based data transmission system according to claim 1, characterized in that: In the cloud node real-time load index calculation module: Obtain the multi-dimensional load vector of the current cloud node in the real-time load perception module; In the multi-dimensional load vector: Assume that CPU utilization is F1, cloud node memory utilization is F2, network bandwidth utilization is F3, and disk IO utilization is F4; Normalize the number of tasks of the shard data in the current cloud node queue and the current processing data size in the current cloud node queue to obtain the normalized value F5 of the number of tasks and the normalized value F6 of the current processing data size; The calculation formula for the current cloud node load index is: Load index=α·F1+β·F2+γ·F3+δ·F4+εF5·+η·F6; Among them, α, β, γ, δ, ε, and η are the weights of F1, F2, F3, F4, F5, and F6, respectively.

3. The cloud storage-based data transmission system according to claim 1, characterized in that: In the task scheduling module, the complexity score of sharded data is calculated as follows: Calculate the table join depth, non-scalar field ratio, and index / constraint characteristics of each shard data through statistical analysis. The logical complexity of data cleaning rules is converted into quantitative indicators through quantitative analysis to obtain the complexity characteristics of sharded data; The complexity features of the sharded data are normalized, and then the entropy weight method is used to calculate the weight of each complexity feature, and the complexity score of the sharded data is obtained by weighted summation.

4. The cloud storage-based data transmission system according to claim 3, characterized in that: In the task scheduling module, let the load index be L j , the maximum amount of data that can be processed by the cloud node is C j , then the remaining processing capacity of the cloud node R j for: R j =C j ·(1-L j ); Assume the amount of shard data is S i , the complexity score of the sharded data is K i , the processing cost of the shard task is T i , T i The calculation formula is: T i =S i ·(1+K i ); Calculate the compatibility M of each shard data with any cloud node i，j : Among them, λ is the sensitivity coefficient.

5. The cloud storage-based data transmission system according to claim 4, characterized in that: In the task scheduling module, press T to schedule all shard data. i Sort in descending order, press T i Calculate the compatibility M of the shard data with any cloud node in descending order i，j .

6. The cloud storage-based data transmission system according to claim 1, characterized in that: In the data sharding transmission module, if the data size of the sharded data exceeds the set threshold, the portion of the sharded data that exceeds the threshold is split into sub-shards.

7. The cloud storage-based data transmission system according to claim 1, characterized in that: Also includes: Dynamic feedback adjustment module: When the cloud node load index exceeds the warning threshold, the task migration process is triggered. The task migration process is as follows: Among the shard data not processed by the cloud node, take T i The shard data with the lowest value is dispatched to the cloud node with the lowest current load index.

8. The cloud storage-based data transmission method according to any one of claims 1 to 7, characterized in that: The following steps are involved: S1. Shard MySQL data by table, row, or column to generate multiple shards. Distribute the shards to multiple cloud nodes, which then transfer the shards to the MongoDB database in parallel. S2 is used to collect resource status parameters of each cloud node in real time, and the obtained resource status parameters form a multi-dimensional load vector; Resource status parameters include CPU utilization, cloud node memory utilization, network bandwidth utilization, disk IO utilization, as well as the number of tasks for sharded data in the cloud node queue and the current amount of data being processed; S3, obtain the real-time load index of the cloud node through the multi-dimensional load vector obtained in S2; S4. Obtain the complexity score of the sharded data through calculation, and calculate the processing cost of the sharded task by combining the complexity score with the data volume of the sharded data. According to the real-time load index of each cloud node calculated by the cloud node real-time load index calculation module, the adaptability of the shard data and the cloud node is obtained by combining the real-time load index of each cloud node with the processing cost of the shard task, and the shard data is allocated to the cloud node with the largest adaptability.

Images