Data transmission method and apparatus, storage medium, and electronic device

Abstract

The application discloses a data transmission method and device, a storage medium and an electronic device. It relates to the field of big data. The method comprises the following steps: determining the initial transmission duration of the to-be-transmitted data of a target system; determining the receiving port and the number of the target database cluster for receiving the to-be-transmitted data; determining the target data volume of the to-be-transmitted data, setting the initial received data volume of each receiving port, transmitting the to-be-transmitted data according to the initial received data volume, and recording the data transmission duration; in the case that the data transmission duration reaches a preset duration, obtaining the unreceived data volume of each receiving port, adjusting the unreceived data volume of each receiving port; continuing to transmit the remaining data in the to-be-transmitted data according to the adjusted data volume, updating the data transmission duration, and until there is no unreceived data in each receiving port. Through the application, the problem of low data import speed when importing data into a database system in the related art is solved.

Description

Technical Field

[0001] This application relates to the field of big data, and more specifically, to a data transmission method, apparatus, storage medium, and electronic device. Background Technology

[0002] In recent years, Clickhouse (a columnar database management system for online analytical processing) has become a commonly used data management system in the field of real-time analytical databases due to its superior analytical capabilities, excellent linear scalability, and rich functionality. In addition to its fundamental features such as columnar storage and distributed architecture, Clickhouse's vectorized execution engine and multi-threaded parallel processing significantly improve its query performance compared to traditional databases.

[0003] However, ClickHouse's advantage lies in real-time data querying, making it unsuitable for large-scale batch calculations and data updates. It requires importing already processed data from other databases before use. Currently, when importing data—for example, after offline processing of a wide table in HIVE and then importing the table data into ClickHouse—the large number of fields and data volume in the wide table leads to lengthy single-concurrency direct data import operations, impacting data usage during business operations and consequently affecting business processes.

[0004] There is currently no effective solution to the problem of slow data import speed when importing data into database systems in related technologies. Summary of the Invention

[0005] This application provides a data transmission method, apparatus, storage medium, and electronic device to solve the problem of low data import speed when importing data into a database system in related technologies.

[0006] According to one aspect of this application, a data transmission method is provided. The method includes: determining the data to be transmitted in the target system and determining the initial transmission duration of the data to be transmitted; determining the number of database nodes in the target database cluster, determining the number of database nodes required to transmit the data to be transmitted within the initial transmission duration, obtaining the first number of nodes, and determining the receiving ports for receiving the data to be transmitted based on the first number of nodes, obtaining P receiving ports, wherein each database node is configured with a receiving port, and the number of the first nodes is less than or equal to the number of database nodes in the target database cluster; determining the data volume of the data to be transmitted, obtaining the target data volume, setting the initial receiving data volume of each receiving port based on the target data volume, obtaining P initial receiving data volumes, transmitting the data to be transmitted based on the P initial receiving data volumes, and recording the data transmission duration; when the data transmission duration reaches the preset duration, obtaining the amount of unreceived data at each receiving port, obtaining P unreceived data volumes, adjusting the amount of unreceived data at each receiving port, obtaining P adjusted data volumes, wherein the amount of unreceived data is less than or equal to the initial receiving data volume; continuing to transmit the remaining data in the data to be transmitted based on the P adjusted data volumes, and updating the data transmission duration, until there is no unreceived data at each receiving port.

[0007] Optionally, determining the number of database nodes required to transmit the data to be transmitted within the initial transmission duration, obtaining the first number of nodes, and determining the receiving ports for receiving the data to be transmitted based on the first number of nodes, obtaining P receiving ports includes: determining the corresponding number of database nodes in a preset table based on the initial transmission duration, obtaining the first number of nodes, wherein the preset table includes multiple initial transmission durations and the number of database nodes corresponding to each initial transmission duration, and the initial transmission duration and the number of database nodes are directly proportional; determining whether the number of database nodes is greater than the first number of nodes; if the number of database nodes is less than or equal to the first number of nodes, obtaining the receiving ports of all database nodes in the target database cluster, obtaining P receiving ports; if the number of database nodes is greater than the first number of nodes, arbitrarily selecting the first number of database nodes from the target database cluster, and obtaining the receiving port of each database node, obtaining P receiving ports.

[0008] Optionally, after determining whether the number of database nodes is greater than the number of first nodes, the method further includes: if the number of database nodes is greater than the number of first nodes, determining the number of first nodes as the number of receiving ports P, and determining the data receiving rate of each database node in the target database cluster to obtain multiple data receiving rates; selecting P rate values ​​from the multiple data receiving rates in descending order, and determining the database node corresponding to each rate value to obtain P target databases; obtaining the receiving port of each target database to obtain P receiving ports.

[0009] Optionally, setting the initial received data volume for each receiving port based on the target data volume includes: obtaining the data receiving rate of the database node to which each receiving port belongs, and obtaining multiple receiving rates; calculating the receiving rate ratio between adjacent receiving ports; and determining the initial received data volume for each receiving port based on the receiving rate ratio and the target data volume, wherein the initial received data volume is proportional to the receiving rate.

[0010] Optionally, after obtaining the amount of unreceived data at each receiving port and obtaining P unreceived data amounts, the method further includes: determining the data reception duration based on the amount of unreceived data at each receiving port, the initial received data amount, and a preset duration, to obtain P data reception durations; determining the remaining transmission duration based on the initial transmission duration and the preset duration; determining whether all P data reception durations are less than the remaining transmission duration; if all P data reception durations are less than or equal to the remaining transmission duration, performing the step of continuing to transmit the remaining data in the data to be transmitted based on the P adjusted data amounts, and updating the data transmission duration; if any of the P data reception durations is greater than the remaining transmission duration, performing the step of adjusting the amount of unreceived data at each receiving port to obtain P adjusted data amounts.

[0011] Optionally, determining the data reception duration based on the amount of unreceived data, the initial amount of received data, and the preset duration for each receiving port to obtain P data reception durations includes: calculating the amount of received data based on the amount of unreceived data and the initial amount of received data for each receiving port; calculating the current transmission rate based on the amount of received data and the preset duration for each receiving port; and calculating the data reception duration for each receiving port based on the current transmission rate and the amount of unreceived data for each receiving port to obtain P data reception durations.

[0012] Optionally, adjusting the amount of unreceived data at each receiving port to obtain P adjusted data amounts includes: adding the P unreceived data amounts to obtain a first data reception amount; obtaining the data reception rate of the database node to which each receiving port belongs to obtain multiple reception rates; calculating the reception rate ratio between adjacent receiving ports; and determining the adjusted data amount for each receiving port based on the reception rate ratio and the first data reception amount, wherein the adjusted data amount is proportional to the reception rate.

[0013] According to another aspect of this application, a data transmission apparatus is provided. The apparatus includes: a first determining unit, configured to determine data to be transmitted in a target system and determine an initial transmission duration for the data to be transmitted; a second determining unit, configured to determine the number of database nodes in a target database cluster, determine the number of database nodes required to transmit the data to be transmitted within the initial transmission duration, obtain a first number of nodes, and determine receiving ports for receiving the data to be transmitted based on the first number of nodes, obtaining P receiving ports, wherein each database node is equipped with a receiving port, and the first number of nodes is less than or equal to the number of database nodes in the target database cluster; and a setting unit, configured to determine the data volume of the data to be transmitted, obtain a target data volume, and set the target data volume... The initial received data volume is set for each receiving port, resulting in P initial received data volumes. Data to be transmitted is then transmitted based on these P initial received data volumes, and the data transmission duration is recorded. An adjustment unit is used to, when the data transmission duration reaches a preset duration, obtain the unreceived data volume for each receiving port, resulting in P unreceived data volumes. The unreceived data volume for each receiving port is adjusted to obtain P adjusted data volumes, where the unreceived data volume is less than or equal to the initial received data volume. A transmission unit is used to continue transmitting the remaining data in the data to be transmitted based on the P adjusted data volumes, and to update the data transmission duration until there is no unreceived data at each receiving port.

[0014] According to another aspect of the present invention, a computer storage medium is also provided for storing a program, wherein the program, when running, controls the device where the computer storage medium is located to execute a data transmission method.

[0015] According to another aspect of the present invention, an electronic device is also provided, comprising one or more processors and a memory; the memory stores computer-readable instructions, and the processor is configured to execute the computer-readable instructions, wherein the computer-readable instructions execute a data transmission method when executed.

[0016] This application employs the following steps: First, determine the data to be transmitted in the target system and the initial transmission duration of the data to be transmitted. Second, determine the number of database nodes in the target database cluster, determine the number of database nodes required to transmit the data to be transmitted within the initial transmission duration, obtain the first number of nodes, and determine the receiving ports for receiving the data to be transmitted based on the first number of nodes, obtaining P receiving ports, where each database node is equipped with a receiving port, and the number of the first nodes is less than or equal to the number of database nodes in the target database cluster. Third, determine the data volume of the data to be transmitted, obtain the target data volume, set the initial receiving data volume for each receiving port based on the target data volume, obtaining P initial receiving data volumes, transmit the data to be transmitted based on the P initial receiving data volumes, and record the data transmission duration. Fourth, when the data transmission duration reaches the preset duration, obtain the amount of unreceived data for each receiving port, obtaining P unreceived data volumes, adjust the unreceived data volumes for each receiving port, obtaining P adjusted data volumes, where the unreceived data volumes are less than or equal to the initial receiving data volumes. Fifth, continue transmitting the remaining data in the data to be transmitted based on the P adjusted data volumes, and update the data transmission duration until there is no unreceived data at each receiving port. This invention addresses the issue of low data import speed when importing data into a database system in related technologies. By determining the amount of data received by each receiving port within a certain time period, the receiving speed of each receiving port is calculated. Based on the receiving speed, the amount of data to be received by each receiving port is adjusted in a timely manner, thereby improving the data receiving speed of the target database cluster. Attached Figure Description

[0017] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0018] Figure 1 This is a flowchart of a data transmission method provided according to an embodiment of this application;

[0019] Figure 2 This is a flowchart of an optional data transmission method provided according to an embodiment of this application;

[0020] Figure 3 This is a schematic diagram of a data transmission apparatus provided according to an embodiment of this application;

[0021] Figure 4 This is a schematic diagram of an electronic device provided according to an embodiment of this application. Detailed Implementation

[0022] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0023] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0024] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0025] It should be noted that all information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are information and data authorized by the user or fully authorized by all parties. For example, this system has an interface with relevant users or organizations. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or organization through the interface, and obtain the relevant information after receiving consent information from the aforementioned user or organization.

[0026] It should be noted that the data transmission methods, apparatus, storage media, and electronic devices defined in this disclosure can be used in the field of big data, or in any field other than big data. The application fields of the data transmission methods, apparatus, storage media, and electronic devices defined in this disclosure are not limited.

[0027] For ease of description, the following explains some of the nouns or terms used in the embodiments of this application:

[0028] Hive: A data warehouse tool based on Hadoop, used for data extraction, transformation, and loading.

[0029] According to an embodiment of this application, a data transmission method is provided.

[0030] Figure 1 This is a flowchart of a data transmission method provided according to an embodiment of this application. For example... Figure 1 As shown, the method includes the following steps:

[0031] Step S101: Determine the data to be transmitted in the target system and determine the initial transmission duration of the data to be transmitted.

[0032] Specifically, the target system can be the Hive data warehouse tool, which can include multiple processed data sets. After processing, the data needs to be sent from Hive to the target database cluster, which can be a Clickhouse cluster.

[0033] It should be noted that different data have different transmission times. For example, data A needs to be transmitted to the server cluster for use within 10 seconds, while data B can be transmitted to the server cluster for storage within 1 day. Therefore, when transmitting data to be transmitted, it is also necessary to determine the maximum time required for transmitting the data to be transmitted, that is, the initial transmission duration. The data to be transmitted must be transmitted within the initial transmission duration.

[0034] Step S102: Determine the number of database nodes in the target database cluster, determine the number of database nodes required to transmit the data to be transmitted within the initial transmission duration, obtain the first number of nodes, and determine the receiving ports used to receive the data to be transmitted based on the first number of nodes, obtaining P receiving ports, wherein each database node is equipped with a receiving port, and the number of the first nodes is less than or equal to the number of database nodes in the target database cluster.

[0035] It should be noted that before transmitting data, the number of shards in the Clickhouse cluster (shards refer to a type of database partitioning that can divide a large database into smaller, more manageable parts), i.e., the number of nodes, needs to be determined. Each node corresponds to a data receiving port, and the data receiving ports of each node do not affect each other.

[0036] Specifically, after determining the total number of database nodes in the target database cluster, the number of database nodes required to transmit the data can be determined based on the initial transmission duration, thus obtaining the first number of nodes. For example, if the initial transmission duration is long, the number of required database nodes can be reduced, and if the initial transmission duration is short, the number of database nodes needs to be increased. Since the number of nodes may be very large, if the initial transmission duration of the data to be transmitted is long, it is not necessary to use the receiving ports corresponding to all database nodes for data reception, thereby managing and using the data receiving ports more rationally.

[0037] Furthermore, once the required number of nodes for the first node is determined, that number of data receiving ports can be obtained from the target database cluster, resulting in P data receiving ports. At this point, the value of P is the number of the first node.

[0038] Step S103: Determine the amount of data to be transmitted, obtain the target amount of data, set the initial amount of data to be received for each receiving port according to the target amount of data, obtain P initial amounts of data to be received, transmit the data to be transmitted according to the P initial amounts of data to be received, and record the data transmission time.

[0039] Specifically, after determining the number of receiving ports, the amount of data to be transmitted can be determined, and an initial data receiving amount can be allocated to each receiving port based on the data amount. For example, if the amount of data to be transmitted is 10GB and P=10, then the initial data receiving amount for each receiving port can be 1GB. After determining the initial data receiving amount for each receiving port, data transmission can begin, and timing can start.

[0040] Step S104: When the data transmission duration reaches the preset duration, obtain the amount of unreceived data for each receiving port to obtain P unreceived data amounts. Adjust the amount of unreceived data for each receiving port to obtain P adjusted data amounts, wherein the amount of unreceived data is less than or equal to the initial received data amount.

[0041] Specifically, when the timing duration equals the preset duration, the amount of unreceived data at each receiving port can be obtained, resulting in P unreceived data amounts. Then, the receiving rate of each receiving port can be calculated based on the amount of unreceived data. For example, if the amount of unreceived data at a certain port is 20G and the total data amount is 25G, then the receiving rate is (25-20) / preset duration. Thus, the unreceived data can be redistributed in a timely manner based on the receiving rate, thereby improving the data receiving speed.

[0042] Step S105: Continue transmitting the remaining data in the data to be transmitted based on the P adjusted data volumes, and update the data transmission duration until there is no unreceived data at each receiving port.

[0043] Specifically, after adjusting the amount of data received by each receiving port for the first time, data transmission continues based on the new amount of data received, and the timer is reset. The above operation is repeated, that is, the amount of data received by each receiving port is re-determined every preset time interval until there is no unreceived data on each receiving port, that is, all data has been received. At this point, all the data to be transmitted has been transmitted from Hive to the Clickhouse cluster, and the data transmission is complete.

[0044] The data transmission method provided in this application involves: determining the data to be transmitted in the target system and determining the initial transmission duration of the data to be transmitted; determining the number of database nodes in the target database cluster and the number of database nodes required to transmit the data to be transmitted within the initial transmission duration, obtaining a first number of nodes; determining receiving ports for receiving the data to be transmitted based on the first number of nodes, obtaining P receiving ports, wherein each database node is equipped with a receiving port, and the number of first nodes is less than or equal to the number of database nodes in the target database cluster; determining the data volume of the data to be transmitted, obtaining a target data volume; setting the initial receiving data volume for each receiving port based on the target data volume, obtaining P initial receiving data volumes; transmitting the data to be transmitted based on the P initial receiving data volumes and recording the data transmission duration; when the data transmission duration reaches a preset duration, obtaining the amount of unreceived data at each receiving port, obtaining P unreceived data volumes; adjusting the amount of unreceived data at each receiving port, obtaining P adjusted data volumes, wherein the amount of unreceived data is less than or equal to the initial receiving data volume; continuing to transmit the remaining data in the data to be transmitted based on the P adjusted data volumes and updating the data transmission duration until there is no unreceived data at each receiving port. This invention addresses the issue of low data import speed when importing data into a database system in related technologies. By determining the amount of data received by each receiving port within a certain time period, the receiving speed of each receiving port is calculated. Based on the receiving speed, the amount of data to be received by each receiving port is adjusted in a timely manner, thereby improving the data receiving speed of the target database cluster.

[0045] To conserve the use of receiving ports, optionally, in the data transmission method provided in this application embodiment, the number of database nodes required to transmit the data to be transmitted within the initial transmission duration is determined to obtain a first number of nodes, and receiving ports for receiving the data to be transmitted are determined based on the first number of nodes to obtain P receiving ports. This includes: determining the corresponding number of database nodes in a preset table based on the initial transmission duration to obtain a first number of nodes, wherein the preset table includes multiple initial transmission durations and the number of database nodes corresponding to each initial transmission duration, and the initial transmission duration and the number of database nodes are directly proportional; determining whether the number of database nodes is greater than the first number of nodes; if the number of database nodes is less than or equal to the first number of nodes, obtaining the receiving ports of all database nodes in the target database cluster to obtain P receiving ports; if the number of database nodes is greater than the first number of nodes, arbitrarily selecting the first number of database nodes from the target database cluster, and obtaining the receiving port to which each database node belongs to, to obtain P receiving ports.

[0046] Specifically, when determining the number of receiving ports, the number of database nodes can be determined from the preset table based on the initial transmission duration to obtain the first number of nodes. That is, the preset table stores multiple sets of correspondences between duration and number of nodes. These multiple sets of correspondences are determined based on historical data transmission experience. The shorter the time, the more nodes are required.

[0047] After determining the number of the first node and the number of nodes in the target database cluster, we can determine whether the number of database nodes is greater than the number of the first node. If it is less than or equal to the number of the first node, it means that even if all nodes in the database cluster are in use, the requirement of the number of the first node cannot be met. In this case, we can only open the data receiving ports corresponding to all nodes in the target database cluster to ensure that the data can be transmitted within the initial transmission time. Therefore, the value of P in this state is also the number of all nodes in the target database cluster.

[0048] Furthermore, when the number of database nodes is greater than the number of the first node, it indicates that there are surplus ports in the database cluster. In this case, the number of receiving ports equal to the number of the first node can be randomly selected from multiple receiving ports in the database cluster, resulting in P receiving ports. In this case, the value of P is the number of the first node. This embodiment achieves the efficiency of reasonably determining and allocating receiving ports.

[0049] To improve data transmission efficiency, optionally, in the data transmission method provided in this application embodiment, after determining whether the number of database nodes is greater than the number of first nodes, the method further includes: if the number of database nodes is greater than the number of first nodes, determining the number of first nodes as the number of receiving ports P, and determining the data receiving rate of each database node in the target database cluster to obtain multiple data receiving rates; selecting P rate values ​​from the multiple data receiving rates in descending order, and determining the database node corresponding to each rate value to obtain P target databases; obtaining the receiving port of each target database to obtain P receiving ports.

[0050] Specifically, when the number of database nodes is greater than the number of first nodes, the number of first nodes is determined as the number of receiving ports P. In this case, when selecting P receiving ports, the receiving ports can be selected based on the receiving rate of each receiving port. That is, based on the historical data receiving rate, the P receiving ports with the highest receiving rate are selected from multiple receiving ports in the target database cluster, and the data to be transmitted is received through these P receiving ports, thereby ensuring the stability and efficiency of data transmission.

[0051] In order to reasonably allocate the data transmission volume of each port and thus ensure that the data can be transmitted in the shortest possible time, optionally, in the data transmission method provided in this application embodiment, setting the initial received data volume of each receiving port according to the target data volume includes: obtaining the data receiving rate of the database node to which each receiving port belongs, and obtaining multiple receiving rates; calculating the receiving rate ratio between adjacent receiving ports; determining the initial received data volume of each receiving port according to the receiving rate ratio and the target data volume, wherein the initial received data volume is proportional to the receiving rate.

[0052] Specifically, when setting the initial amount of data to be received for each receiving port, you can first determine the data receiving rate of each receiving port, and then determine the initial amount of data to be received for each receiving port based on the receiving rate.

[0053] For example, with 5 receiving ports, their receiving rates are 10G / h, 20G / h, 30G / h, 40G / h, and 50G / h. The ratio of these receiving rates is 1:2:3:4:5. Therefore, if the amount of data to be received is 15G, it can be allocated as 1G, 2G, 3G, 4G, and 5G. This achieves the goal of allocating the initial amount of data to be received based on the receiving rate, ensuring data transmission efficiency.

[0054] To adjust the amount of data received at a port according to the real-time transmission rate, optionally, in the data transmission method provided in this application embodiment, after obtaining the amount of unreceived data at each receiving port and obtaining P amounts of unreceived data, the method further includes: determining a data reception duration based on the amount of unreceived data at each receiving port, the initial amount of received data, and a preset duration, to obtain P data reception durations; determining the remaining transmission duration based on the initial transmission duration and the preset duration; determining whether all P data reception durations are less than the remaining transmission duration; if all P data reception durations are less than or equal to the remaining transmission duration, performing the step of continuing to transmit the remaining data in the data to be transmitted based on the P adjusted data amounts, and updating the data transmission duration; if any of the P data reception durations is greater than the remaining transmission duration, performing the step of adjusting the amount of unreceived data at each receiving port to obtain P adjusted data amounts.

[0055] Specifically, after the first round of data reception is completed, the data reception duration can be determined based on the amount of unreceived data, the initial amount of received data, and the preset duration for each receiving port, resulting in P data reception durations.

[0056] To accurately calculate the data reception duration, optionally, in the data transmission method provided in this application embodiment, determining the data reception duration based on the amount of unreceived data, the initial amount of received data, and the preset duration for each receiving port to obtain P data reception durations includes: calculating the amount of received data based on the amount of unreceived data and the initial amount of received data for each receiving port; calculating the current transmission rate based on the amount of received data and the preset duration for each receiving port; and calculating the data reception duration for each receiving port based on the current transmission rate and the amount of unreceived data for each receiving port to obtain P data reception durations.

[0057] Specifically, after the first round of data reception, the amount of data received can be calculated based on the amount of unreceived data and the initial amount of data received. For example, if the amount of unreceived data is 8GB and the initial amount of data received is 10GB, then the amount of data received is 10 - 8 = 2GB. After obtaining the amount of data received, since the reception time of the first round of data is a preset duration, the amount of data received can be divided by the preset duration to obtain the current transmission rate. Then, the data reception time of the receiving port can be calculated based on the current transmission rate and the amount of unreceived data. The calculated data reception time is the time required for each receiving port to receive all the unreceived data.

[0058] For example, if the preset duration is 1 hour, the current transmission rate can be 2G / 1h = 2G / h. Then, the amount of unreceived data can be divided by the current transmission rate to get a data reception duration of 4 hours. That is, the receiving port needs another 4 hours to complete the data reception operation.

[0059] Furthermore, after obtaining the reception duration of P data points, the initial transmission duration and the preset duration can be subtracted to obtain the remaining transmission duration. It should be noted that the preset duration used in this calculation is related to the number of repetitions. In the first calculation, the runtime is 1 times the preset duration; in the second calculation, the runtime is 2 times the preset duration. Therefore, the preset duration needs to be multiplied by the number of loops to obtain the accurate preset duration. For example, if the initial transmission duration is 5 hours and the preset duration is 1 hour, the second determination to adjust the transmitted data amount indicates that data has already been transmitted for 2 preset durations. Therefore, the remaining transmission duration is the initial transmission duration - 2 * preset duration = 3 hours.

[0060] After obtaining the remaining transmission time, it is necessary to determine whether the reception time of all P data points is less than the remaining transmission time. If the reception time of all P data points is less than or equal to the remaining transmission time, it indicates that the transmission operation of the data to be transmitted can be completed within the initial transmission time according to the current transmission rate. In this case, no adjustment is required, and data transmission can continue.

[0061] If any of the P data reception times exceeds the remaining transmission time, it indicates that the data transmission operation cannot be completed within the initial transmission time according to the current data transmission method. Therefore, it is necessary to adjust the data transmission rate of each receiving port, thereby shortening the transmission time and ensuring that the data transmission operation can be completed within the initial transmission time.

[0062] Optionally, in the data transmission method provided in this application embodiment, adjusting the amount of unreceived data at each receiving port to obtain P adjusted data amounts includes: adding the P unreceived data amounts to obtain a first data reception amount; obtaining the data reception rate of the database node to which each receiving port belongs to obtain multiple reception rates; calculating the reception rate ratio between adjacent receiving ports; and determining the adjusted data amount for each receiving port based on the reception rate ratio and the first data reception amount, wherein the adjusted data amount is proportional to the reception rate.

[0063] Specifically, when adjusting the amount of unreceived data for each receiving port, it is first necessary to calculate the receiving rate of each receiving port in the previous preset time period, add up the amount of unreceived data for each receiving port to obtain the total amount of unreceived data, and then distribute the total amount of unreceived data according to the ratio of the receiving rates of each receiving port in the previous preset time period to obtain the adjusted amount of data.

[0064] For example, within a preset duration period of 1 hour, if port 1 receives 1GB of data, port 2 receives 2GB of data, and port 3 receives 3GB of data, then the rate ratio is 1:2:3. When the total amount of unreceived data is less than 18GB, the original data allocation method of 6G / 6G / 6G can be changed to 3G / 6G / 9G, thereby improving the overall transmission efficiency of the data to be transmitted.

[0065] Figure 2 This is a flowchart of an optional data transmission method provided according to an embodiment of this application, such as... Figure 2 As shown, first, the number of files in the Hive table is obtained, where the number of files represents the amount of data to be transferred. Then, the number of shards in the Clickhouse cluster and the IP addresses of the shard nodes are obtained, and the number of ports to be used is determined.

[0066] After determining the number of ports, all data files are evenly distributed to each port according to the number of files, and transmission is carried out. After a preset time, it is determined whether there are any data files that have not been transmitted. If there are no untransmitted files, the data transmission is considered complete. If there are still untransmitted files, the concurrent transmission speed of each port is calculated, and the number of concurrent transmission files is redistributed according to the transmission speed. Transmission continues until there is no untransmitted data, thus completing the data transmission. In this way, by continuously adjusting the data transmission speed of each port during the data transmission process, the data transmission efficiency is improved.

[0067] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0068] This application also provides a data transmission apparatus. It should be noted that the data transmission apparatus of this application can be used to execute the data transmission method provided in this application. The data transmission apparatus provided in this application will be described below.

[0069] Figure 3 This is a schematic diagram of a data transmission apparatus provided according to an embodiment of this application. For example... Figure 3 As shown, the device includes: a first determining unit 31, a second determining unit 32, a setting unit 33, an adjusting unit 34, and a transmission unit 35.

[0070] The first determining unit 31 is used to determine the data to be transmitted in the target system and to determine the initial transmission duration of the data to be transmitted.

[0071] The second determining unit 32 is used to determine the number of database nodes in the target database cluster, determine the number of database nodes required to transmit the data to be transmitted within the initial transmission duration, obtain the first number of nodes, and determine the receiving ports for receiving the data to be transmitted based on the first number of nodes, thereby obtaining P receiving ports, wherein each database node is equipped with a receiving port, and the first number of nodes is less than or equal to the number of database nodes in the target database cluster.

[0072] Setting unit 33 is used to determine the amount of data to be transmitted, obtain the target data amount, set the initial receive data amount of each receiving port according to the target data amount, obtain P initial receive data amounts, transmit the data to be transmitted according to the P initial receive data amounts, and record the data transmission time.

[0073] The adjustment unit 34 is used to obtain the amount of unreceived data at each receiving port when the data transmission duration reaches the preset duration, obtain P unreceived data amounts, adjust the amount of unreceived data at each receiving port, and obtain P adjusted data amounts, wherein the amount of unreceived data is less than or equal to the initial received data amount.

[0074] The transmission unit 35 is used to continue transmitting the remaining data in the data to be transmitted according to the P adjusted data amounts, and to update the data transmission duration until there is no unreceived data at each receiving port.

[0075] The data transmission apparatus provided in this application embodiment determines the data to be transmitted in the target system by a first determining unit 31 and determines the initial transmission duration of the data to be transmitted; a second determining unit 32 determines the number of database nodes in the target database cluster, determines the number of database nodes required to transmit the data to be transmitted within the initial transmission duration, obtains a first number of nodes, and determines receiving ports for receiving the data to be transmitted based on the first number of nodes, obtaining P receiving ports, wherein each database node is equipped with a receiving port, and the number of first nodes is less than or equal to the number of database nodes in the target database cluster; a setting unit 33 determines the data volume of the data to be transmitted, obtains the target data volume, and sets the target data volume according to the data volume. The target data volume is set, and an initial received data volume is determined for each receiving port, resulting in P initial received data volumes. Data to be transmitted is then transmitted based on these P initial received data volumes, and the data transmission duration is recorded. When the data transmission duration reaches a preset time, the adjustment unit 34 obtains the amount of unreceived data from each receiving port, resulting in P unreceived data volumes. These unreceived data volumes are adjusted to obtain P adjusted data volumes, where the unreceived data volume is less than or equal to the initial received data volume. The transmission unit 35 continues to transmit the remaining data from the data to be transmitted based on the P adjusted data volumes and updates the data transmission duration until there is no unreceived data at each receiving port. This solves the problem of low data import speed when importing data into a database system in related technologies. By determining the amount of data received by each receiving port within a certain time, the receiving speed of each receiving port is calculated, and the amount of data to be received by each receiving port is adjusted in a timely manner based on the receiving speed, thereby improving the data receiving speed of the target database cluster.

[0076] Optionally, in the data transmission apparatus provided in this application embodiment, the second determining unit 32 includes: a first determining module, configured to determine the corresponding number of database nodes in a preset table according to the initial transmission duration, to obtain a first number of nodes, wherein the preset table includes multiple initial transmission durations and the number of database nodes corresponding to each initial transmission duration, and the initial transmission duration and the number of database nodes are directly proportional; a judging module, configured to judge whether the number of database nodes is greater than the number of first nodes; a first obtaining module, configured to obtain the receiving ports of all database nodes in the target database cluster when the number of database nodes is less than or equal to the number of first nodes, to obtain P receiving ports; and a second obtaining module, configured to arbitrarily select the number of database nodes of the first number from the target database cluster when the number of database nodes is greater than the number of first nodes, and obtain the receiving port to which each database node belongs, to obtain P receiving ports.

[0077] Optionally, in the data transmission apparatus provided in the embodiments of this application, the apparatus further includes: a third determining unit, configured to determine the number of first nodes as the number of receiving ports P when the number of database nodes is greater than the number of first nodes, and determine the data receiving rate of each database node in the target database cluster to obtain multiple data receiving rates; a fourth determining unit, configured to select P rate values ​​from the multiple data receiving rates in descending order, and determine the database node corresponding to each rate value to obtain P target databases; and an obtaining unit, configured to obtain the receiving port of each target database to obtain P receiving ports.

[0078] Optionally, in the data transmission apparatus provided in this application embodiment, the setting unit 33 includes: a third acquisition module, used to acquire the data receiving rate of the database node to which each receiving port belongs, and obtain multiple receiving rates; a first calculation module, used to calculate the receiving rate ratio between adjacent receiving ports; and a second determination module, used to determine the initial receiving data amount of each receiving port according to the receiving rate ratio and the target data amount, wherein the initial receiving data amount is proportional to the receiving rate.

[0079] Optionally, in the data transmission apparatus provided in this application embodiment, the apparatus further includes: a fifth determining unit, configured to determine a data reception duration based on the amount of unreceived data, the initial amount of received data, and a preset duration for each receiving port, to obtain P data reception durations; a sixth determining unit, configured to determine the remaining transmission duration based on the initial transmission duration and the preset duration; a judging unit, configured to judge whether all P data reception durations are less than the remaining transmission duration; a first execution unit, configured to execute the step of continuing to transmit the remaining data in the data to be transmitted based on P adjusted data amounts and updating the data transmission duration when all P data reception durations are less than or equal to the remaining transmission duration; and a second execution unit, configured to execute the step of adjusting the amount of unreceived data for each receiving port to obtain P adjusted data amounts when there is one of the P data reception durations that is greater than the remaining transmission duration.

[0080] Optionally, in the data transmission apparatus provided in this application embodiment, the fifth determining unit includes: a second calculation module, used to calculate the received data amount based on the unreceived data amount and the initial received data amount of each receiving port; a third calculation module, used to calculate the current transmission rate based on the received data amount and the preset duration of each receiving port; and a fourth calculation module, used to calculate the data reception duration of each receiving port based on the current transmission rate and the unreceived data amount of each receiving port, to obtain P data reception durations.

[0081] Optionally, in the data transmission apparatus provided in this application embodiment, the adjustment unit 34 includes: a fifth calculation module, used to add up P unreceived data amounts to obtain a first data reception amount; a fourth acquisition module, used to acquire the data reception rate of the database node to which each receiving port belongs, to obtain multiple reception rates; a sixth calculation module, used to calculate the reception rate ratio between adjacent receiving ports; and a third determination module, used to determine the adjusted data amount of each receiving port based on the reception rate ratio and the first data reception amount, wherein the adjusted data amount is proportional to the reception rate.

[0082] The aforementioned data transmission device includes a processor and a memory. The first determining unit 31, the second determining unit 32, the setting unit 33, the adjusting unit 34, the transmission unit 35, etc., are all stored in the memory as program units. The processor executes the aforementioned program units stored in the memory to realize the corresponding functions.

[0083] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured; adjusting kernel parameters addresses the issue of slow data import speeds when importing data into database systems in related technologies.

[0084] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0085] This invention provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements the data transmission method.

[0086] This invention provides a processor for running a program, wherein the program executes the data transmission method during runtime.

[0087] like Figure 4 As shown, this embodiment of the invention provides an electronic device 40, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements the aforementioned data transmission method. The device in this document can be a server, PC, PAD, mobile phone, etc.

[0088] This application also provides a computer program product that, when executed on a data processing device, is suitable for executing a program that initializes the above-described data transmission method steps.

[0089] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0090] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0091] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0092] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0093] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0094] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0095] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0096] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0097] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A data transmission method, characterized in that, include: Identify the data to be transmitted in the target system and determine the initial transmission duration of the data to be transmitted; The number of database nodes in the target database cluster is determined, the number of database nodes required to transmit the data to be transmitted within the initial transmission duration is determined, the first number of nodes is obtained, and the receiving ports for receiving the data to be transmitted are determined according to the first number of nodes, resulting in P receiving ports, wherein each database node is equipped with a receiving port, and the first number of nodes is less than or equal to the number of database nodes in the target database cluster. Determine the amount of data to be transmitted to obtain the target amount of data. Set the initial amount of data to be received for each receiving port according to the target amount of data to obtain P initial amounts of data to be received. Transmit the data to be transmitted according to the P initial amounts of data to be received and record the data transmission time. When the data transmission duration reaches the preset duration, the amount of unreceived data at each receiving port is obtained, resulting in P unreceived data amounts. The amount of unreceived data at each receiving port is adjusted to obtain P adjusted data amounts, wherein the amount of unreceived data is less than or equal to the initial received data amount. Based on the P adjusted data volumes, continue transmitting the remaining data in the data to be transmitted, and update the data transmission duration until there is no unreceived data at each receiving port; Specifically, after obtaining the amount of unreceived data at each receiving port, resulting in P unreceived data amounts: The data reception duration is determined based on the amount of unreceived data at each receiving port, the initial received data amount, and a preset duration, resulting in P data reception durations; the remaining transmission duration is determined based on the initial transmission duration and the preset duration; it is determined whether all P data reception durations are less than the remaining transmission duration; if all P data reception durations are less than or equal to the remaining transmission duration, the step of continuing to transmit the remaining data in the data to be transmitted based on the P adjusted data amounts and updating the data transmission duration is executed; if any of the P data reception durations is greater than the remaining transmission duration, the step of adjusting the amount of unreceived data at each receiving port to obtain P adjusted data amounts is executed. The process of determining the data reception duration based on the amount of unreceived data, the initial amount of received data, and the preset duration for each receiving port, to obtain P data reception durations, includes: calculating the amount of received data based on the amount of unreceived data and the initial amount of received data for each receiving port; calculating the current transmission rate based on the amount of received data and the preset duration for each receiving port; and calculating the data reception duration for each receiving port based on the current transmission rate and the amount of unreceived data for each receiving port, thus obtaining P data reception durations.

2. The method according to claim 1, characterized in that, The number of database nodes required to transmit the data to be transmitted within the initial transmission duration is determined to obtain a first number of nodes. Based on this first number of nodes, receiving ports for receiving the data to be transmitted are determined, resulting in P receiving ports, including: The number of database nodes is determined from the preset table based on the initial transmission duration to obtain the first number of nodes. The preset table includes multiple initial transmission durations and the number of database nodes corresponding to each initial transmission duration. The initial transmission duration and the number of database nodes are directly proportional. Determine whether the number of database nodes is greater than the number of the first nodes; If the number of database nodes is less than or equal to the number of the first nodes, obtain the receiving ports of all database nodes in the target database cluster to obtain P receiving ports; If the number of database nodes is greater than the number of the first nodes, select the first number of database nodes from the target database cluster and obtain the receiving port to which each database node belongs, thus obtaining P receiving ports.

3. The method according to claim 2, characterized in that, After determining whether the number of database nodes is greater than the number of the first nodes, the method further includes: When the number of database nodes is greater than the number of the first nodes, the number of the first nodes is determined as the number of receiving ports P, and the data receiving rate of each database node in the target database cluster is determined to obtain multiple data receiving rates. P data receiving rates are selected from the plurality of rates in descending order, and the database node corresponding to each rate value is determined to obtain P target databases; Obtain the receiving port for each target database, resulting in P receiving ports.

4. The method according to claim 1, characterized in that, Setting the initial receive data volume for each receiving port based on the target data volume includes: Obtain the data reception rate of the database node to which each receiving port belongs, and get multiple reception rates; Calculate the ratio of the receiving rates between adjacent receiving ports; The initial received data amount for each receiving port is determined based on the received rate ratio and the target data amount, wherein the initial received data amount is proportional to the received rate.

5. The method according to claim 1, characterized in that, The amount of unreceived data at each receiving port is adjusted to obtain P adjusted data amounts, including: The first data received quantity is obtained by adding the P unreceived data quantities together. Obtain the data reception rate of the database node to which each receiving port belongs, and get multiple reception rates; Calculate the ratio of the receiving rates between adjacent receiving ports; The adjusted data volume for each receiving port is determined based on the receiving rate ratio and the first data receiving volume, wherein the adjusted data volume is proportional to the receiving rate.

6. A data transmission device, characterized in that, include: The first determining unit is used to determine the data to be transmitted in the target system and to determine the initial transmission duration of the data to be transmitted. The second determining unit is used to determine the number of database nodes in the target database cluster, determine the number of database nodes required to transmit the data to be transmitted within the initial transmission duration, obtain a first number of nodes, and determine a receiving port for receiving the data to be transmitted based on the first number of nodes, obtaining P receiving ports, wherein each database node is provided with a receiving port, and the first number of nodes is less than or equal to the number of database nodes in the target database cluster. The setting unit is used to determine the amount of data to be transmitted, obtain the target amount of data, set the initial amount of data to be received for each receiving port according to the target amount of data, obtain P initial amounts of data to be received, transmit the data to be transmitted according to the P initial amounts of data to be received, and record the data transmission time. An adjustment unit is configured to, when the data transmission duration reaches a preset duration, acquire the amount of unreceived data at each receiving port, obtain P unreceived data amounts, adjust the amount of unreceived data at each receiving port, and obtain P adjusted data amounts, wherein the amount of unreceived data is less than or equal to the initial received data amount. The transmission unit is used to continue transmitting the remaining data in the data to be transmitted according to the P adjusted data amounts, and to update the data transmission duration until there is no unreceived data at each receiving port; The fifth determining unit is used to determine the data reception duration based on the amount of unreceived data, the initial amount of received data, and the preset duration for each receiving port, thereby obtaining P data reception durations; the sixth determining unit is used to determine the remaining transmission duration based on the initial transmission duration and the preset duration; the judging unit is used to judge whether all P data reception durations are less than the remaining transmission duration; the first execution unit is used to execute the step of continuing to transmit the remaining data in the data to be transmitted based on the P adjusted data amounts and updating the data transmission duration when all P data reception durations are less than or equal to the remaining transmission duration; the second execution unit is used to execute the step of adjusting the amount of unreceived data for each receiving port to obtain P adjusted data amounts when there is one of the P data reception durations that is greater than the remaining transmission duration. The fifth determining unit includes: a second calculation module, used to calculate the amount of received data based on the amount of unreceived data and the initial amount of received data for each receiving port; a third calculation module, used to calculate the current transmission rate based on the amount of received data and the preset duration for each receiving port; and a fourth calculation module, used to calculate the data reception duration for each receiving port based on the current transmission rate and the amount of unreceived data for each receiving port, to obtain P data reception durations.

7. A computer storage medium, characterized in that, The computer storage medium is used to store a program, wherein the program, when running, controls the device where the computer storage medium is located to execute the data transmission method according to any one of claims 1 to 5.

8. An electronic device, characterized in that, It includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the data transmission method according to any one of claims 1 to 5.

Images