Data transmission method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a data transmission method and device, equipment and a storage medium. The method comprises the following steps: in response to an execution instruction of any data exchange task defined in a management and control cluster, determining a preferred management and control node and a specified data processing node corresponding to the data exchange task; and according to the type of the data exchange task, executing data transmission corresponding to the data exchange task through a target limited cache pool in the first limited cache pool of the preferred management and control node and the second limited cache pool of the specified data processing node. According to the embodiment of the invention, cross-network and cross-region data batch transmission can be realized through the limited cache pool, the data does not need to be transmitted as a file, and the high efficiency and accuracy of data transmission are improved on the basis of ensuring the safety and reliability of data transmission.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data processing technology, and specifically to a data transmission method, apparatus, device and storage medium. Background Art

[0002] With the rapid rise of Internet technology, there will be more and more demands for data transmission across networks and regions, and the data suppliers across networks and regions are mostly different.

[0003] Normally, in order to ensure the security and reliability of data transmission across networks and regions, for data transmission between two servers, the corresponding data in the database of one server is usually extracted into a file, and the secure file transfer protocol (SSH File Transfer Protocol, referred to as STFP) is used to transfer the encrypted file to the other server. After the other server decrypts it, it can be written into the database of the other server to realize cross-network and cross-regional data transmission.

[0004] However, the five data operations involved in the above data transmission process, namely data extraction, encryption, transmission, decryption and writing, are all executed sequentially, and each data operation is restricted by the execution result of the previous data operation, which greatly affects the efficiency of data transmission. Summary of the invention

[0005] The embodiments of the present application provide a data transmission method, apparatus, device and storage medium, which realize batch transmission of data across networks and regions through a limited cache pool, without the need to convert data into files for transmission, thereby improving the efficiency and accuracy of data transmission.

[0006] In a first aspect, an embodiment of the present application provides a data transmission method, the method comprising:

[0007] In response to an execution instruction of any data exchange task defined in the management and control cluster, determine a preferred management and control node and a designated data processing node corresponding to the data exchange task;

[0008] According to the type of the data exchange task, data transmission corresponding to the data exchange task is performed through the first limited buffer pool of the preferred management and control node and the target limited buffer pool in the second limited buffer pool of the designated data processing node.

[0009] In a second aspect, an embodiment of the present application provides a data transmission device, the device comprising:

[0010] A node determination module, configured to respond to an execution instruction of any data exchange task defined in the management and control cluster and determine a preferred management and control node and a designated data processing node corresponding to the data exchange task;

[0011] The data transmission module is used to perform data transmission corresponding to the data exchange task through the first limited buffer pool of the preferred management and control node and the target limited buffer pool in the second limited buffer pool of the designated data processing node according to the type of the data exchange task.

[0012] In a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising:

[0013] A processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the data transmission method provided in the first aspect of the present application.

[0014] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium for storing a computer program, wherein the computer program enables a computer to execute the data transmission method provided in the first aspect of the present application.

[0015] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the data transmission method provided in the first aspect of the present application.

[0016] The embodiment of the present application provides a data transmission method, apparatus, device and storage medium. When the execution instruction of any data exchange task defined in the control cluster is detected, the preferred control node and the designated data processing node corresponding to the data exchange task are first determined. Then, according to the type of the data exchange task, the data transmission corresponding to the data exchange task can be executed through the first limited cache pool of the preferred control node and one of the target limited cache pools in the second limited cache pool of the designated data processing node, thereby realizing cross-network and cross-regional data batch transmission through the limited cache pool without the need to transfer the data into files, thereby improving the efficiency and accuracy of data transmission while ensuring the security and reliability of data transmission. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 A data transmission architecture diagram suitable for cross-network and cross-region data transmission scenarios shown in an embodiment of the present application;

[0019] Figure 2A flowchart of a data transmission method shown in an embodiment of the present application;

[0020] Figure 3 A flowchart of another data transmission method shown in an embodiment of the present application;

[0021] Figure 4 A principle block diagram of a data transmission device shown in an embodiment of the present application;

[0022] Figure 5 A schematic block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0025] In the cross-network and cross-region data transmission scenario, the corresponding database data of any server can usually be landed into corresponding files, and transmitted to other servers using the STFP protocol for parsing and storage processing. However, the data transmission between any two servers is usually executed sequentially, which greatly affects the efficiency of data transmission. So, in order to ensure the efficiency and accuracy of data transmission, such as Figure 1 As shown, the present application will construct a data transmission architecture suitable for cross-network and cross-regional data transmission scenarios, and the data transmission architecture may include two parts: a management and control cluster and a data processing cluster.

[0026] Among them, the control cluster may include multiple control nodes, each of which can be used to uniformly manage and monitor the relevant data transmission requirements on each data processing node in the data processing cluster, to generate corresponding data exchange tasks and perform task scheduling, distribute task execution commands, and perform related data processing operations under different types of data exchange tasks. The data processing cluster may include multiple data processing nodes, each of which can be used to perform specific data processing operations corresponding to the relevant data exchange tasks according to the task execution commands distributed by the control node, such as extracting local data and reporting it to a certain control node, or receiving relevant data issued by a certain control node and performing corresponding processing on it.

[0027] Specifically, each data processing node can be a server with various data processing capabilities and data transmission requirements. The corresponding database data corresponding to each data processing node is uniformly managed and monitored by managing each control node in the cluster, so as to analyze whether there is a data transmission requirement between each control node and each data processing node.

[0028] It is understandable that the data transmission scenarios between each management and control node and each data processing node may include the following three situations:

[0029] First, each control node within the control cluster can use a total storage model to comprehensively store the database data corresponding to each data processing node, and uniformly control whether there is a transmission demand for the corresponding database data corresponding to each data processing node. Then, when the database data corresponding to a certain data processing node is updated or reaches a preset data collection time point, it is necessary for the data processing node to report the updated database data or the related data that needs to be collected to a certain control node, and the control node will synchronously write the updated database data or the related data that needs to be collected into the total storage model to ensure that the control cluster has comprehensive and accurate control over various data transmission tasks on each data processing node. In this case, there will be corresponding data reporting requirements between the control node and the data processing node, so as to achieve comprehensive and accurate control of the database data reporting tasks corresponding to the data processing node by the control node.

[0030] Secondly, each control node in the control cluster can determine whether the corresponding database data in the total storage model corresponding to the control cluster has distribution requirements for each data processing node, or determine whether each data processing node has viewing requirements for the corresponding database data in the total storage model by uniformly controlling whether each data processing node has data transmission requirements. Then, when a data processing node needs to view the database data corresponding to other data processing nodes stored in the total storage model, the control node can send the corresponding database data stored in the total storage model to the data processing node, and the data processing node will synchronously write the corresponding database data after sending to the storage model corresponding to the data processing node to ensure that the data processing node obtains the corresponding database data in a timely manner. In this case, there will be a corresponding data sending requirement between the control node and the data processing node, so as to realize the timely and effective viewing of any database data by each data processing node.

[0031] Third, each control node in the control cluster can determine whether there is a synchronization demand for corresponding database data between any two data processing nodes by uniformly controlling whether there is a data transmission demand for each data processing node. Then, when there is a data synchronization demand between any two data processing nodes, each control node in the control cluster can act as an intermediate node when any two data processing nodes perform data transmission to forward the corresponding data to be transmitted, so as to realize data exchange between any two data processing nodes. In this case, for any two data processing nodes with data synchronization demand, there will be a corresponding data reporting demand between the control node and one of the data processing nodes, and there will be a corresponding data sending demand between the control node and another of the data processing nodes, so as to use the control node as an intermediate node when any two data processing nodes perform data synchronization to realize data exchange between any two data processing nodes.

[0032] It can be seen that for each control node in the control cluster, the control node will have two data transmission situations for each data processing node: data reporting and data sending. Among them, data reporting means that the data processing node can report the corresponding data of the corresponding database to the control node, and the control node will synchronize it to the overall storage model for unified caching. Data sending means that the control node sends the corresponding data that has been reported by a data processing node and stored in the overall storage model to another data processing node, so as to write it into the storage model corresponding to the other data processing node, so as to realize data synchronization between any two data processing nodes.

[0033] It can be understood that in order to ensure that each management and control node accurately controls the data of each data processing node, the present application can set a unique node identifier for each management and control node and each data processing node to accurately implement data exchange between any management and control node and any data processing node.

[0034] Moreover, since the five data operations of data extraction, encryption, transmission, decryption and writing involved in the data transmission process are all executed sequentially when the data is transferred into files between any control node and any data processing node, each data operation will be restricted by the execution result of the previous data operation, which greatly affects the efficiency of data transmission. Therefore, in order to improve the efficiency and accuracy of data transmission, the present application can also set a limited cache pool for each control node and each data processing node, which is used to cache the corresponding data transmitted by the other party in batches according to the capacity limit of the limited cache pool, so as to realize the batch transmission of data across networks and regions without transferring the data into files.

[0035] Therefore, by utilizing the data transmission architecture constructed above, the present application can complete batch transmission of data between any two data processing nodes through each control node in the control cluster, thereby improving the efficiency and accuracy of data transmission.

[0036] Next, the present application may provide a detailed explanation of the specific data transmission process performed by each control node within the control cluster to each data processing node.

[0037] Figure 2 This is a flow chart of a data transmission method shown in an embodiment of the present application. Figure 2 , the method may include the following steps:

[0038] S210, in response to an execution instruction of any data exchange task defined in the management and control cluster, determining a preferred management and control node and a designated data processing node corresponding to the data exchange task.

[0039] For each control node in the control cluster, unified control will be performed on whether there is a data transmission demand for the corresponding database data corresponding to each data processing node, so that each control node can analyze in real time whether there is a corresponding database data transmission demand between any data processing node and the control cluster. Therefore, each control node can pre-define the corresponding data exchange task according to the data transmission demand between each data processing node and the control cluster. Among them, the data exchange task can be a data collection task for any data processing node by the control node, a data distribution task for any data processing node by the control node, and a data processing task that uses each control node in the control cluster as an intermediate node to forward the corresponding data that needs to be transmitted between any two data processing nodes in the data processing cluster.

[0040] Specifically, the various data exchange tasks defined in the control cluster can be automatically generated by each control node in the control cluster through unified control of whether there is a data transmission demand for each data processing node, and when there is a corresponding database data transmission demand at any data processing node; or they can be manually customized by the control personnel according to the specific operation requirements of each control node and each data processing node for the relevant database data.

[0041] Moreover, the data exchange tasks defined in the control cluster in the present application may include two types: data reporting tasks and data sending tasks, so as to achieve efficient data transmission between the control cluster and the data processing cluster.

[0042] It is understandable that in order to ensure accurate data transmission between each control node and each data processing node in the control cluster, it is first necessary to ensure that the corresponding control nodes in the control cluster can execute each defined data exchange task in a timely and effective manner. Therefore, for each data exchange task defined in the control cluster, the data exchange task will carry the specific task execution time, the specific data processing node participating in this data exchange, and the specific data information that needs to be transmitted to the data processing node.

[0043] Then, when each control node in the control cluster performs unified control over whether each data processing node has a data transmission demand, it can also monitor the specific execution time of each data exchange task defined therein in real time to determine whether the data exchange task needs to be executed at the current moment to ensure that the corresponding data transmission operation represented by the data exchange task can be completed in time. Alternatively, each control node in the control cluster can support each control personnel to actively initiate corresponding execution trigger operations for each data exchange task defined therein to indicate that the data exchange task needs to be executed immediately at the current moment to ensure that the corresponding data transmission operation represented by the data exchange task can be completed in time.

[0044] Therefore, for each management and control node within the management and control cluster, if it is monitored that any data exchange task defined therein has currently reached the specific execution time of the data exchange task, or if an execution trigger operation initiated by the management and control personnel for any data exchange task is detected, an execution instruction for the data exchange task can be generated first to indicate that the data exchange task needs to be executed immediately at the current moment, so as to ensure that the corresponding data transmission operation represented by the data exchange task can be completed in time.

[0045] Then, in order to ensure the efficient execution of each data exchange task, in response to the execution instruction of any data exchange task defined in the control cluster, a control node with the highest current performance can be selected from each control node in the control cluster as the preferred control node in this application, which is used to execute the specific data transmission operation under the data exchange task. Moreover, when receiving the execution instruction of any data exchange task defined in the control cluster, the data exchange task can also be parsed to determine the specific data processing node participating in this data exchange represented by the data exchange task as the designated data processing node in this application.

[0046] It can be seen that the above data exchange task can instruct to perform corresponding data transmission between the preferred management and control node and the designated data processing node.

[0047] S220: According to the type of the data exchange task, data transmission corresponding to the data exchange task is performed by optimizing the first limited buffer pool of the management and control node and the target limited buffer pool in the second limited buffer pool of the designated data processing node.

[0048] For each data exchange task defined in the control cluster, the data exchange task may include two types: a data reporting task that reports the corresponding data extracted from the corresponding storage model of a data processing node to a certain control node, and a data sending task that sends the corresponding data reported by each data processing node in the total storage model corresponding to the control cluster to a certain data processing node. Then, for the two different types of data exchange tasks, data reporting tasks and data sending tasks, the transmission directions of different types of data exchange tasks when transmitting data between the preferred control node in the control cluster and the designated data processing node in the data processing cluster will also be different.

[0049] Moreover, in order to improve the efficiency and accuracy of data transmission and avoid the problem that the data transmission performance is greatly affected by the need to execute various data transmission steps sequentially when the data is transferred into files, this application will set up a limited cache pool for each control node in the control cluster and each data processing node in the data processing cluster, which is used to cache the corresponding transmission data in batches according to the capacity limit of the limited cache pool. Then, for the data exchange task executed this time, after determining the preferred control node and the designated data processing node corresponding to the data exchange task, this application will also determine the limited cache pool set for the preferred control node as the first priority cache pool in this application; and determine the limited cache pool set for the designated data processing node as the second priority cache pool in this application.

[0050] Based on the above content, it can be known that for the data exchange task to be executed this time, the corresponding data transmission needs to be completed between the preferred control node and the designated data processing node corresponding to the data exchange task. Therefore, after determining the preferred control node and the designated data processing node corresponding to the data exchange task, the present application will also parse out the specific data information that the data exchange task needs to transmit to the designated data processing node to determine the data to be transmitted corresponding to the data exchange task. Then, by judging which node of the preferred control node and the designated data processing node the data to be transmitted is actually stored in, the data transmission direction corresponding to the data exchange task is determined, so as to select a limited cache pool from the first limited cache pool of the preferred control node and the second limited cache pool of the designated data processing node for caching the data to be transmitted corresponding to the data exchange task during the data transmission process as the corresponding target limited cache pool.

[0051] Furthermore, when the data to be transmitted corresponding to the data exchange task is transmitted between the preferred control node and the designated data processing node, the target limited cache pool can be used to continuously cache the corresponding data to be transmitted, and the corresponding node of the target limited cache pool (a node among the preferred control node and the designated data processing node) can monitor the cached data of the target limited cache pool in real time, so as to determine whether the cached capacity of the target limited cache pool has reached the upper limit of the capacity of the target limited cache pool, thereby controlling the data to be transmitted to be transmitted in batches between the preferred control node and the designated data processing node. That is, if the cached capacity of the target limited cache pool has not been transmitted when the upper limit of the capacity of the target limited cache pool has been reached, a period of idle time can be waited so that the cached data in the target limited cache pool can be successfully written into the data storage model corresponding to the corresponding node of the target limited cache pool, and after the corresponding capacity is vacated in the target limited cache pool, the data to be transmitted can be controlled to perform the corresponding data transmission between the preferred control node and the designated data processing node, and the corresponding data to be transmitted can be cached through the target limited cache pool, thereby realizing the batch transmission of data across networks and regions, and improving the efficiency and accuracy of data transmission.

[0052] In some possible implementations, considering that when executing the data transmission corresponding to the data exchange task between the preferred control node and the designated data processing node, the data transmission may fail due to network errors such as network disconnection. Then, in order to solve the above problem, for the data transmission between the preferred control node and the designated data processing node, the present application can use the first limited buffer pool of the preferred control node and the target limited buffer pool in the second limited buffer pool of the designated data processing node to perform the data transmission corresponding to the data exchange task using a transmission compensation mechanism.

[0053] That is, the present application pre-sets a default maximum number of transmissions to generate a transmission compensation mechanism. The default maximum number of transmissions can be set according to a specific data transmission scenario, and the present application does not limit this.

[0054] Furthermore, when the data transmission corresponding to the data exchange task is executed through the first limited cache pool of the preferred control node and the target limited cache pool in the second limited cache pool of the designated data processing node, if there is a data transmission failure, the corresponding secondary data transmission can be initiated for the data exchange task through the first limited cache pool of the preferred control node and the target limited cache pool in the second limited cache pool of the designated data processing node according to the default maximum number of transmissions, and this cycle is repeated until the transmission is successfully completed or the number of transmissions reaches the default maximum number of transmissions, indicating that the transmission has failed, thereby ensuring the accuracy of data transmission.

[0055] The technical solution provided by the embodiment of the present application, when detecting the execution instruction of any data exchange task defined in the control cluster, first determines the preferred control node and the designated data processing node corresponding to the data exchange task. Then, according to the type of the data exchange task, the data transmission corresponding to the data exchange task can be executed through the first limited cache pool of the preferred control node and one of the target limited cache pools in the second limited cache pool of the designated data processing node, thereby realizing cross-network and cross-regional data batch transmission through the limited cache pool without the need to transfer the data into files, thereby improving the efficiency and accuracy of data transmission while ensuring the security and reliability of data transmission.

[0056] As an optional implementation scheme in the present application, in order to ensure the safe and reliable transmission of data between each control node in the control cluster and each data processing node in the data processing cluster, the present application can provide a detailed explanation of the specific determination process of the preferred control node and the designated data processing node corresponding to any data exchange task and the specific data transmission process between the preferred control node and the designated data processing node.

[0057] Figure 3 FIG. 1 is a flow chart of another data transmission method shown in an embodiment of the present application. Figure 3 As shown, the method may specifically include the following steps:

[0058] S310, in response to an execution instruction of any data exchange task defined in the control cluster, assigning a corresponding preferred control node to the data exchange task according to load information of each control node in the control cluster.

[0059] Considering that the data processing performance of each control node in the control cluster for any data exchange task can be determined by the real-time load status of each control node. Then, in order to ensure the efficient execution of each data exchange task defined in the control cluster, when the application detects the execution instruction of any data exchange task defined in the control cluster, it will first obtain the current load information of each control node in the control cluster, and conduct a comprehensive analysis of the current load information of each control node to determine the real-time load status of each control node.

[0060] Then, by comparing the real-time load status of each control node, it can be determined which control node has the smallest real-time load, indicating that the control node with the smallest real-time load has the best data processing performance. Therefore, the present application can select a control node with the smallest real-time load from the control cluster as the corresponding preferred control node to perform the data transmission corresponding to the data exchange task.

[0061] S320: Determine a designated data processing node corresponding to the data exchange task according to the parsed transmission node information of the data exchange task.

[0062] While determining the preferred management and control node corresponding to the data exchange task, the present application can also parse the data exchange task to determine the specific data processing node information participating in the data transmission represented by the data exchange task, so as to determine the designated data processing node corresponding to the data exchange task.

[0063] S330: Sending task execution information of the data exchange task to the designated data processing node through the preferred management and control node to instruct the designated data processing node to enter a transmission working state.

[0064] After determining the preferred management and control node and designated data processing node corresponding to the data exchange task, in order to ensure the effective transmission response of the designated data processing node to the data exchange task, the present application will first parse the data exchange task to determine the specific task execution time carried by the data exchange task, the specific data processing nodes participating in this data exchange, and the specific data information that needs to be transmitted to the data processing node, etc., so as to generate the task execution information of the data exchange task. The task execution information can be used as command information initiated to indicate that the data exchange task needs to be executed immediately.

[0065] It is understandable that each control node in the control cluster will schedule the corresponding preferred control node to perform data transmission processing on the current data exchange task according to the specific information carried by each data exchange task defined therein.

[0066] Then, through the preferred management and control node, the task execution information of the data exchange task can be sent to the designated data processing node, that is, the command information used to instruct the immediate execution of the data exchange task can be distributed to the designated data processing node, so that the designated data processing node can understand the specific execution status of the data exchange task by parsing the task execution information of the data exchange task, and control the designated data processing node from the idle state to the transmission working state, and make corresponding data transmission preparations for the data exchange task in advance, so that the designated data processing node can execute the data transmission corresponding to the data exchange task as quickly and effectively as possible.

[0067] In some achievable methods, considering that each data exchange task defined in the control cluster may include a data reporting task and a data sending task, under different types of data exchange tasks, the data transmission preparation work made in advance by the designated data processing node for the data exchange task is also different. To this end, according to the types of different data exchange tasks, the preferred control node can send different task execution information to the designated data processing node to instruct the designated data processing node to make different data transmission preparation work in advance.

[0068] Therefore, if the data exchange task is a data reporting task, then the task execution information of the data exchange task may at least include the node identifier of the preferred control node and the data identifier and local extraction indication information of the data to be transmitted corresponding to the data exchange task. Among them, the local extraction indication information can be used to indicate that the designated data processing node needs to extract relevant data from the local corresponding storage model and report it to the preferred control node under this data transmission. Then, when the task execution information of the data exchange task is sent to the designated data processing node through the preferred control node, the designated data processing node can effectively understand the need to extract the data to be transmitted locally through the local extraction indication information, so as to continuously extract the data to be transmitted from the local corresponding storage model according to the data identifier included in the task execution information, and report the extracted data to be transmitted to the preferred control node according to the node identifier included in the task execution information, so as to execute the data transmission corresponding to the data exchange task.

[0069] If the data exchange task is a data sending task, then the task execution information of the data exchange task may at least include cache preparation indication information of the data to be transmitted corresponding to the data exchange task. Among them, the cache preparation indication information can be used to instruct the designated data processing node to prepare in advance for the accurate work of receiving and caching the data to be transmitted sent by the preferred control node through the second limited cache pool of the designated data processing node, without performing other data transmission operations. Then, when the task execution information of the data exchange task is sent to the designated data processing node through the preferred control node, the designated data processing node can effectively understand that the local data transmitted by the preferred control node can be directly received and cached through the second limited cache pool of the designated data processing node, thereby realizing the data transmission corresponding to the data exchange task.

[0070] S340, if the data exchange task is a data reporting task, the data to be transmitted extracted by the designated data processing node for the data exchange task is cached in batches through the first limited cache pool of the preferred management and control node, and the cached data in the first limited cache pool is processed into a warehouse to realize the data transmission corresponding to the data exchange task.

[0071] After the task execution information of the data exchange task is sent to the designated data processing node through the preferred management and control node, the preferred management and control node and the designated data processing node will prepare for the data transmission of the data exchange task in advance so as to effectively transmit the data corresponding to the data exchange task.

[0072] Therefore, if the data exchange task is a data reporting task, it means that the data exchange task can instruct the designated data processing node to report the data to be transmitted to the preferred control node. Then, the preferred control node can directly prepare in advance to receive and cache the data to be transmitted reported by the designated data processing node through the first limited cache pool of the preferred control node, and monitor the cached data in the first limited cache pool in real time so as to successfully write the cached data into the storage model corresponding to the preferred control node. After the designated data processing node receives the task execution information of the data exchange task and enters the transmission working state, it will continuously extract the corresponding data to be transmitted from the local corresponding storage model according to the data identification information of the data to be transmitted included in the task execution information of the data exchange task, and encrypt the data to be transmitted according to the node identification in the task execution information of the data exchange task and continuously report it to the first limited cache pool of the preferred control node for caching.

[0073] Among them, the designated data processing node can adopt small batch and segmented data reporting methods during the data reporting process until all the data to be transmitted are extracted from the corresponding local storage model. The end reporting command of this data reporting task can be sent back to the preferred control node, and various resources involved in this data reporting can be destroyed.

[0074] Therefore, according to the upper limit of the capacity of the first limited buffer pool of the preferred control node, the data to be transmitted that is extracted and reported by the designated data node for the data exchange task can be cached in batches through the first limited buffer pool of the preferred control node. Then, in the process of caching the data to be transmitted by the first limited buffer pool of the preferred control node, the preferred control node can monitor the cached data in the first limited buffer pool in real time to read, decrypt, and store the cached data in the first limited buffer pool one by one, and continuously write the cached data in the first limited buffer pool into the storage model corresponding to the preferred control node, thereby realizing the batch transmission of data for the data exchange task between the designated data processing node and the preferred control node.

[0075] S350, if the data exchange task is a data sending task, the data to be transmitted corresponding to the data exchange task is extracted through the preferred management and control node, and the data to be transmitted is sent in batches to the second limited cache pool of the designated data processing node, so that the cached data in the second limited cache pool is processed into the warehouse through the designated data processing node, thereby realizing the data transmission corresponding to the data exchange task.

[0076] If the data exchange task is a data delivery task, it means that the data exchange task can instruct the preferred control node to deliver the data to be transmitted to the designated data processing node. Then, after the preferred control node delivers the task execution information of the data exchange task to the designated data processing node, the preferred control node will continuously extract the corresponding data to be transmitted from the local according to the data identification information of the data to be transmitted after the data exchange task is parsed, and encrypt the data to be transmitted and continuously deliver it to the second limited buffer pool of the designated data processing node.

[0077] Among them, the preferred control node can adopt small batch and segmented data delivery method during the data delivery process until all the data to be transmitted are extracted in the total storage model. It can also send back the end delivery command of this data delivery task to the designated data processing node and destroy all kinds of resources involved in this data delivery.

[0078] Therefore, according to the upper capacity limit of the second limited cache pool of the designated data processing node, the preferred control node can send the encrypted data to be transmitted in batches to the designated data processing node, so as to cache the data to be transmitted sent by the preferred control node for the data exchange task in batches through the second limited cache pool of the designated data processing node. After receiving the task execution information of the data exchange task, the designated data processing node will prepare in advance to receive and cache the data to be transmitted sent by the preferred control node through the second limited cache pool of the designated data processing node according to the cache preparation indication information of the data to be transmitted corresponding to the data exchange task included in the task execution information, and monitor the cached data in the second limited cache pool in real time, so as to successfully write the cached data into the storage model corresponding to the designated data processing node. Therefore, in the process of caching the data to be transmitted by the second limited cache pool of the designated data processing node, the designated data processing node can monitor the cached data in the second limited cache pool in real time to perform corresponding reading, decryption, and storage processing on the cached data in the second limited cache pool one by one, and continuously write the cached data in the second limited cache pool into the data storage model corresponding to the designated data processing node, thereby realizing the batch transmission of data for the data exchange task between the preferred control node and the designated data processing node.

[0079] The technical solution provided by the embodiment of the present application, when detecting the execution instruction of any data exchange task defined in the control cluster, first determines the preferred control node and the designated data processing node corresponding to the data exchange task. Then, according to the type of the data exchange task, the data transmission corresponding to the data exchange task can be executed through the first limited cache pool of the preferred control node and one of the target limited cache pools in the second limited cache pool of the designated data processing node, thereby realizing cross-network and cross-regional data batch transmission through the limited cache pool without the need to transfer the data into files, thereby improving the efficiency and accuracy of data transmission while ensuring the security and reliability of data transmission.

[0080] Figure 4 This is a principle block diagram of a data transmission device shown in an embodiment of the present application. Figure 4 As shown, the apparatus 400 may include:

[0081] The node determination module 410 is used to determine the preferred control node and the designated data processing node corresponding to any data exchange task defined in the control cluster in response to the execution instruction of the data exchange task;

[0082] The data transmission module 420 is used to perform data transmission corresponding to the data exchange task through the first limited buffer pool of the preferred management and control node and the target limited buffer pool in the second limited buffer pool of the designated data processing node according to the type of the data exchange task.

[0083] In some implementations, the node determination module 410 may be specifically configured to:

[0084] According to the load information of each control node in the control cluster, the corresponding preferred control node is allocated to the data exchange task;

[0085] According to the transmission node information after parsing the data exchange task, a designated data processing node corresponding to the data exchange task is determined.

[0086] In some implementations, the data transmission module 420 may be specifically configured to:

[0087] If the data exchange task is a data reporting task, the data to be transmitted extracted by the designated data processing node for the data exchange task is cached in batches through the first limited cache pool of the preferred control node, and the cached data in the first limited cache pool is processed into a warehouse to realize the data transmission corresponding to the data exchange task;

[0088] If the data exchange task is a data sending task, the data to be transmitted corresponding to the data exchange task is extracted through the preferred management and control node, and the data to be transmitted is sent in batches to the second limited cache pool of the designated data processing node, so that the cached data in the second limited cache pool can be stored through the designated data processing node to realize the data transmission corresponding to the data exchange task.

[0089] In some implementations, the data transmission device 400 may further include:

[0090] The task execution information sending module is used to send the task execution information of the data exchange task to the designated data processing node through the preferred management and control node to instruct the designated data processing node to enter the transmission working state.

[0091] In some possible implementations, if the data exchange task is a data reporting task, the task execution information includes at least the node identifier of the preferred management and control node and the data identifier and local extraction indication information of the data to be transmitted corresponding to the data exchange task.

[0092] Correspondingly, if the data exchange task is a data delivery task, the task execution information at least includes cache preparation indication information of the to-be-transmitted data corresponding to the data exchange task.

[0093] In some implementations, the data transmission module 420 may also be specifically configured to:

[0094] The data transmission corresponding to the data exchange task is performed by adopting a transmission compensation mechanism through the first limited buffer pool of the preferred management and control node and the target limited buffer pool in the second limited buffer pool of the designated data processing node.

[0095] In the embodiment of the present application, when the execution instruction of any data exchange task defined in the control cluster is detected, the preferred control node and the designated data processing node corresponding to the data exchange task will be determined first. Then, according to the type of the data exchange task, the data transmission corresponding to the data exchange task can be executed through the first limited cache pool of the preferred control node and one of the target limited cache pools in the second limited cache pool of the designated data processing node, thereby realizing cross-network and cross-regional data batch transmission through the limited cache pool without the need to transfer the data into files, thereby improving the efficiency and accuracy of data transmission while ensuring the security and reliability of data transmission.

[0096] It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, they will not be described here. Specifically, Figure 4 The device 400 shown can execute any method embodiment in the present application, and the aforementioned and other operations and / or functions of each module in the device 400 are respectively for implementing the corresponding processes in each method in the embodiment of the present application. For the sake of brevity, they will not be repeated here.

[0097] The above describes the device 400 of the embodiment of the present application from the perspective of the functional module in conjunction with the accompanying drawings. It should be understood that the functional module can be implemented in hardware form, can be implemented by instructions in software form, and can also be implemented by a combination of hardware and software modules. Specifically, the steps of the method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and / or software form instructions in the processor, and the steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or a combination of hardware and software modules in the decoding processor to perform. Optionally, the software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory, and completes the steps in the above method embodiment in conjunction with its hardware.

[0098] Figure 5 A schematic block diagram of an electronic device according to an embodiment of the present application.

[0099] like Figure 5As shown, the electronic device 500 may include:

[0100] The memory 510 and the processor 520, the memory 510 is used to store the computer program and transmit the program code to the processor 520. In other words, the processor 520 can call and run the computer program from the memory 510 to implement the method in the embodiment of the present application.

[0101] For example, the processor 520 may be configured to execute the above method embodiments according to instructions in the computer program.

[0102] In some embodiments of the present application, the processor 520 may include but is not limited to:

[0103] General-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, etc.

[0104] In some embodiments of the present application, the memory 510 includes but is not limited to:

[0105] Volatile memory and / or non-volatile memory. Among them, the non-volatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM) or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0106] In some embodiments of the present application, the computer program may be divided into one or more modules, which are stored in the memory 510 and executed by the processor 520 to complete the method provided by the present application. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device.

[0107] like Figure 5 As shown, the electronic device may also include:

[0108] The transceiver 530 may be connected to the processor 520 or the memory 510 .

[0109] The processor 520 may control the transceiver 530 to communicate with other devices, specifically, to send information or data to other devices, or to receive information or data sent by other devices. The transceiver 530 may include a transmitter and a receiver. The transceiver 530 may further include an antenna, and the number of antennas may be one or more.

[0110] It should be understood that the various components in the electronic device are connected via a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus and a status signal bus.

[0111] The present application also provides a computer storage medium on which a computer program is stored, and when the computer program is executed by a computer, the computer can perform the method of the above method embodiment. In other words, the present application embodiment also provides a computer program product containing instructions, and when the instructions are executed by a computer, the computer can perform the method of the above method embodiment.

[0112] When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integration. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (digital video disc, DVD)), or a semiconductor medium (e.g., a solid state drive (solid state disk, SSD)), etc.

[0113] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0114] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the module is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0115] The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. For example, each functional module in each embodiment of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0116] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A data transmission method, characterized in that: include: In response to an execution instruction of any data exchange task defined in the management and control cluster, determine a preferred management and control node and a designated data processing node corresponding to the data exchange task; According to the type of the data exchange task, data transmission corresponding to the data exchange task is performed through the first limited buffer pool of the preferred management and control node and the target limited buffer pool in the second limited buffer pool of the designated data processing node.

2. The method according to claim 1, characterized in that The determining of the preferred management and control node and the designated data processing node corresponding to the data exchange task includes: According to the load information of each control node in the control cluster, the corresponding preferred control node is allocated to the data exchange task; According to the transmission node information after parsing the data exchange task, a designated data processing node corresponding to the data exchange task is determined.

3. The method according to claim 1, characterized in that The performing of data transmission corresponding to the data exchange task through the first limited buffer pool of the preferred management and control node and the target limited buffer pool in the second limited buffer pool of the designated data processing node according to the type of the data exchange task includes: If the data exchange task is a data reporting task, the data to be transmitted extracted by the designated data processing node for the data exchange task is cached in batches through the first limited cache pool of the preferred control node, and the cached data in the first limited cache pool is processed into a warehouse to realize the data transmission corresponding to the data exchange task; If the data exchange task is a data sending task, the data to be transmitted corresponding to the data exchange task is extracted through the preferred management and control node, and the data to be transmitted is sent in batches to the second limited cache pool of the designated data processing node, so that the cached data in the second limited cache pool can be stored through the designated data processing node to realize the data transmission corresponding to the data exchange task.

4. The method according to claim 1, characterized in that: After determining the preferred control node and the designated data processing node corresponding to the data exchange task, the method further includes: The task execution information of the data exchange task is sent to the designated data processing node through the preferred management and control node to instruct the designated data processing node to enter a transmission working state.

5. The method according to claim 4, characterized in that If the data exchange task is a data reporting task, the task execution information at least includes the node identifier of the preferred control node and the data identifier and local extraction indication information of the data to be transmitted corresponding to the data exchange task; If the data exchange task is a data delivery task, the task execution information at least includes cache preparation indication information of the to-be-transmitted data corresponding to the data exchange task.

6. The method according to claim 1, characterized in that The performing of data transmission corresponding to the data exchange task through the first limited buffer pool of the preferred management and control node and the target limited buffer pool in the second limited buffer pool of the designated data processing node specifically includes: The data transmission corresponding to the data exchange task is performed by adopting a transmission compensation mechanism through the first limited buffer pool of the preferred management and control node and the target limited buffer pool in the second limited buffer pool of the designated data processing node.

7. A data transmission device, characterized in that: include: A node determination module, configured to respond to an execution instruction of any data exchange task defined in the management and control cluster and determine a preferred management and control node and a designated data processing node corresponding to the data exchange task; The data transmission module is used to perform data transmission corresponding to the data exchange task through the first limited buffer pool of the preferred management and control node and the target limited buffer pool in the second limited buffer pool of the designated data processing node according to the type of the data exchange task.

8. An electronic device, characterized in that: include: A processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the data transmission method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: Used to store a computer program, wherein the computer program enables a computer to execute the data transmission method according to any one of claims 1 to 6.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the data transmission method according to any one of claims 1 to 6 is implemented.

Images