Data transmission method, device, equipment, medium and program product

Abstract

The invention provides a data transmission method which can be applied to the technical field of cloud computing. The method comprises the steps that in response to a data transmission instruction, a data transmission task is created, and a data transmission rule is defined in the data transmission task and used for obtaining target data from a target server side and transmitting the target data to a target user side; starting a data transmission task based on a task starting period set by the target data transmission instruction, and establishing an encrypted transmission channel between the target server side and the target user side; generating a data transmission script based on the target data transmission rule, limiting a transmission condition of the target data according to the data transmission script, and transmitting the target data through the encrypted transmission channel under the transmission condition; and in response to a data transmission completion instruction fed back by the target server, ending the data transmission task. The invention further provides a data transmission device, equipment, a storage medium and a program product.

Description

Technical Field

[0001] The present disclosure relates to the field of cloud computing, and in particular to a data transmission method, apparatus, device, medium and program product. Background Art

[0002] In the big data processing and cloud computing environment, with the explosive growth of data volume and the complexity of business, efficient and secure data transmission and management have become the core requirements of enterprise operations. Traditional file transfer methods such as FTP can no longer meet the high standards of modern enterprises for data real-time, integrity and security. At the same time, the automation and intelligence of business processes have also become key factors in improving corporate competitiveness. Therefore, the development of a comprehensive solution that integrates secure file transfer, automated task scheduling and efficient distributed file storage has become an urgent need in the current industry. Summary of the invention

[0003] In view of the above problems, the present disclosure provides a data transmission method, apparatus, device, medium and program product for improving the security of data transmission.

[0004] According to the first aspect of the present disclosure, a data transmission method is provided, the method comprising: creating a data transmission task in response to a data transmission instruction, wherein the data transmission task defines a data transmission rule for acquiring target data from a target server and transmitting it to a target user; starting the data transmission task based on a task start cycle set by the target data transmission instruction, and establishing an encrypted transmission channel between the target server and the target user; generating a data transmission script based on the target data transmission rule, limiting the transmission conditions of the target data according to the data transmission script, and transmitting the target data through the encrypted transmission channel under the transmission conditions; and ending the data transmission task in response to a data transmission completion instruction fed back by the target server.

[0005] According to an embodiment of the present disclosure, the creation of a data transmission task in response to a data transmission instruction from a front end includes: receiving the data transmission instruction through an interactive interface of the front end; generating a task template on the interactive interface in response to receiving the data transmission instruction, the task template being used to define the data transmission task; obtaining multiple task parameters in the data transmission instruction, and obtaining task transmission configuration information based on each of the task parameters, the task transmission configuration information including the data transmission rules and the task start-up cycle; and inputting the task transmission configuration information into the task template to generate the data transmission task.

[0006] According to an embodiment of the present disclosure, the task start cycle set based on the target data transmission instruction starts the data transmission task and establishes an encrypted transmission channel between the target server and the target user terminal, including: responding to the task start instruction, starting a task timer, the task timer is used to generate a task execution instruction for building the encrypted transmission channel when the start time of the task start cycle is reached; responding to the task execution instruction, obtaining two encrypted ports between the target server and the target user terminal, communicating between the two encrypted ports based on a preset encryption protocol, and forming the encrypted transmission channel.

[0007] According to an embodiment of the present disclosure, the target data transmission rules include a first data transmission script, a second data transmission script and a third data transmission script, the first data transmission script is used to specify the transmission form of the target data during the transmission process, the second data transmission script is used to verify the integrity of the target data after the target data reaches the target client, and the third data transmission script is used to specify the storage rules of the target data at the target end; the data transmission script is generated based on the target data transmission rules, the transmission conditions of the target data are restricted according to the data transmission script, and the target data is transmitted through the encrypted transmission channel under the transmission conditions, including: according to the first data transmission script, the target data is transmitted in the form of a compressed file; after the target client obtains the compressed file, according to the second data script, the integrity of the target data obtained after decompression is verified; when the target data is complete, the target data is stored in a database based on the format and storage path specified by the third data script.

[0008] According to an embodiment of the present disclosure, after the target client obtains the target data, the method further includes: generating a target metadata node and at least one data block of the target data in a storage module, storing the target data in the at least one data block, and associating the at least one data block with the target metadata node; and copying and associating the at least one data block to at least one other metadata node of the storage module.

[0009] According to an embodiment of the present disclosure, it also includes: when there is a data block on the target metadata node that cannot be read, obtaining the target data from at least one other metadata node associated with the data block.

[0010] 7. The second aspect of the present disclosure provides a data transmission device, which includes: a task creation module, which is used to create a data transmission task in response to a data transmission instruction, and the data transmission task defines a data transmission rule for obtaining target data from a target server and transmitting it to a target user; a channel creation module, which is used to start the data transmission task based on the task start cycle set by the target data transmission instruction, and establish an encrypted transmission channel between the target server and the target user; a data transmission module, which is used to generate a data transmission script based on the target data transmission rule, limit the transmission conditions of the target data according to the data transmission script, and transmit the target data through the encrypted transmission channel under the transmission conditions; a task recovery module, which is used to end the data transmission task in response to a data transmission completion instruction fed back by the target server.

[0011] A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0012] The fourth aspect of the present disclosure further provides a computer-readable storage medium having a computer program or instructions stored thereon, which implements the steps of the above method when the above computer program or instructions are executed by a processor.

[0013] The fifth aspect of the present disclosure further provides a computer program product, including a computer program or instructions, which implement the steps of the above method when the above computer program or instructions are executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0015] Figure 1 The application scenario diagram of the data transmission method and device according to the embodiment of the present disclosure is schematically shown;

[0016] Figure 2 A flowchart of a data transmission method according to an embodiment of the present disclosure is schematically shown;

[0017] Figure 3 A structural block diagram of a data transmission device according to an embodiment of the present disclosure is schematically shown; and

[0018] Figure 4 A block diagram of an electronic device suitable for implementing a data transmission method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0019] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0020] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0021] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0022] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0023] It should be noted that the data transmission method and device provided by the present disclosure can be used for cloud computing data transmission in the financial field, and can also be used in any field other than the financial field. The application field of the data transmission method and device provided by the present disclosure is not limited.

[0024] The acquisition, transmission, storage, use, and processing of data in the technical solution of this application are in compliance with the relevant provisions of national laws and regulations. It should be noted that in the embodiments of this application, some existing solutions in the industry such as certain software, components, and models may be mentioned, and they should be considered as exemplary. Their purpose is only to illustrate the feasibility of the implementation of the technical solution of this application, but it does not mean that the applicant has or will necessarily use the solution.

[0025] The embodiment of the present disclosure provides a data transmission method, which includes: creating a data transmission task in response to a data transmission instruction, wherein a data transmission rule is defined in the data transmission task, and is used to obtain target data from a target server and transmit it to a target user; starting the data transmission task based on a task start cycle set by the target data transmission instruction, and establishing an encrypted transmission channel between the target server and the target user; generating a data transmission script based on the target data transmission rule, limiting the transmission conditions of the target data according to the data transmission script, and transmitting the target data through an encrypted transmission channel under the transmission conditions; and ending the data transmission task in response to a data transmission completion instruction fed back by the target server. The method provides a comprehensive solution integrating secure file transfer, automated task scheduling, and distributed file storage.

[0026] Figure 1 The application scenario diagram of the data transmission method and device according to the embodiments of the present disclosure is schematically shown.

[0027] like Figure 1 As shown, the application scenario 100 according to this embodiment may include the application of the financial field in cloud computing data transmission. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0028] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0029] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0030] The server 105 may be a server that provides various services, such as providing cloud data support for users to use the first terminal device 101, the second terminal device 102, and the third terminal device 103. The server 105 may respond to the data requests of the first terminal device 101, the second terminal device 102, and the third terminal device 103, and feed back the requested data to the terminal device.

[0031] It should be noted that the data transmission method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the data transmission device provided in the embodiment of the present disclosure can generally be set in the server 105. The data transmission method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the data transmission device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0032] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0033] The following will be based on Figure 1 The scene described by Figure 2 The data transmission method of the disclosed embodiment is described in detail.

[0034] Figure 2 The flowchart of the data transmission method according to the embodiment of the present disclosure is schematically shown.

[0035] like Figure 2 As shown, the data transmission method of this embodiment includes operations S210 to S240, and the transaction processing method can be executed sequentially.

[0036] In operation S210, in response to a data transmission instruction, a data transmission task is created, wherein a data transmission rule is defined in the data transmission task for acquiring target data from a target server and transmitting the target data to a target user.

[0037] Operation S210 may include S211 to S214.

[0038] In operation S211, a data transmission instruction is received through the front-end interactive interface.

[0039] The front-end interactive interface is deployed on the target server or a server cluster that establishes communication with the target server. The interactive interface can be provided to the user in the form of a front-end interface such as a web page or an application. The user can input or select a data transmission instruction through the interactive interface. The data transmission instruction may include basic information such as the type of data to be transmitted, the target location, and the transmission method.

[0040] For example, a user opens an application called "Data Management Center" and clicks the "New Transfer Task" button in the "Data Transfer" module to enter a form page. On this page, the user fills in the location of the source data (such as database A), the location of the target data (such as database B), and selects the transfer method (such as real-time synchronization or scheduled synchronization).

[0041] In operation S212, in response to receiving the data transmission instruction, a task template is generated on the interactive interface, where the task template is used to define the data transmission task.

[0042] When the interactive interface system receives data transfer instructions submitted by the user, it automatically generates a task template based on these instructions. A task template is a framework or structure that defines the specific details of a data transfer task, but at this point, specific task parameters or configurations have not yet been filled in.

[0043] For example, the system to which the interactive interface belongs generates a task template in the background based on the user's selection (such as source data location, target data location, transmission method, etc.). This template contains the basic structure required for the data transmission task, such as the target client ID, target server ID, data content to be obtained, data reading module to be accessed, data processing module to be accessed, data writing module to be accessed, task execution cycle, task start time, etc., but the specific configuration of each module (such as database connection information, data screening conditions, etc.) has not yet been determined.

[0044] In operation S213, a plurality of task parameters in the data transmission instruction are obtained, and task transmission configuration information is obtained based on each task parameter. The task transmission configuration information includes a data transmission rule and a task start cycle.

[0045] The system to which the interactive interface belongs extracts specific task parameters (such as target client ID, target server ID, database user name, password, port number, table name to be transferred, etc.) from the data transmission instructions submitted by the user, and further obtains or calculates the configuration information of the task transmission based on these parameters, including specific rules for data transmission (such as data filtering conditions, data conversion rules, etc.) and the task startup cycle (such as execution at 1 a.m. every day, once an hour, etc.).

[0046] For example, the system extracts the connection information of database A and database B (including user name, password, port number, etc.) and the table name to be transferred from the form filled out by the user. Then, the system generates the task transfer configuration information based on this information, including specifying that only records with "sales" greater than 1,000 in the "sales data" table will be transferred, and setting the task start cycle to execute once a day at 2 a.m.

[0047] In operation S214, the task transmission configuration information is input into the task template to generate a data transmission task.

[0048] The task transmission configuration information acquired in step S213 is filled into the task template generated in step S212, thereby generating a complete data transmission task. This task contains all the details of the data transmission and can be directly executed.

[0049] For example, the system fills the task transfer configuration information (such as database connection information, data screening conditions, data conversion rules, task start cycle, etc.) into the task template and generates a data transfer task named "Sales Data Synchronization Task". This task is configured to filter out records with "Sales" greater than 1,000 from the "Sales Data" table of database A at 2 a.m. every day, and synchronize these records to the corresponding table of database B. After the task is generated, users can view, edit or start the task.

[0050] In operation S220, the data transmission task is started based on the task start period set by the target data transmission instruction, and an encrypted transmission channel is established between the target server and the target user.

[0051] The task start cycle is the time and start cycle of the task start. For example, you can set it to start a data transmission task at 1:00 a.m. every day, or once on Monday, Wednesday, and Friday.

[0052] The encrypted transmission channel is implemented based on SFTP technology. SFTP is a secure file transfer protocol based on the SSH (Secure Shell) protocol. It ensures the security and integrity of data during transmission through encryption and authentication mechanisms. SFTP not only inherits the functions of FTP (File Transfer Protocol), but also encrypts all transmitted data through the SSH protocol, thus avoiding the risk of data leakage or tampering.

[0053] Operation S220 includes S221-S222.

[0054] In operation S221 , in response to a task start instruction, a task timer is started, and the task timer is used to generate a task execution instruction for constructing an encrypted transmission channel when a start time of a task start cycle is reached.

[0055] In the management process of data transmission tasks, when the system receives a task start instruction, it will not immediately execute the data transmission operation, but first start a task timer. The purpose of this timer is to wait until the start time of the preset task start cycle is reached. Once this time point is reached, the timer will trigger an event and generate a task execution instruction to build an encrypted transmission channel.

[0056] For example, suppose there is a task called "Daily Financial Data Transfer" whose start period is set to 1:00 a.m. every day. When the user clicks the "Start Task" button on the front-end interface, the system will record this instruction and start a task timer. The timer will run until 1:00 a.m. the next day. Once this time point is reached, the timer will generate a task execution instruction, instructing the system to start building an encrypted transmission channel for the transmission of financial data.

[0057] In operation S222, in response to the task execution instruction, two encryption ports between the target server and the target client are acquired, and communication between the two encryption ports is performed based on a preset encryption protocol to form an encrypted transmission channel.

[0058] In the disclosed embodiment, a server software supporting SFTP, such as OpenSSH, can be installed on the target server; relevant parameters of the SFTP server, including port number, user permissions, file access restrictions, etc., can be configured; an SSH key pair (public key and private key) can be generated for the SFTP server, and the public key can be placed on the server, while the private key can be distributed to legitimate target clients. SFTP client software, such as WinSCP, FileZilla, etc., can be installed on the target client; the target client can be configured to connect to the target server using the SFTP protocol, and the host name, port number, and private key file of the target server can be specified. When the target client attempts to connect to the target server, the SFTP protocol uses the SSH protocol for handshake and identity authentication. The target client provides a private key for authentication, and the target server verifies the validity of the private key. Once the authentication is successful, SFTP will establish an encrypted communication channel between the target client and the target server.

[0059] For example, during the execution of the "Daily Financial Data Transfer" task, after receiving the task execution instruction, the system will first determine the encryption ports of the target server (such as the SFTP server in the data center) and the target client (such as the SFTP client in a branch office). Assume that the encryption port of the server is 22 (the default port of the SSH protocol), and the encryption port of the client is also 22 (or configured as another port, but the same port is used here to simplify the explanation). Then, the system will use the SSH protocol to establish encrypted communication between the two ports, thus forming an encrypted transmission channel. Once the channel is successfully established, the system can securely transmit financial data through this channel. Through the above steps, enterprises can ensure the security and integrity of sales data during transmission, while realizing automated data transmission and management.

[0060] In operation S230, a data transmission script is generated based on the target data transmission rule, a transmission condition of the target data is restricted according to the data transmission script, and the target data is transmitted through the encrypted transmission channel under the transmission condition.

[0061] In the automation process of data transfer, generating a data transfer script based on the target data transfer rules is a key step. This script defines how to select and limit the target data, and how to securely transfer this data through an encrypted transmission channel. The target data transfer rules may include the type, source, destination, transfer time window, data size limit, data format requirements, etc. of the data. For example, the rules may specify that only certain types of files (such as financial statements in CSV format) are transferred, data is only read from a specific source directory, and the transfer is only performed during a specific time period every day.

[0062] Based on the defined target data transfer rules, a data transfer script can be generated. This script is usually an automated program or script file that contains all the logic and instructions required to perform the data transfer task. The script may be written in various programming languages, such as Bash, Python, PowerShell, etc., depending on the system requirements and the developer's preferences. Based on the transfer rules, the script will first determine which data is the target data to be transferred. This may involve operations such as traversing the file system, matching file names, and checking file contents.

[0063] Before transmission, some preprocessing of the data may be required, such as compression, encryption (although encryption is usually performed at the transmission channel level, in some cases additional encryption may be required at the data level), format conversion, etc. The script will contain the logic for transmitting data through an encrypted transmission channel. This usually involves communication with the target server, including authentication, port connection, data transmission and other steps. In order to ensure the reliability and traceability of data transmission, the script needs to include an error handling mechanism and record detailed log information during execution.

[0064] In the data transfer script, it is necessary to clearly limit the transfer conditions of the target data. These conditions can be based on the specific implementation of the transfer rules, or they can be additional security or business logic requirements. For example, the script can check whether the data meets specific data integrity checks (such as hash value matching) or whether the latest version already exists at the target location, thereby avoiding unnecessary data transfer.

[0065] Once the data transfer script is prepared and the target data has been restricted and prepared according to the transfer conditions, the data can be transferred through the encrypted transmission channel. During this process, the script will communicate with the SFTP server (or other SSH-based encryption service) on the target server, establish an encrypted connection, and transfer data. Since the SFTP protocol itself is based on the SSH protocol, the data will be encrypted during the transmission process, thus ensuring the confidentiality and integrity of the data.

[0066] For example, suppose there is a task called "Monthly Sales Report Transfer" whose goal is to transfer the latest sales report from the servers of various branches to the data center of the headquarters every day. Based on this goal, a transfer rule can be defined, such as only transferring files named "sales_report_YYYY_MM.csv" and only transferring on the first day of each month. Then, a data transfer script can be generated, which traverses the specified directories on the servers of the branches, finds the sales report files that meet the naming rules, and transfers them to the data center of the headquarters via the SFTP protocol. Before the transfer, the script may check whether the file already exists at the target location, and may calculate the hash value of the file to ensure the integrity of the data. During the transfer process, the SFTP protocol ensures that the data is transmitted through an encrypted channel, thereby preventing the risk of data being stolen or tampered with during the transfer process. Finally, the script records a detailed transfer log for subsequent auditing and troubleshooting.

[0067] In some embodiments of the present disclosure, the target data transmission rule includes a first data transmission script, a second data transmission script, and a third data transmission script, the first data transmission script is used to specify the transmission form of the target data during the transmission process, the second data transmission script is used to perform integrity verification of the target data after the target data reaches the target client, and the third data transmission script is used to specify the storage rules of the target data at the target end. Operation S230 includes S231~S233.

[0068] In operation S231, target data is transmitted in the form of a compressed file according to the first data transmission script.

[0069] The first data transfer script defines how the target data should be prepared before transmission and in what form it should be transmitted. In this example, the script specifies that the target data should be transmitted in the form of a compressed file. Compressing files not only reduces the time and bandwidth required for the transfer, but also increases the security of the data during transmission (although compression itself does not provide encryption).

[0070] For example, if there is a batch of sales data to be transferred, the first data transfer script will instruct the system to package this batch of data (which may be multiple CSV files or database export files) into a compressed file in ZIP or TAR.GZ format. Then, this compressed file will be sent to the target client for further processing.

[0071] In operation S232, after the target client obtains the compressed file, the integrity of the target data obtained after decompression is verified according to the second data script.

[0072] The second data transfer script defines how to receive and verify the integrity of the compressed file at the target client. This usually involves checking whether the compressed file arrived intact and whether the decompressed data is consistent with the original data. The verification process may include calculating the hash value of the file (such as MD5, SHA-256, etc.) and comparing it with the hash value of the original target data.

[0073] For example, when a compressed file arrives at the destination client, the second data transfer script is automatically executed. The script first decompresses the file and then calculates a hash value for the decompressed data. It then compares this hash value with the original data (or a hash value that was previously stored in a secure location). If the two match, the data is intact; if they do not match, the data may need to be retransmitted or other remedial measures may need to be taken.

[0074] In operation S233, when the target data is complete, the target data is stored in the database based on the format and storage path specified by the third data script.

[0075] The third data transfer script defines how the target data should be stored in the target database after passing integrity verification. This includes specifying the data storage format (such as database table structure), storage path (i.e., which database and table the data should be inserted into), and any necessary transformation or cleaning operations.

[0076] For example, when the target data passes the integrity verification, the third data transfer script will take over. The script will import the decompressed data into the target database according to the preset database table structure. This may involve converting the data from a CSV file or other format to a format that the database can recognize and inserting it into the corresponding table. For example, if the target database is a relational database, the script may use SQL statements to insert the data into the corresponding table and may perform some data cleaning operations to ensure the consistency and accuracy of the data.

[0077] Based on the first data transmission script, the second data transmission script and the third data transmission script, it can be ensured that the target data can be safely and completely transmitted to the target client and stored in the target database according to the predetermined rules. This process is achieved through three carefully designed transmission scripts, each of which is responsible for a key link in the data transmission process.

[0078] In operation S240, in response to a data transmission completion instruction fed back by the target server, the data transmission task is ended.

[0079] After receiving the data transmission completion instruction from the target server, the system to which the interactive interface belongs officially ends the current data transmission task. This step is an important part of the data transmission process because it ensures the integrity and timeliness of data transmission and allows the system to release related resources for subsequent operations.

[0080] In the disclosed embodiment, the system needs to continuously monitor the communication from the target server, especially the feedback about the data transmission status. This is usually implemented through a network protocol (such as HTTP, TCP / IP, etc.), in which the target server sends a specific feedback instruction after the data is received. After receiving the feedback instruction, the system needs to verify the authenticity and validity of the instruction. This can be done by checking the format, signature, timestamp of the instruction or in combination with other verification mechanisms to ensure that the feedback instruction is not forged or repeated. Once the validity of the feedback instruction is verified, the system triggers the logic to end the data transmission task. This may include closing the network connection, releasing the occupied memory and processor resources, updating the task status, etc. After the task is ended, the system may send a notification to the user or administrator to inform that the data transmission task has been completed. At the same time, the system will also record relevant log information for subsequent auditing or troubleshooting.

[0081] According to the data transmission method provided by the embodiment of the present disclosure, after the target client obtains the target data, the method may also include: generating a target metadata node and at least one data block of the target data in a storage module, storing the target data in at least one data block, and associating at least one data block with the target metadata node; and copying and associating at least one data block to at least one other metadata node of the storage module.

[0082] In the disclosed embodiment, in order to ensure high availability and fault tolerance of data, the storage architecture of the target data is HDFS, which uses a combination of metadata nodes and data blocks to manage and store data. This architecture allows data to be stored in multiple physical locations, while tracking the storage location and status of the data through metadata nodes. When the target data needs to be stored, the system first generates one or more target metadata nodes in the storage module. These metadata nodes are used to store metadata about the data blocks, such as the location, size, status (whether readable), etc. of the data blocks. At the same time, the system will split the target data into at least one data block. The size of the data block can be adjusted according to the system configuration, data type, and storage requirements.

[0083] To improve data reliability and fault tolerance, the system replicates data blocks to at least one other metadata node in the storage module. This means that each data block will have multiple copies, which can be located on different physical devices or geographical locations. The replication process can be synchronous (i.e., waiting for all copies to be successfully written before confirming the success of the write) or asynchronous (i.e., writing to one copy first, and then asynchronously replicating to other copies).

[0084] When a data block exists on a target metadata node and cannot be read, the target data is obtained from at least one other metadata node associated with the data block.

[0085] In the disclosed embodiment, when the target data needs to be read, the system will first try to read from the data block associated with the target metadata node. If it is found that a data block cannot be read (for example, due to hardware failure, network problems, or data corruption), the system will try to read the data from the replicas on other metadata nodes associated with the data block. If all replicas cannot be read, the system may take further recovery measures, such as restoring the data block from a backup, or marking the data block as unavailable and triggering a data repair process.

[0086] For example, a user uploads a file named "report.docx". The system splits the file into three data blocks (Block1, Block2, Block3) and generates unique identifiers for them. The system creates a target metadata node (MetadataNode1) in the storage module and associates the metadata of the three data blocks (including their identifiers and locations) with the metadata node. Each data block is replicated to two other metadata nodes in the storage module (for example, Block1 is replicated to MetadataNode2 and MetadataNode3). When a user requests to read the "report.docx" file, the system first queries MetadataNode1 to obtain the location of the data block. The system tries to read the file from the data block associated with MetadataNode1. If everything goes well, it reads Block1, Block2, and Block3 in sequence and reassembles them into the original file. Suppose during the reading process, the system finds that Block2 cannot be read on MetadataNode1. At this time, the system will try to read the data from Block2's replica on MetadataNode2 or MetadataNode3 instead.

[0087] If all replicas cannot read Block 2, the system may restore the data block from the backup, or trigger a data repair process to try to recover or regenerate the lost data block. At the same time, the system may record this event and take measures to reduce the occurrence of similar problems during subsequent system maintenance or optimization.

[0088] This paper proposes a comprehensive solution that integrates secure file transfer, automated task scheduling, and distributed file storage, aiming to build a secure, efficient, and flexible data processing framework. The core of this technical solution is to achieve end-to-end encrypted transmission through SFTP to ensure the security of the data transmission process; to achieve automated task management through an interactive interface to improve the automation level and efficiency of data processing; HDFS provides high fault tolerance and high throughput data storage capabilities, supporting the storage and access of PB-level data volumes.

[0089] Based on the above data transmission method, the present disclosure also provides a data transmission device. Figure 3 The device is described in detail.

[0090] Figure 3 The structural block diagram of the data transmission device according to an embodiment of the present disclosure is schematically shown.

[0091] like Figure 3As shown, the data transmission device 300 of this embodiment includes a task creation module 310 , a channel creation module 320 , a data transmission module 330 and a task recycling module 340 .

[0092] The task creation module 310 is used to create a data transmission task in response to the data transmission instruction. The data transmission task defines a data transmission rule for obtaining target data from the target server and transmitting it to the target user. In one embodiment, the task creation module 310 can be used to perform the operation S210 described above, which will not be repeated here.

[0093] The channel creation module 320 is used to start the data transmission task based on the task start cycle set by the target data transmission instruction, and establish an encrypted transmission channel between the target server and the target client. In one embodiment, the channel creation module 320 can be used to perform the operation S220 described above, which will not be repeated here.

[0094] The data transmission module 330 is used to generate a data transmission script based on the target data transmission rule, limit the transmission conditions of the target data according to the data transmission script, and transmit the target data through the encrypted transmission channel under the transmission conditions. In one embodiment, the data transmission module 330 can be used to perform the operation S230 described above, which will not be repeated here.

[0095] The task recovery module 340 is used to respond to the data transmission completion instruction fed back by the target server to end the data transmission task. In one embodiment, the task recovery module 340 can be used to perform the operation S240 described above, which will not be described in detail here.

[0096] According to an embodiment of the present disclosure, any multiple modules of the task creation module 310, the channel creation module 320, the data transmission module 330 and the task recovery module 340 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the task creation module 310, the channel creation module 320, the data transmission module 330 and the task recovery module 340 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in a suitable combination of any of them. Alternatively, at least one of the task creation module 310 , the channel creation module 320 , the data transmission module 330 and the task recycling module 340 may be at least partially implemented as a computer program module, which may perform corresponding functions when executed.

[0097] Figure 4 A block diagram of an electronic device suitable for implementing a data transmission method according to an embodiment of the present disclosure is schematically shown.

[0098] like Figure 4 As shown, the electronic device 400 according to an embodiment of the present disclosure includes a processor 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage part 408 to a random access memory (RAM) 403. The processor 401 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a dedicated microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 401 may also include an onboard memory for caching purposes. The processor 401 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0099] In RAM 403, various programs and data required for the operation of electronic device 400 are stored. Processor 401, ROM 402 and RAM 403 are connected to each other via bus 404. Processor 401 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 402 and / or RAM 403. It should be noted that the program can also be stored in one or more memories other than ROM 402 and RAM 403. Processor 401 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.

[0100] According to an embodiment of the present disclosure, the electronic device 400 may further include an input / output (I / O) interface 405, which is also connected to the bus 404. The electronic device 400 may further include one or more of the following components connected to the input / output (I / O) interface 405: an input portion 406 including a keyboard, a mouse, etc.; an output portion 407 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 408 including a hard disk, etc.; and a communication portion 409 including a network interface card such as a LAN card, a modem, etc. The communication portion 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the input / output (I / O) interface 405 as needed. A removable medium 411, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 410 as needed, so that a computer program read therefrom is installed into the storage portion 408 as needed.

[0101] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0102] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 402 and / or RAM 403 described above and / or one or more memories other than ROM 402 and RAM 403.

[0103] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the data transmission method provided by the embodiment of the present disclosure.

[0104] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 401. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0105] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 409, and / or installed from the removable medium 411. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0106] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 409, and / or installed from the removable medium 411. When the computer program is executed by the processor 401, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, apparatus, module, unit, etc. described above can be implemented by a computer program module.

[0107] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0108] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0109] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations or combinations are not explicitly described in the present disclosure. In particular, without departing from the spirit and teachings of the present disclosure, the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0110] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A data transmission method, characterized in that: The method comprises: In response to the data transmission instruction, a data transmission task is created, wherein the data transmission task defines a data transmission rule for acquiring target data from a target server and transmitting the target data to a target user; Starting the data transmission task based on the task start cycle set by the target data transmission instruction, and establishing an encrypted transmission channel between the target service end and the target user end; Generate a data transmission script based on the target data transmission rule, restrict the transmission conditions of the target data according to the data transmission script, and transmit the target data through the encrypted transmission channel under the transmission conditions; In response to a data transmission completion instruction fed back by the target server, the data transmission task is terminated.

2. The method according to claim 1, characterized in that In response to the data transmission instruction, creating a data transmission task includes: Receiving the data transmission instruction through the front-end interactive interface; In response to receiving the data transmission instruction, generating a task template on the interactive interface, wherein the task template is used to define the data transmission task; Acquire multiple task parameters in the data transmission instruction, and acquire task transmission configuration information based on each of the task parameters, wherein the task transmission configuration information includes the data transmission rule and the task start cycle; The task transmission configuration information is input into the task template to generate the data transmission task.

3. The method according to claim 1, characterized in that The starting of the data transmission task based on the task starting cycle set by the target data transmission instruction and the establishment of an encrypted transmission channel between the target server and the target user include: In response to the task start instruction, start a task timer, wherein the task timer is used to generate a task execution instruction for constructing the encrypted transmission channel when the start time of the task start cycle is reached; In response to the task execution instruction, two encrypted ports between the target service end and the target user end are acquired, and communication between the two encrypted ports is performed based on a preset encryption protocol to form the encrypted transmission channel.

4. The method according to claim 1, characterized in that: The target data transmission rule includes a first data transmission script, a second data transmission script and a third data transmission script, wherein the first data transmission script is used to specify the transmission form of the target data during the transmission process, the second data transmission script is used to perform integrity verification of the target data after the target data arrives at the target client, and the third data transmission script is used to specify the storage rules of the target data at the target end; Generating a data transmission script based on the target data transmission rule, limiting the transmission conditions of the target data according to the data transmission script, and transmitting the target data through the encrypted transmission channel under the transmission conditions includes: According to the first data transmission script, the target data is transmitted in the form of a compressed file; After the target client obtains the compressed file, verifying the integrity of the target data obtained after decompression according to the second data script; When the target data is complete, the target data is stored in a database based on the format and storage path specified by the third data script.

5. The method according to claim 1, characterized in that After the target client obtains the target data, the method further includes: Generating a target metadata node and at least one data block of the target data in a storage module, storing the target data in the at least one data block, wherein the at least one data block is associated with the target metadata node; The at least one data block is copied and associated to at least one other metadata node of the storage module.

6. The method according to claim 5, characterized in that Also includes: When there is a data block in the target metadata node that cannot be read, the target data is obtained from at least one other metadata node associated with the data block.

7. A data transmission device, characterized in that: The device comprises: A task creation module, for creating a data transmission task in response to a data transmission instruction, wherein the data transmission task defines a data transmission rule for acquiring target data from a target server and transmitting the target data to a target user; A channel creation module, used to start the data transmission task based on the task start cycle set by the target data transmission instruction, and establish an encrypted transmission channel between the target service end and the target user end; A data transmission module, configured to generate a data transmission script based on the target data transmission rule, restrict the transmission conditions of the target data according to the data transmission script, and transmit the target data through the encrypted transmission channel under the transmission conditions; The task recovery module is used to terminate the data transmission task in response to the data transmission completion instruction fed back by the target server.

8. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images