File transmission method and device, equipment and storage medium

Abstract

The invention discloses a file transmission method and device, equipment and a storage medium, and relates to the technical field of data transmission, and discloses the steps that a target file uploaded by a user is acquired, a to-be-transmitted task is generated according to the target file, and the to-be-transmitted task comprises a to-be-transmitted file, target node information and a transmission mode; when the idle thread exists, performing target node election based on the target node information to obtain a target node; sending the to-be-transmitted file to a target node based on the transmission mode, and calling a target node interface to perform file verification; when the file verification succeeds, the to-be-transmitted task is completed; according to the method, a transmission task is directly generated for a to-be-transmitted file uploaded by a user, after a target node is determined based on the file uploaded by the user and configured target node information, point-to-point asynchronous file transmission between a local node and the target node is carried out based on the transmission task and an idle thread, and the file transmission efficiency is improved. The time for waiting for file transmission and thread resources can be saved, and the file transmission efficiency is improved.

Description

Technical Field

[0001] The present application relates to the technical field of data transmission, and in particular to a file transmission method, device, equipment and storage medium. Background Art

[0002] At present, when uploading and downloading data files, not only a large number of database connections are occupied, but also the server memory and CPU resources deployed by the business system application are occupied. In the prior art, data is processed in a synchronous manner. When the amount of data is large, users need to wait for a long time for the system to complete processing. When a large number of users operate the system at the same time, multiple upload and download processing threads occupy a large number of server resources, which will affect the response speed and stability of the business system.

[0003] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0004] The main purpose of the present application is to provide a file transfer method, device, equipment and storage medium, aiming to solve the current technical problem that file transfer requires synchronous calls and wastes thread resources.

[0005] To achieve the above object, the present application proposes a file transmission method, which includes:

[0006] Obtain the target file uploaded by the user, and generate a task to be transmitted according to the target file, wherein the task to be transmitted includes the file to be transmitted, target node information, and transmission mode;

[0007] When there is an idle thread, a target node is selected based on the target node information to obtain the target node;

[0008] Sending the file to be transmitted to the target node based on the transmission mode, and calling the target node interface to perform file verification;

[0009] When the file verification succeeds, the task to be transmitted is completed.

[0010] In one embodiment, when there is an idle thread, performing target node election based on the target node information to obtain the target node includes:

[0011] When there are idle threads, determining a target cluster based on the target node information;

[0012] Performing random hashing screening on multiple nodes in the target cluster to obtain an initial node;

[0013] When the initial node is in a connectable state, the initial node is used as a target node.

[0014] In one embodiment, sending the file to be transmitted to the target node based on the transmission mode, and calling the target node interface to perform file verification, includes:

[0015] Sending the file to be transmitted to the target node based on the transmission mode, and obtaining a user-specified refresh threshold;

[0016] Obtaining a refresh point based on the user-specified refresh threshold, and identifying a file transfer progress based on the refresh point;

[0017] When the file transfer progress is complete, the target node interface is called to perform file verification.

[0018] In one embodiment, before sending the file to be transmitted to the target node based on the transmission mode and calling the target node interface to perform file verification, the method further includes:

[0019] Get the authentication information of the locally configured target node;

[0020] A file transfer connection is established with the target node based on a preset communication protocol according to the target node authentication information, so as to send the file to be transferred to the target node based on the file transfer connection.

[0021] In one embodiment, calling the target node interface to perform file verification includes:

[0022] When the file transfer progress is complete, calling the target node interface to identify the file to be transferred;

[0023] Classifying the files to be transmitted according to the sizes of the files to be transmitted to obtain file categories;

[0024] A file summary of the file to be transmitted is extracted based on the file category, and verification is performed based on the file summary.

[0025] In one embodiment, extracting a file digest of the file to be transmitted based on the file category, and verifying based on the file digest includes:

[0026] When the file category is a type one file, obtaining a file summary according to the file to be transmitted, and performing verification based on the file summary;

[0027] When the file category is a Class II file, data sampling is performed on the file to be transmitted to obtain sample data, sample summaries of each sample data are extracted respectively, and each sample summary is merged in sequence to obtain a file summary and a sample merge position; verification is performed based on the file summary and the sample merge position.

[0028] In one embodiment, the task to be transmitted further includes a callback name;

[0029] When the file verification succeeds, after the task to be transmitted is completed, the method further includes:

[0030] Get callback configuration according to callback name;

[0031] Generate a splicing message according to the task to be transmitted and the callback configuration;

[0032] The spliced ​​message is called back according to a preset number of callbacks.

[0033] In addition, to achieve the above-mentioned purpose, the present application also proposes a file transmission device, the file transmission device comprising:

[0034] An acquisition module is used to acquire a target file uploaded by a user and generate a task to be transmitted according to the target file. The task to be transmitted includes the file to be transmitted, target node information and transmission mode;

[0035] A node election module, used for, when there is an idle thread, performing target node election based on the target node information to obtain a target node;

[0036] A file transmission module, used for sending the file to be transmitted to the target node based on the transmission mode, and calling the target node interface to perform file verification;

[0037] The file transmission module is further used to complete the task to be transmitted when the file verification succeeds.

[0038] In addition, to achieve the above objectives, the present application also proposes a file transfer device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the file transfer method described above.

[0039] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the file transfer method described above are implemented.

[0040] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the file transfer method described above are implemented.

[0041] One or more technical solutions proposed in this application have at least the following technical effects:

[0042] By directly generating transmission tasks for the files to be transferred uploaded by users, and determining the target node based on the files uploaded by users and the configured target node information, point-to-point asynchronous file transfer is performed between the local and target nodes based on the transmission tasks and idle threads. This can save time and thread resources waiting for file transfer, improve file transfer efficiency, and ensure the stability of data transmission. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0044] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0045] Figure 1 A flowchart of the first embodiment of the file transmission method of the present application is provided;

[0046] Figure 2 A schematic diagram of the relationship between the local node and the target node provided in Example 1 of the file transmission method of the present application;

[0047] Figure 3 A schematic diagram of node deployment provided for the first embodiment of the file transmission method of the present application;

[0048] Figure 4 A schematic diagram of the system architecture of a file transmission system provided in Example 1 of the file transmission method of the present application;

[0049] Figure 5 A schematic diagram of the calling relationship between the various modules provided in the first embodiment of the file transmission method of the present application;

[0050] Figure 6 A schematic diagram of asynchronous scheduling of files based on user-uploaded files and idle threads provided in the first embodiment of the file transmission method of the present application;

[0051] Figure 7 A schematic diagram of the target node election process provided in Example 1 of the file transmission method of this application;

[0052] Figure 8 A schematic diagram of a file transfer process provided for the first embodiment of the file transfer method of the present application;

[0053] Fig. 9 A flowchart of the second embodiment of the file transmission method of the present application is provided;

[0054] Fig.10 A schematic diagram of sliding IO progress monitoring provided in Example 2 of the file transfer method of this application;

[0055] Fig.11 A schematic diagram of a uniform random sampling summary provided for Example 2 of the file transmission method of this application;

[0056] Fig.12 This is a schematic diagram of the module structure of the file transmission device according to an embodiment of the present application;

[0057] Fig.13 This is a schematic diagram of the device structure of the hardware operating environment involved in the file transfer method in the embodiment of the present application. DETAILED DESCRIPTION

[0058] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0059] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0060] The main solution of the embodiment of the present application is: obtain the target file uploaded by the user, generate a task to be transmitted according to the target file, and the task to be transmitted includes the file to be transmitted, the target node information and the transmission mode; when there is an idle thread, select the target node based on the target node information to obtain the target node; send the file to be transmitted to the target node based on the transmission mode, and call the target node interface to perform file verification; when the file verification is successful, complete the task to be transmitted.

[0061] In this embodiment, for the convenience of description, the following description is made by taking the identification of the file transfer device as the execution subject.

[0062] The existing technology currently occupies a large number of database connections when uploading and downloading data files, and also occupies resources such as server memory and CPU deployed by the business system application. The existing technology uses a synchronous method to process data. When the amount of data is large, users need to wait for a long time for the system to complete processing. When a large number of users operate the system at the same time, multiple upload and download processing threads occupy a large number of server resources, which will affect the response speed and stability of the business system.

[0063] The present application provides a solution, which directly generates a transmission task for the files to be transferred uploaded by the user, determines the target node based on the files uploaded by the user and the configured target node information, and then performs point-to-point asynchronous file transfer between the local and target nodes based on the transmission task and idle threads, which can save time and thread resources waiting for file transfer, improve file transfer efficiency, and ensure the stability of data transmission.

[0064] It can be seen from the above embodiments that the present application discloses a file transfer method, apparatus, device and storage medium, which relates to the technical field of data transmission, and discloses: obtaining a target file uploaded by a user, generating a task to be transferred according to the target file, wherein the task to be transferred includes the file to be transferred, target node information and a transmission mode; when there is an idle thread, selecting a target node based on the target node information to obtain the target node; sending the file to be transferred to the target node based on the transmission mode, and calling the target node interface to perform file verification; when the file verification is successful, completing the task to be transferred; the method directly generates a transmission task for the file to be transferred uploaded by the user, determines the target node based on the file uploaded by the user and the configured target node information, and then performs point-to-point asynchronous file transfer between the local and target nodes based on the transmission task and the idle thread, which can save time and thread resources waiting for file transmission, improve file transmission efficiency, and ensure the stability of data transmission.

[0065] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a file transfer device, etc. The following takes a file transfer device as an example to illustrate this embodiment and the following embodiments.

[0066] Based on this, the present application embodiment provides a file transmission method, referring to Figure 1 , Figure 1 This is a flowchart of the first embodiment of the file transmission method of the present application.

[0067] In this embodiment, the file transmission method includes steps S10 to S40:

[0068] Step S10, obtaining the target file uploaded by the user, and generating a task to be transmitted according to the target file, wherein the task to be transmitted includes the file to be transmitted, target node information and a transmission mode.

[0069] It is understandable that the target file may be a file that needs to be transferred.

[0070] It should be noted that, based on the transmission mode, the target file can be transmitted from the target node to the local node, or from the local node to the target node.

[0071] It should be understood that the transmission mode may include a PUT mode and a GET mode, wherein the PUT mode may be sending a target file from a local node to a target node; and the GET mode may be sending a target file from a target node to a local node.

[0072] It should be noted that when a user uploads a target file on a local node, the user can prepare the file to be transmitted on the local node or the target node, and can configure the target node information on the local node.

[0073] The target node information may include but is not limited to the node name, node IP, node port, node user name, node password, etc. The target node information may be configured according to the needs of the user or the user's requirements for file transfer.

[0074] It should be noted that the file transfer method described in the present invention can be applied to a file transfer system. Each instance deployed by the system is called a node. The user registers a transfer task by calling the interface provided by the node, thereby transferring files between nodes. The node called by the user is called a local node, and the node communicating with the local node is called a target node. Files are transferred between the local node and the target node. The relationship diagram between the local node and the target node can be referred to in Figure 2 , Figure 2 There is one local node and three target nodes. The number of target nodes and the number of local nodes are only used as examples, and there is no specific limit on the number of nodes.

[0075] It should be further explained that the node deployment method of the present invention is flexible and supports single-machine and cluster deployment. One or more nodes can be deployed on a single server. The node deployment method can refer to Figure 3 ,The figure shows the server where the local node is located and the server where the target node is located as a cluster.

[0076] It should be emphasized that the file transfer system adopts a layered modular design. The system architecture of the file transfer system can refer to Figure 4 The system in the figure includes the interface layer, control layer, service layer and basic layer. Among them, the interface layer includes the console and interface; the control layer includes the configuration management module, the transmission task registration module and the transmission task scheduling module; the service layer includes the callback module, the transmission task module, the transmission execution module, the node module, the transmission monitoring module and the file module; the basic layer includes the system data and the SSH protocol.

[0077] Further, the call relationship diagram between each module can refer to Figure 5, the interface layer in the figure provides a graphical console page and a RESTFUL-style interface for users to use. The console is used for configuration management and query of various types of data in the system, and the interface is used to register or query transmission tasks. The control layer isolates the user-oriented interface layer from the underlying service layer, and coordinates and schedules the orderly operation of various modules in the service layer. The configuration management module is used for configuration management of various types of data in the system; the transmission task registration module is used to register transmission tasks; and the transmission task scheduling module is used to schedule transmission tasks. The service layer provides modular core functions and provides services for applications under the unified scheduling of the control layer. The callback module is responsible for the maintenance of callback data, the assembly of callback messages and the execution of callbacks; the node module is responsible for the maintenance of node data and the election of nodes; the transmission task module is responsible for the maintenance of transmission task data, the archiving and cleaning of historical data, etc.; the transmission monitoring module is responsible for the monitoring and progress refresh of transmission progress; the transmission execution module is responsible for the serial execution of transmission processes; the file module is responsible for the maintenance of files, including the detection, creation, deletion, renaming, and permission maintenance of directories and files. The basic layer provides persistent storage of data and secure network communication support. The console calls the callback module, node module and transmission task module through the configuration management module to configure and query system data. The interface calls the transmission task module to register the transmission task through the transmission task registration module. The transmission task scheduling module calls each module through the transmission execution module to execute the transmission task and update the transmission progress.

[0078] Step S20: When there is an idle thread, a target node is selected based on the target node information to obtain the target node.

[0079] It should be noted that the present invention supports high-availability deployment with load balancing, that is, there can be multiple target nodes, and after thread resources are allocated, one needs to be selected from the multiple target nodes for file transfer.

[0080] It is understandable that there may be multiple thread resources for executing the file transfer task, and an idle thread is selected from all thread resources for transmission, and when there is no idle thread, the transmission is waited.

[0081] It should be understood that after the user uploads the target file, a task to be transferred can be generated locally. For asynchronous transmission, the user does not need to wait, and a list of tasks to be transferred can be generated locally based on the tasks to be transferred. When there are idle threads, the tasks to be transferred in the list can be executed in sequence based on the generation time of each task to be transferred.

[0082] It should be noted that asynchronous scheduling of files based on user uploaded files and idle threads can be referred to Figure 6In the figure, after the transmission task is successfully registered, it is in a pending state. Every once in a while, the system will check whether there are idle thread resources and pending tasks. If so, the system will try to assign the task to the thread and update the task status to processing. Since thread resources are always changing dynamically, in order to ensure that a task can only be processed by one thread at a time and avoid phantom reading, an optimistic locking mechanism is adopted here. When the allocation is successful, the thread executes the transmission process normally; when the allocation fails, the task status is rolled back and the processing is canceled.

[0083] It should be emphasized that the target node information can point to the target node cluster where the target node is located. The target node cluster may include multiple target nodes that can receive the target file. A node that can support file reception is selected from the multiple target nodes as the target node to complete the target node election.

[0084] In a feasible implementation, step S20 may include steps A21 to A23:

[0085] Step A21: when there are idle threads, determine the target cluster based on the target node information.

[0086] It should be noted that when there is an idle thread, the task to be transmitted is executed, and the target cluster where the target node is located is determined based on the target node information in the task to be transmitted configured locally by the user. It can be confirmed through relevant information such as the target node IP, target node port, target node user name, target node password, etc. in the target node information.

[0087] It is understandable that the target cluster may be a server including multiple target nodes.

[0088] Step A22, performing random hash screening on multiple nodes in the target cluster to obtain initial nodes.

[0089] It should be noted that the basic idea of ​​random hash screening is to assign a unique identifier (ID) to each node, and then use a hash function to convert these IDs into values ​​within a fixed output range, and then determine which nodes are selected by generating a random number.

[0090] In the specific implementation, an ID is assigned to each target node in the target cluster: this ID can be any form of unique identification, such as node name, IP address, or a custom number or string. The ID of each node is converted into a hash value within a fixed numerical range through a hash function. Commonly used hash functions include MD5, SHA-256, etc. Generate one or more random numbers to determine which nodes' hash values ​​meet the screening conditions. For example, you can select all nodes whose hash values ​​match the random number, or select those nodes whose hash values ​​fall within a specified interval. Filter out nodes that meet the conditions as candidate initial nodes. Select one or more nodes from the candidate initial nodes as the final initial nodes. If you need to select a node, you can select the node whose hash value is closest to the random number; if you need to select multiple nodes, you can select multiple nodes whose hash values ​​are close to the random number according to actual needs.

[0091] Step A23: when the initial node is in a connectable state, use the initial node as a target node.

[0092] It should be noted that the initial node selected by random hashing may not necessarily be able to be connected. It is necessary to determine the status of the initial node, whether it is available, whether it is occupied, and whether it can be used normally. When the initial node can be used normally and can support the transmission of the target file, the initial node is considered to be in a connectable state.

[0093] In the specific implementation, the target node election process can refer to Figure 7 In the figure, the target cluster deploys multiple instances, that is, there are multiple target nodes. Before transmission, the local node uses a random hash algorithm to randomly select a node from the target cluster as the target node, and then detects the availability of the target node. If the detection passes, the connection is established normally; if the detection fails, the hash election is re-performed until an available node is obtained or the traversed target cluster nodes are all unavailable.

[0094] In this implementation, random hash screening can ensure that the selection of nodes is random and fair. At the same time, due to the use of hash functions, a certain distribution uniformity can also be guaranteed, effectively avoiding centralized decision-making or when it is necessary to reduce the risk of single point failures, effectively achieving load balancing.

[0095] The above are only feasible implementations of step S20 provided in this embodiment, and this embodiment does not specifically limit the specific implementation of step S20.

[0096] Step S30: sending the file to be transmitted to the target node based on the transmission mode, and calling the target node interface to perform file verification.

[0097] It is understandable that the transmission mode includes PUT mode and GET mode; wherein, PUT mode means pushing the file from the local node to the target node; GET mode means pulling the file from the target node to the local node.

[0098] It should be noted that after determining the target node for the file transfer, the system executes the file transfer according to the transfer mode selected by the user when registering.

[0099] It should be noted that calling the target node interface to perform file verification may be to verify the transmitted target file at the target node to ensure the accuracy and security of the target file during the transmission process.

[0100] Step S40, when the file verification is successful, the task to be transmitted is completed.

[0101] It should be noted that the verification of the target file is carried out by extracting the summary of the target file before transmission to obtain the summary before transmission, and then calling the interface of the target node to extract the summary of the target file after transmission from the target node, and comparing the summary before and after transmission. If the summary before and after are the same, the verification is successful. If there is an error, a certain error range can be set. Within the error range, it is also considered that the verification is successful. If the error exceeds the error range, it is considered that the verification failed and the transmission can be repeated. The transmission failure can also be fed back to the user to determine whether to retransmit. It can also be directly reselected to transmit after the transmission fails.

[0102] In a feasible implementation manner, step S40 may include steps A50 to A70:

[0103] Step A50, obtaining callback configuration according to the callback name.

[0104] Callback is an optional configuration. If callback is not needed, callback information can be omitted. The existence of a callback name can be understood as determining which user needs callback. By finding the local node of the user who needs callback, the callback configuration based on the user's configuration on the local node can be obtained.

[0105] It should be understood that the callback can be simply understood as feedback on the file transfer result, and the callback configuration can be the category included in the feedback content. For example, the callback configuration can include multiple configurations such as the file transfer start time, the file transfer target node, the file output local node, and the file transfer time.

[0106] It should be noted that the user can select multiple callback names, or select the same callback name to generate multiple callback messages with different configurations, which can be determined based on the callback configuration pre-set by the user in the local node when uploading the target file.

[0107] It should be noted that when uploading files, users can pre-configure callback information locally, including callback name, callback method, callback connection timeout, callback communication timeout, callback address, callback message header, callback message body, callback retry rules, etc.

[0108] Furthermore, the user calls the task registration interface provided by the local node, provides the transmission mode, target node name, callback name (optional), source file path, and target file path to register a transmission task, and the scheduling module will allocate resources in the background to asynchronously process the file transfer. Since the transmission task is registered by calling the interface, the user can call it locally or remotely.

[0109] Step A60: Generate a splicing message according to the task to be transmitted and the callback configuration.

[0110] It is understandable that at this time the task to be transmitted has been transmitted to the target node, and the relevant transmission information of the task to be transmitted from the user uploading it to the local to the transmission from the local to the target node is recorded, such as the transmission time, start time, transmission speed and other related information.

[0111] It should be understood that the relevant information in the file transfer process required for the user callback is determined according to the callback configuration.

[0112] In a specific implementation, based on the task to be transferred, the file transfer start time, target node, local node, transfer end time and file transfer time can be known. The callback configuration includes the file transfer start time, target node and transfer end time. The file transfer start time, target node and transfer end time of the task to be transferred are spliced ​​to generate a spliced ​​message.

[0113] Step A70, calling back the spliced ​​message according to a preset number of callbacks.

[0114] It is understandable that the preset callback number can be the callback number and callback object selected by the user when determining that a callback is needed. The callback objects can be the same or different. The same callback object can also be called back multiple times, and the content of each callback can be the same or different.

[0115] It should be emphasized that the file transfer is basically completed after the file verification is passed and the file is renamed and authorized. If the parameter submitted by the user when registering the task contains a callback name, the callback will be executed. The callback configuration is obtained from the configuration of the local node through the callback name, the callback message is spliced ​​according to the callback configuration, and the POST request or GET request of the HTTP protocol is executed.

[0116] It should be further explained that callback configurations can have the same name, so multiple addresses can be configured to perform multiple callbacks, and when the callback fails, it can also be automatically retried according to the user configuration. The message header and message body in the callback support user-defined format and dynamic generation, and can be spliced ​​and supplemented with any field information in the transmission task.

[0117] In this implementation, the user configures a callback with unlimited times and callback objects in advance on the local node, and the automatic callback after transmission generates a message, which effectively feeds back the file transfer status to the user in real time, thereby improving the user experience.

[0118] The above are merely feasible implementations after step S40 provided in this embodiment, and this embodiment does not specifically limit the specific implementation of step S40.

[0119] It should be noted that based on the above steps 10 to 70, the file transfer process can refer to Figure 8 In the figure, after the user uploads the target file to generate the task to be transferred, it waits for the idle thread to execute the task, and then determines whether there are available thread resources and whether there are pending transfer tasks in the list of tasks to be transferred. If there are idle threads and pending transfer tasks, the task to be transferred is assigned to the idle thread and the task status of the task to be transferred in the list to be transferred is updated. After the assignment is successful, the target node is selected, and data is transmitted according to the transmission mode selected by the user. When the transmission mode is PUT mode, node authentication is first performed and the target node interface is called. According to the user's storage requirements for the target file, a file directory corresponding to the storage requirements is created on the target node, and then PUT transmission is performed. After the transmission, the target node interface is called to perform file verification. After the verification is successful, the file to be transferred that has been successfully verified on the target node is renamed according to the load user's requirements;

[0120] When the transmission mode is GET mode, a directory is created locally according to user needs, and then GET transmission is performed. The target node interface and the locally received files are called for verification. After the verification is successful, the files are renamed locally to meet user needs.

[0121] After the transmission verification is successful, the user will pre-select whether to call back. If the user does not choose callback, the transmission ends. If the user chooses callback, the corresponding callback method is selected according to the transmission method. The callback method can include GET callback method, which generates a spliced ​​message according to the callback configuration and calls the interface through the GET method for callback; it also includes POST mode, which generates a spliced ​​message according to the callback configuration and calls the interface through the POST method.

[0122] This embodiment provides a file transfer method, which directly generates a transmission task for the file to be transferred uploaded by the user, determines the target node based on the file uploaded by the user and the configured target node information, and then performs point-to-point asynchronous file transfer between the local and target nodes based on the transmission task and idle threads, which can save time and thread resources waiting for file transfer, improve file transfer efficiency, and ensure the stability of data transmission.

[0123] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Fig. 9 , step S30, the file transmission method further includes steps S31 to S33:

[0124] Step S31: sending the file to be transmitted to the target node based on the transmission mode, and obtaining a user-specified refresh threshold.

[0125] It should be noted that during the file transfer process, users need to obtain the transfer progress, which needs to be achieved by monitoring IO. There are two traditional monitoring methods, one is timed calculation and the other is real-time calculation. Timed calculation is not accurate enough and is easily affected by the network. Real-time calculation affects efficiency. The cumulative update operations during each IO will seriously affect the transmission efficiency. Frequent updates will also bring higher risks.

[0126] It is understandable that during the file transfer process, the user can set or input in real time to confirm the time end for obtaining the file transfer progress, that is, to formulate a refresh threshold.

[0127] Step S32, obtaining a refresh point based on the user-specified refresh threshold, and identifying the file transfer progress based on the refresh point.

[0128] It is understandable that the refresh point is obtained based on the user-specified refresh threshold, and the file transfer progress can be identified based on the refresh point by sliding IO, so that the refresh threshold slides synchronously with the transfer IO, thereby realizing real-time update of the refresh point.

[0129] It should be understood that identifying the file transfer progress based on the refresh point can reduce the refresh frequency according to user needs, improve the transmission efficiency, and accurately reflect the current transmission progress to meet user needs.

[0130] In the specific implementation, the sliding IO progress monitoring diagram can refer to Fig.10 , the user can specify the refresh threshold, the refresh threshold slides synchronously with the transmission IO, when the tail of the refresh threshold leaves the previous refresh point, the position corresponding to the head of the refresh threshold is the new refresh point, the refresh point position is updated and a progress refresh is performed.

[0131] Step S33, when the file transfer progress is completed, calling the target node interface to perform file verification.

[0132] It should be noted that for calling the target interface for file verification, analysis can be performed in both PUT mode and GET mode; PUT mode can be simply understood as file sending, and GET mode can be simply understood as file receiving; when the file is sent, the local node is the sending node, and a summary of the file uploaded by the user at the sending node is extracted, and then a summary of the file received by the target node is extracted, and the summaries extracted by the two nodes are compared to determine whether a file error occurs during the file transmission process.

[0133] Furthermore, when receiving a file, the local node is the receiving node, which extracts a summary of the received file, then calls the target node interface, extracts a summary of the file uploaded by the user at the target node, and compares the summaries extracted by the two nodes. If they are identical or meet the similarity requirements, the file verification is completed.

[0134] In a feasible implementation, step S33 may include steps A331 to A333:

[0135] Step A331, when the file transfer progress is complete, call the target node interface to identify the file to be transferred.

[0136] It is understandable that, no matter in GET mode or PUT mode, after the file transfer is completed, the interface of the target node is called to extract the summary of the target file at the target node.

[0137] It should be understood that the transmission file to be identified in the PUT mode may be a target file transmitted to the target node; and the transmission file to be identified in the GET mode may be a target file uploaded by the user.

[0138] It should be noted that before sending the file to be transferred to the target node based on the transfer mode and calling the target node interface to perform file verification, it also includes: obtaining the locally configured target node authentication information; establishing a file transfer connection with the target node according to the target node authentication information based on a preset communication protocol, so as to send the file to be transferred to the target node based on the file transfer connection.

[0139] It is understandable that before transferring files between a local node and a target node, the security of each target node needs to be verified.

[0140] It should be understood that the local node can be pre-configured with verification information of the target node for file transfer. When the user determines the target node, the verification information is sent to the target node at the same time as the file to be transferred for verification to ensure the security of file transfer.

[0141] It should be noted that the target node needs to be authenticated before the formal transmission to ensure the node security. The authentication information comes from the target node information configured on the local node in the preparation stage. After the authentication is passed, an SSH connection will be established between the nodes. After that, a directory for storing files will be established locally or on the target, and then the encrypted file will be pushed or pulled through the SSH protocol.

[0142] Step A332: classify the files to be transmitted according to their sizes to obtain file categories.

[0143] It should be noted that file categories can include Category I files and Category II files. A file size threshold is set. Files smaller than the threshold are considered Category I files, and files greater than or equal to the threshold are considered Category II files. Category I files can be simply understood as small files, and Category II files can be simply understood as large files.

[0144] It is understandable that currently when data is being transmitted, when the file is large, it is easy to be affected during the transmission process, resulting in file transmission failure.

[0145] It should be noted that in the traditional file transfer process, users can see and operate the data in transfer, which can easily cause the transfer to fail. The present invention adopts a triple control mechanism to avoid this problem: Data name control: The data in transfer uses a random temporary name instead of a formal name. Data access control: Based on the ACL access control list, the data in transfer is set to a special state that is unreadable, unwritable, and unexecutable by other users. Data visibility control: The data in transfer is set to invisible to hide the data in transfer. After the transfer is completed, the temporary data is renamed as formal data, and its access control permissions and visibility are modified. This can avoid user misoperation as much as possible and ensure stable and reliable transmission.

[0146] Step A333: extracting the file digest of the file to be transmitted based on the file category, and performing verification based on the file digest.

[0147] It should be noted that the extracting the file summary of the file to be transmitted based on the file category and performing verification based on the file summary include: when the file category is a Category I file, obtaining a file summary based on the file to be transmitted, and performing verification based on the file summary; when the file category is a Category II file, performing data sampling on the file to be transmitted to obtain sample data, respectively extracting the sample summary of each sample data and merging the sample summaries in sequence to obtain a file summary and a sample merge position; and performing verification based on the file summary and the sample merge position.

[0148] Furthermore, a schematic diagram of uniform random sampling summary can be found at Fig.11 , Fig.11 The summary extraction method can be data sampling, using a uniform random sampling strategy to obtain representative data samples, dividing the data evenly into multiple intervals, randomly intercepting a random length of data from each interval as a sample, and recording the sample position; using the national secret summary algorithm SM3 to calculate the summary for each sample; concatenating all sample summaries into one sample in sequence, and then calculating the summary of the sample, and the final deliverable is the sample summary and sample position; using the same sample position and method to calculate the summary of the data to be compared, and judging whether the data is consistent by comparing whether the summaries are the same.

[0149] In this implementation, by comprehensively considering efficiency and user needs, the sliding IO progress monitoring model can more accurately reflect the transmission progress while ensuring efficient transmission.

[0150] The above are only feasible implementations of step S33 provided in this embodiment, and this embodiment does not specifically limit the specific implementation of step S33.

[0151] This embodiment provides a file transmission method, which improves transmission stability and efficient file integrity verification through a triple control mechanism, and layers files according to size. For small files, the complete file summary is directly calculated; for large files, the summary is calculated using a uniform random sampling summary algorithm, which can significantly improve efficiency while ensuring accuracy.

[0152] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the file transmission method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0153] This application also provides a file transmission device, please refer to Fig.12 , the file transmission device comprises:

[0154] An acquisition module 10 is used to acquire a target file uploaded by a user, and generate a task to be transmitted according to the target file, wherein the task to be transmitted includes the file to be transmitted, target node information, and a transmission mode;

[0155] A node election module 20, configured to select a target node based on the target node information to obtain the target node when there is an idle thread;

[0156] A file transmission module 30, configured to send the file to be transmitted to the target node based on the transmission mode, and call the target node interface to perform file verification;

[0157] The file transmission module 30 is further configured to complete the task to be transmitted when the file verification succeeds.

[0158] The node election module 20 is further configured to determine a target cluster based on the target node information when there is an idle thread;

[0159] Performing random hashing screening on multiple nodes in the target cluster to obtain an initial node;

[0160] When the initial node is in a connectable state, the initial node is used as a target node.

[0161] The file transmission module 30 is further used to send the file to be transmitted to the target node based on the transmission mode, and obtain a user-specified refresh threshold;

[0162] Obtaining a refresh point based on the user-specified refresh threshold, and identifying a file transfer progress based on the refresh point;

[0163] When the file transfer progress is complete, the target node interface is called to perform file verification.

[0164] The file transfer module 30 is also used to obtain the locally configured target node authentication information;

[0165] A file transfer connection is established with the target node based on a preset communication protocol according to the target node authentication information, so as to send the file to be transferred to the target node based on the file transfer connection.

[0166] The file transmission module 30 is further configured to call the target node interface to identify the file to be transmitted when the file transmission progress is completed;

[0167] Classifying the files to be transmitted according to the sizes of the files to be transmitted to obtain file categories;

[0168] A file summary of the file to be transmitted is extracted based on the file category, and verification is performed based on the file summary.

[0169] The file transmission module 30 is further configured to obtain a file summary according to the file to be transmitted and perform verification based on the file summary when the file category is a type one file;

[0170] When the file category is a Class II file, data sampling is performed on the file to be transmitted to obtain sample data, sample summaries of each sample data are extracted respectively, and each sample summary is merged in sequence to obtain a file summary and a sample merge position; verification is performed based on the file summary and the sample merge position.

[0171] The file transfer module 30 is also used to obtain the callback configuration according to the callback name;

[0172] Generate a splicing message according to the task to be transmitted and the callback configuration;

[0173] The spliced ​​message is called back according to a preset number of callbacks.

[0174] The file transmission device provided by the present application adopts the file transmission method in the above embodiment, which can solve the technical problem that the current file transmission requires synchronous calling and wastes thread resources. Compared with the prior art, the beneficial effects of the file transmission device provided by the present application are the same as the beneficial effects of the file transmission method provided by the above embodiment, and other technical features in the file transmission device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0175] The present application provides a file transfer device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the file transfer method in the above-mentioned embodiment 1.

[0176] Reference below Fig.13 , which shows a schematic diagram of the structure of a file transmission device suitable for implementing the embodiment of the present application. The file transmission device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.13 The file transfer device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0177] like Fig.13 As shown, the file transfer device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the file transfer device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the file transfer device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a file transfer device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or provided instead.

[0178] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0179] The file transfer device provided by the present application adopts the file transfer method in the above embodiment, which can solve the technical problem that the current file transfer requires synchronous calls and wastes thread resources. Compared with the prior art, the beneficial effects of the file transfer device provided by the present application are the same as the beneficial effects of the file transfer method provided by the above embodiment, and other technical features in the file transfer device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0180] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0181] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0182] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the file transmission method in the above-mentioned embodiment.

[0183] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0184] The computer-readable storage medium may be included in the file transmission device; or may exist independently without being assembled into the file transmission device.

[0185] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by a file transfer device, the file transfer device: obtains a target file uploaded by a user, generates a task to be transferred according to the target file, and the task to be transferred includes the file to be transferred, target node information, and a transfer mode; when there is an idle thread, performs a target node election based on the target node information to obtain a target node; sends the file to be transferred to the target node based on the transfer mode, and calls the target node interface to perform file verification; when the file verification is successful, completes the task to be transferred.

[0186] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0187] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0188] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0189] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned file transfer method, and can solve the technical problem that the current file transfer requires synchronous calls and wastes thread resources. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the file transfer method provided by the above-mentioned embodiment, and will not be repeated here.

[0190] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned file transmission method when executed by a processor.

[0191] The computer program product provided by the present application can solve the technical problem that the current file transfer requires synchronous calls and wastes thread resources. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the file transfer method provided by the above embodiment, which will not be repeated here.

[0192] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A file transmission method, characterized in that: The file transmission method comprises: Obtain the target file uploaded by the user, and generate a task to be transmitted according to the target file, wherein the task to be transmitted includes the file to be transmitted, target node information, and transmission mode; When there is an idle thread, a target node is selected based on the target node information to obtain the target node; Sending the file to be transmitted to the target node based on the transmission mode, and calling the target node interface to perform file verification; When the file verification succeeds, the task to be transmitted is completed.

2. The file transmission method according to claim 1, characterized in that: When there is an idle thread, selecting a target node based on the target node information to obtain the target node includes: When there are idle threads, determining a target cluster based on the target node information; Performing random hashing screening on multiple nodes in the target cluster to obtain an initial node; When the initial node is in a connectable state, the initial node is used as a target node.

3. The file transmission method according to claim 1, characterized in that: The sending the file to be transmitted to the target node based on the transmission mode, and calling the target node interface to perform file verification, includes: Sending the file to be transmitted to the target node based on the transmission mode, and obtaining a user-specified refresh threshold; Obtaining a refresh point based on the user-specified refresh threshold, and identifying a file transfer progress based on the refresh point; When the file transfer progress is complete, the target node interface is called to perform file verification.

4. The file transmission method according to claim 3, characterized in that: Before sending the file to be transmitted to the target node based on the transmission mode and calling the target node interface to perform file verification, the method further includes: Get the authentication information of the locally configured target node; A file transfer connection is established with the target node based on a preset communication protocol according to the target node authentication information, so as to send the file to be transferred to the target node based on the file transfer connection.

5. The file transmission method according to claim 3, characterized in that: When the file transfer progress is complete, calling the target node interface to perform file verification includes: When the file transfer progress is complete, calling the target node interface to identify the file to be transferred; Classifying the files to be transmitted according to the sizes of the files to be transmitted to obtain file categories; A file summary of the file to be transmitted is extracted based on the file category, and verification is performed based on the file summary.

6. The file transmission method as claimed in claim 5, characterized in that: The extracting the file summary of the to-be-transmitted file based on the file category and performing verification based on the file summary includes: When the file category is a type one file, obtaining a file summary according to the file to be transmitted, and performing verification based on the file summary; When the file category is a Class II file, data sampling is performed on the file to be transmitted to obtain sample data, sample summaries of each sample data are extracted respectively, and each sample summary is merged in sequence to obtain a file summary and a sample merge position; verification is performed based on the file summary and the sample merge position.

7. The file transmission method according to claim 1, characterized in that: The task to be transmitted also includes a callback name; When the file verification succeeds, after the task to be transmitted is completed, the method further includes: Get callback configuration according to callback name; Generate a splicing message according to the task to be transmitted and the callback configuration; The spliced ​​message is called back according to a preset number of callbacks.

8. A file transmission device, characterized in that: The file transmission device comprises: An acquisition module is used to acquire a target file uploaded by a user and generate a task to be transmitted according to the target file. The task to be transmitted includes the file to be transmitted, target node information and transmission mode; A node election module, used for, when there is an idle thread, performing target node election based on the target node information to obtain a target node; A file transmission module, used for sending the file to be transmitted to the target node based on the transmission mode, and calling the target node interface to perform file verification; The file transmission module is further used to complete the task to be transmitted when the file verification succeeds.

9. A file transfer device, characterized in that: The device comprises: a memory, a processor, and a file transfer program stored in the memory and executable on the processor, wherein the file transfer program is configured to implement the file transfer method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores a file transfer program, and when the file transfer program is executed by the processor, the file transfer method according to any one of claims 1 to 7 is implemented.

Images