File transmission method, device, equipment and storage medium

Abstract

This application discloses a file transfer method, apparatus, device, and storage medium, relating to the field of data transmission technology. The method involves: acquiring a target file uploaded by a user; generating a transfer task based on the target file, the transfer task including the file to be transferred, target node information, and a transfer mode; when an idle thread exists, electing a target node based on the target node information; sending the file to be transferred to the target node based on the transfer mode, and calling the target node interface for file verification; and completing the transfer task upon successful file verification. This method directly generates a transfer task from the user-uploaded file, determines the target node based on the uploaded file and configured target node information, and then performs point-to-point asynchronous file transfer between the local machine and the target node based on the transfer task and an idle thread. This saves time and thread resources spent waiting for file transfer, improving file transfer efficiency.

Description

Technical Field

[0001] This application relates to the field of data transmission technology, and in particular to file transmission methods, apparatus, devices and storage media. Background Technology

[0002] Currently, uploading and downloading data files not only consumes a large number of database connections but also consumes server resources such as memory and CPU for the business system applications. Existing technologies process data synchronously, which means that when the data volume is large, users need to wait a considerable amount of time for the system to complete processing. Furthermore, when a large number of users are operating the system simultaneously, the multiple upload and download processing threads consume significant server resources, impacting the response speed and stability of the business system.

[0003] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0004] The main purpose of this application is to provide a file transfer method, apparatus, device and storage medium, which aims to solve the technical problem that current file transfer requires synchronous calls and wastes thread resources.

[0005] To achieve the above objectives, this application proposes a file transfer method, the file transfer method comprising:

[0006] Obtain the target file uploaded by the user, and generate a transmission task based on the target file. The transmission task includes the file to be transmitted, target node information, and transmission mode.

[0007] When an idle thread exists, a target node is selected based on the target node information.

[0008] Based on the transmission mode, the file to be transmitted is sent to the target node, and the target node's interface is called to perform file verification;

[0009] Upon successful file verification, the transmission task is completed.

[0010] In one embodiment, the step of selecting a target node based on the target node information when an idle thread exists includes:

[0011] When idle threads exist, the target cluster is determined based on the target node information;

[0012] Randomly hash and filter multiple nodes in the target cluster to obtain initial nodes;

[0013] When the initial node is in a connectable state, the initial node is used as the target node.

[0014] In one embodiment, sending the file to be transmitted to the target node based on the transmission mode and calling the target node's interface to perform file verification includes:

[0015] Based on the transmission mode, the file to be transmitted is sent to the target node, and the user-specified refresh threshold is obtained;

[0016] A refresh point is obtained based on the user-specified refresh threshold, and the file transfer progress is identified based on the refresh point.

[0017] When the file transfer progress indicates that the transfer is complete, the target node interface is invoked to perform file verification.

[0018] In one embodiment, before sending the file to be transmitted to the target node based on the transmission mode and calling the target node interface to perform file verification, the method further includes:

[0019] Obtain the target node authentication information configured locally;

[0020] Based on a preset communication protocol, a file transfer connection is established with the target node according to the target node authentication information, so as to send the file to be transferred to the target node based on the file transfer connection.

[0021] In one embodiment, the step of calling the target node interface to perform file verification includes:

[0022] When the file transfer progress indicates that the transfer is complete, the target node interface is invoked to identify the file to be transferred;

[0023] The files to be transmitted are classified according to their size to obtain file categories;

[0024] The file digest of the file to be transmitted is extracted based on the file category, and the file digest is used for verification.

[0025] In one embodiment, the step of extracting a file digest of the file to be transmitted based on the file category and performing verification based on the file digest includes:

[0026] When the file category is a type of file, a file digest is obtained based on the file to be transmitted, and verification is performed based on the file digest.

[0027] When the file category is Class II, data sampling is performed on the file to be transmitted to obtain sample data. Sample summaries are extracted from each sample data and merged sequentially to obtain a file summary and sample merge position. Verification is performed based on the file summary and the sample merge position.

[0028] In one embodiment, the task to be transmitted further includes a callback name;

[0029] After the file verification is successful and the transmission task is completed, the process further includes:

[0030] Get the callback configuration based on the callback name;

[0031] Generate a spliced ​​message based on the task to be transmitted and the callback configuration;

[0032] The concatenated message is callbackd according to a preset number of callbacks.

[0033] Furthermore, to achieve the above objectives, this application also proposes a file transfer device, which includes:

[0034] The acquisition module is used to acquire the target file uploaded by the user and generate a transmission task based on the target file. The transmission task includes the file to be transmitted, target node information, and transmission mode.

[0035] The node election module is used to elect a target node based on the target node information when there is an idle thread.

[0036] The file transfer module is used to send the file to be transferred to the target node based on the transfer mode, and to call the target node interface to perform file verification;

[0037] The file transfer module is also used to complete the transfer task when the file verification is successful.

[0038] In addition, to achieve the above objectives, this application also proposes a file transfer device, the device comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, the computer program being configured to implement the steps of the file transfer method as described above.

[0039] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and which, when executed by a processor, implements the steps of the file transfer method described above.

[0040] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the file transfer method described above.

[0041] One or more technical solutions proposed in this application have at least the following technical effects:

[0042] By directly generating transmission tasks from user-uploaded files, and determining the target node based on the uploaded file and configured target node information, asynchronous point-to-point file transfer between the local machine and the target node is performed based on the transmission task and idle threads. This saves time and thread resources waiting for file transfer, improves file transfer efficiency, and ensures data transmission stability. Attached Figure Description

[0043] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0044] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0045] Figure 1 This is a flowchart illustrating an embodiment of the document transmission method of this application.

[0046] Figure 2 This is a schematic diagram illustrating the relationship between the local node and the target node provided in Embodiment 1 of the file transfer method of this application;

[0047] Figure 3 This is a schematic diagram of node deployment provided in Embodiment 1 of the file transmission method of this application;

[0048] Figure 4 This is a schematic diagram of the system architecture of the file transfer system provided in Embodiment 1 of the file transfer method of this application;

[0049] Figure 5 This is a schematic diagram illustrating the calling relationships between the various modules provided in Embodiment 1 of the file transfer method of this application;

[0050] Figure 6 This is a schematic diagram of asynchronous file scheduling based on user-uploaded files and idle threads, provided in Embodiment 1 of the file transfer method of this application.

[0051] Figure 7 This is a schematic diagram of the target node election process provided in Embodiment 1 of the document transmission method of this application;

[0052] Figure 8 This is a schematic diagram of the file transfer process provided in Embodiment 1 of the file transfer method of this application;

[0053] Figure 9 This is a flowchart illustrating Embodiment 2 of the document transmission method of this application;

[0054] Figure 10 This is a schematic diagram of the sliding I / O progress monitoring provided in Embodiment 2 of the file transfer method of this application;

[0055] Figure 11 This is a schematic diagram of a uniform random sampling summary provided in Embodiment 2 of the document transmission method of this application;

[0056] Figure 12 This is a schematic diagram of the module structure of the file transfer device according to an embodiment of this application;

[0057] Figure 13 This is a schematic diagram of the device structure of the hardware operating environment involved in the file transfer method in the embodiments of this application. Detailed Implementation

[0058] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0059] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0060] The main solution of this application embodiment is as follows: obtain the target file uploaded by the user, generate a transmission task based on the target file, the transmission task includes the file to be transmitted, target node information and transmission mode; when there is an idle thread, elect a target node based on the target node information to obtain a target node; send the file to be transmitted to the target node based on the transmission mode, and call the target node interface to perform file verification; when the file verification is successful, complete the transmission task.

[0061] In this embodiment, for ease of description, the following description will focus on identifying the file transfer device as the execution subject.

[0062] Current technologies consume a large number of database connections and server resources such as memory and CPU when uploading and downloading data files. Furthermore, the synchronous processing of data in existing technologies results in long waiting times for users to complete processing large volumes of data. When many users operate the system simultaneously, the multiple upload and download threads consume significant server resources, impacting the responsiveness and stability of the business system.

[0063] This application provides a solution that directly generates a transmission task based on the user-uploaded file to be transmitted. After determining the target node based on the user-uploaded file and the configured target node information, it performs point-to-point asynchronous file transmission between the local machine and the target node based on the transmission task and idle threads. This can save the time and thread resources spent waiting for file transmission, improve file transmission efficiency, and ensure the stability of data transmission.

[0064] As can be seen from the above embodiments, this application discloses a file transfer method, apparatus, device, and storage medium, relating to the field of data transmission technology. The method involves: acquiring a target file uploaded by a user; generating a transfer task based on the target file, the transfer task including the file to be transferred, target node information, and a transfer mode; when an idle thread exists, electing a target node based on the target node information; sending the file to be transferred to the target node based on the transfer mode, and calling the target node interface for file verification; and completing the transfer task upon successful file verification. This method directly generates a transfer task from the user-uploaded file, determines the target node based on the user-uploaded file and configured target node information, and then performs point-to-point asynchronous file transfer between the local machine and the target node based on the transfer task and an idle thread. This saves time and thread resources waiting for file transfer, improves file transfer efficiency, and ensures data transmission stability.

[0065] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or file transfer device capable of performing the above functions. The following description uses a file transfer device as an example to illustrate this embodiment and the subsequent embodiments.

[0066] Based on this, embodiments of this application provide a file transfer method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the file transmission method of this application.

[0067] In this embodiment, the file transfer method includes steps S10 to S40:

[0068] Step S10: Obtain the target file uploaded by the user, and generate a transmission task based on the target file. The transmission task includes the file to be transmitted, target node information, and transmission mode.

[0069] Understandably, the target file can be a file that needs to be transmitted.

[0070] It should be noted that, based on the transmission mode, the target file can be transmitted from the target node to the local node, or from the local node to the target node.

[0071] It should be understood that the transfer mode can include PUT mode and GET mode. PUT mode can be used to send the target file from the local node to the target node; GET mode can be used to send the target file from the target node to the local node.

[0072] It should be noted that when a user uploads a target file on a local node, the file to be transferred can be prepared on either the local node or the target node, and the target node information can be configured on the local node.

[0073] The target node information may include, but is not limited to, node name, node IP, node port, node username, node password, etc., and can be configured according to the user's needs or the user's requirements for file transfer.

[0074] It should be noted that the file transfer method described in this invention can be applied to a file transfer system. Each instance deployed in the system is called a node. Users register transfer tasks by calling the interface provided by the node, thereby transferring files between nodes. The node initiated by the user is called the local node, and the node communicating with the local node is called the target node. Files are transferred between the local node and the target node. The relationship diagram between the local node and the target node can be found in [reference needed]. Figure 2 , Figure 2 It includes one local node and three target nodes. The number of target nodes and local nodes here are only for illustrative purposes and there is no specific limit to the number of nodes.

[0075] It should be further noted that the node deployment method of this invention is flexible, supporting both single-machine and cluster deployment. One or more nodes can be deployed on a single server. The node deployment method can be found in [reference needed]. Figure 3 The diagram shows the local node's server and the target node's server as a cluster.

[0076] It should be emphasized that a layered, modular design is adopted for the file transfer system. The system architecture of the file transfer system can refer to... Figure 4 The system in the diagram comprises an interface layer, a control layer, a service layer, and a foundation layer. The interface layer includes a console and interfaces; the control layer includes a configuration management module, a transmission task registration module, and a transmission task scheduling module; the service layer includes a callback module, a transmission task module, a transmission execution module, a node module, a transmission monitoring module, and a file module; and the foundation layer includes system data and the SSH protocol.

[0077] Furthermore, a diagram showing the call relationships between the various modules can be found here. Figure 5The interface layer in the diagram provides a graphical console page and RESTful interfaces for users. The console is used for configuration management and querying of various data within the system, while the interfaces are used to register or query transmission tasks. The control layer isolates the user-facing interface layer from the underlying service layer, coordinating and scheduling the orderly operation of various modules within the service layer. The configuration management module is used for configuration management of various data within the system; the transmission task registration module is used for registering transmission tasks; and the transmission task scheduling module is used for scheduling transmission tasks. The service layer provides modular core functions, providing services to applications under the unified scheduling of the control layer. The callback module is responsible for maintaining callback data, assembling callback messages, and executing callbacks; the node module is responsible for maintaining node data and electing nodes; the transmission task module is responsible for maintaining transmission task data, archiving and cleaning historical data, etc.; the transmission monitoring module is responsible for monitoring and refreshing transmission progress; the transmission execution module is responsible for the sequential execution of the transmission process; and the file module is responsible for file maintenance, including directory and file detection, creation, deletion, renaming, and permission maintenance. The base layer provides persistent data storage and secure network communication support. The console calls the callback module, node module, and transmission task module through the configuration management module to configure and query system data. The interface registers transmission tasks by calling the transmission task registration module. The transmission task scheduling module executes transmission tasks and updates transmission progress by calling various modules through the transmission execution module.

[0078] Step S20: When there is an idle thread, a target node is selected based on the target node information.

[0079] It should be noted that this invention supports high-availability deployment with load balancing, meaning that there can be multiple target nodes. After the thread resources are allocated, one of the target nodes needs to be elected from among the many target nodes for file transfer.

[0080] Understandably, multiple thread resources can exist to execute file transfer tasks. An idle thread is selected from all thread resources for transfer, and if no idle thread exists, the transfer is waited for.

[0081] It should be understood that after a user uploads the target file, a task to be transferred can be generated locally. For asynchronous transfer, the user does not need to wait. A list of tasks to be transferred is generated locally based on the task to be transferred. When there is an idle thread, the tasks in the list to be transferred can be executed sequentially according to the generation time of each task.

[0082] It should be noted that asynchronous file scheduling based on user-uploaded files and idle threads can be referenced. Figure 6In the diagram, after a transmission task is successfully registered, it is in a pending state. Every so often, the system checks whether there are any idle thread resources and pending tasks. If so, the system will try to assign the task to the thread and update the task's status to "processing". Since thread resources are constantly changing, in order to ensure that a task can only be processed by one thread at a time and to avoid phantom reads, an optimistic locking mechanism is used. When the allocation is successful, the thread executes the transmission process normally; when the allocation fails, the task status is rolled back and processing is canceled.

[0083] It should be emphasized that the target node information may point to the target node cluster where the target node is located. This target node cluster may include multiple target nodes that can receive the target file. The target node election is completed by selecting a node that can support file reception from among the multiple target nodes.

[0084] In one feasible implementation, step S20 may include steps A21 to A23:

[0085] Step A21: When there are idle threads, determine the target cluster based on the target node information.

[0086] It should be noted that when an idle thread exists, the task to be transferred is executed. The target cluster where the target node is located is determined based on the target node information in the task to be transferred configured locally by the user. This can be confirmed by relevant information such as the target node IP, target node port, target node username, and target node password in the target node information.

[0087] Understandably, the target cluster can be a server that includes multiple target nodes.

[0088] Step A22: Randomly hash and filter multiple nodes in the target cluster to obtain the initial nodes.

[0089] It should be noted that the basic idea of ​​random hash filtering is to assign a unique identifier (ID) to each node, and then use a hash function to convert these IDs into values ​​within a fixed output range, and generate a random number to determine which nodes are selected.

[0090] In implementation, each target node in the target cluster is assigned an ID. This ID can be any unique identifier, such as a node name, IP address, or a custom number or string. Each node's ID is then converted into a hash value within a fixed range using a hash function. Common hash functions include MD5 and SHA-256. One or more random numbers are generated to determine which nodes' hash values ​​meet the filtering criteria. For example, all nodes whose hash values ​​match the random number can be selected, or nodes whose hash values ​​fall within a specified range can be selected. Nodes meeting the criteria are selected as candidate initial nodes. One or more nodes are then chosen from the candidate initial nodes as the final initial nodes. If only one node needs to be selected, the node with the hash value closest to the random number can be chosen; if multiple nodes need to be selected, multiple nodes with hash values ​​close to the random number can be selected according to actual needs.

[0091] Step A23: When the initial node is in a connectable state, the initial node is used as the target node.

[0092] It should be noted that the initial node selected by random hashing may not necessarily be able to be connected. It is necessary to determine the status of the initial node, whether it is available, whether it is occupied, and whether it can be used normally. If the initial node can be used normally and can support the transfer of the target file, the initial node is considered to be in a connectable state.

[0093] In practical implementation, the target node election process can be referenced. Figure 7 The target cluster in the diagram has multiple instances deployed, meaning there are multiple target nodes. Before transmission, the local node uses a random hash algorithm to randomly elect a node from the target cluster as the target node, and then checks the availability of the target node. If the check passes, a connection is established normally; if the check fails, the node is re-hashed and elected until an available node is found or all nodes in the target cluster are unavailable.

[0094] In this embodiment, random hashing ensures that node selection is random and fair. At the same time, the use of a hash function guarantees a certain degree of distribution uniformity, effectively avoiding centralized decision-making or reducing the risk of single points of failure, and effectively achieving load balancing.

[0095] The above are merely feasible implementation methods for step S20 provided in this embodiment. This embodiment does not specifically limit the specific implementation method of step S20.

[0096] Step S30: Based on the transmission mode, send the file to be transmitted to the target node, and call the target node interface to perform file verification.

[0097] Understandably, the transfer modes include PUT mode and GET mode; PUT mode means pushing the file from the local node to the target node; GET mode means pulling the file from the target node to the local node.

[0098] It should be noted that after the target node for the file transfer is determined, the system will perform the file transfer according to the transfer mode selected by the user during registration.

[0099] It should be noted that calling the target node interface to perform file verification can verify the transmitted target file at the target node to ensure the accuracy and security of the target file during the transmission process.

[0100] Step S40: When the file verification is successful, the task to be transmitted is completed.

[0101] It should be noted that the verification of the target file is achieved by extracting the digest of the target file before transmission, then calling the target node's interface to extract the digest of the target file after transmission from the target node, and comparing the digests before and after transmission. If the digests are the same, the verification is successful. If there is an error, a certain error range can be set. If the error is within the error range, the verification is considered successful. If the error exceeds the error range, the verification is considered to have failed. In this case, the transmission can be re-attempted, or a transmission failure can be reported to the user to determine whether to re-attempt the transmission, or the target node can be directly selected for transmission after the transmission fails.

[0102] In one possible implementation, steps A50 to A70 may be included after step S40:

[0103] Step A50: Obtain the callback configuration based on the callback name.

[0104] Callbacks are optional. If a callback is not needed, callback information can be left unconfigured. The existence of a callback name can be understood as determining which user needs a callback. By finding the user who needs a callback, the local node of that user can be found, and the callback configuration based on the user's configuration on that local node can be obtained.

[0105] It should be understood that a callback can be simply understood as feedback on the result of file transfer. The callback configuration can be the categories included in the feedback content. For example, the callback configuration can include multiple configurations such as file transfer start time, file transfer target node, file output local node, and file transfer time.

[0106] It should be noted that users can select multiple callback names, or select the same callback name to generate multiple callback messages with different configurations, which can be determined according to the callback configuration that the user pre-sets on the local node when uploading the target file.

[0107] It should be noted that when uploading files, users can pre-configure callback information locally, including callback name, callback method, callback connection timeout, callback communication timeout, callback address, callback header, callback body, callback retry rules, etc.

[0108] Furthermore, users can register a transfer task by calling the task registration interface provided by the local node, providing the transfer mode, target node name, callback name (optional), source file path, and target file path. The scheduling module will then allocate resources in the background to asynchronously process the file transfer. Since the transfer task is registered by calling an interface, users can call it locally or remotely.

[0109] Step A60: Generate a spliced ​​message based on the task to be transmitted and the callback configuration.

[0110] Understandably, at this point, the task to be transmitted has been transmitted to the target node, and the relevant transmission information of the task from the user's upload to the local machine to the transmission from the local machine to the target node is recorded, such as transmission time, start time, transmission speed and other related information.

[0111] It should be understood that the relevant information required for the file transfer process during the user callback is determined based on the callback configuration.

[0112] In practice, based on the task to be transmitted, the file transmission start time, target node, local node, transmission end time, and file transmission time can be known. The callback configuration includes the file transmission start time, target node, and transmission end time. Then, the file transmission start time, target node, and transmission end time of the task to be transmitted are concatenated to generate a concatenated message.

[0113] Step A70: Perform a callback on the spliced ​​message according to the preset number of callbacks.

[0114] Understandably, the preset number of callbacks allows users to select the number of callbacks and the callback object when they determine that a callback is needed. The callback object can be the same or different, and the same callback object can be called multiple times. The content of each callback can be the same or different.

[0115] It is important to emphasize that after file verification and renaming / authorization are completed, the file transfer is essentially finished. If the parameters submitted by the user during task registration include a callback name, the callback will be executed. The callback configuration is retrieved from the local node's configuration using the callback name, the callback message is constructed according to the callback configuration, and an HTTP POST or GET request is executed.

[0116] It should be further noted that callback configurations can have the same name, thus allowing multiple addresses to be configured to execute multiple callbacks. If a callback fails, it can also be automatically retried according to user configuration. The message header and body in the callback support user-defined formats and dynamic generation, and can be supplemented using any field information from the transmission task.

[0117] In this embodiment, by having the user pre-configure callbacks with an unlimited number of times and callback objects on the local node, the automatic callback after transmission generates messages, providing real-time and effective feedback on the file transmission status to the user, thereby improving the user experience.

[0118] The above are merely feasible implementation methods after step S40 provided in this embodiment. This embodiment does not specifically limit the specific implementation method of step S40.

[0119] It should be noted that the file transfer process based on steps 10-70 above can be referenced. Figure 8 In the diagram, after the user uploads the target file and generates a task to be transferred, it waits for an idle thread to execute the task. Then, it checks whether there are available thread resources and whether there are any pending transfer tasks in the task list. If there are idle threads and pending transfer tasks, the task to be transferred is assigned to the idle thread and the task status of the task to be transferred in the task list is updated. After successful assignment, the target node is selected, and data is transferred according to the transfer mode selected by the user. When the transfer mode is PUT mode, node authentication is performed first and the target node interface is called. According to the user's storage requirements for the target file, the corresponding file directory for storage requirements is created on the target node, and then PUT transfer is performed. After the transfer, the target node interface is called to verify the file. After successful verification, the file to be transferred that has been verified by the target node is renamed according to the user's requirements.

[0120] When the transmission mode is GET, a directory is created locally according to the user's requirements, and then a GET transmission is performed. The target node interface is called to verify the file received locally. After successful verification, the file is renamed locally to meet the user's requirements.

[0121] After successful transmission verification, the user selects whether to initiate a callback. If the user does not select a callback, the transmission ends. If the user selects a callback, the corresponding callback method is selected based on the transmission method. The callback method can include GET callback, which generates a concatenated message based on the callback configuration and calls the interface via GET; or POST callback, which generates a concatenated message based on the callback configuration and calls the interface via POST.

[0122] This embodiment provides a file transfer method that directly generates a transfer task based on the user-uploaded file to be transferred. After determining the target node based on the user-uploaded file and the configured target node information, it performs point-to-point asynchronous file transfer between the local machine and the target node based on the transfer task and idle threads. This method can save time and thread resources waiting for file transfer, improve file transfer efficiency, and ensure the stability of data transmission.

[0123] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 9 In step S30, the file transfer method further includes steps S31 to S33:

[0124] Step S31: Based on the transmission mode, send the file to be transmitted to the target node and obtain the user-specified refresh threshold.

[0125] It should be noted that users need to obtain the transfer progress during file transfer, which needs to be achieved by monitoring I / O. There are two traditional monitoring methods: timed calculation and real-time calculation. Timed calculation is not accurate enough and is easily affected by the network. Real-time calculation affects efficiency. The cumulative update operations during each I / O will seriously affect the transfer efficiency, and frequent updates will also bring higher risks.

[0126] Understandably, during file transfer, users can set or input in real time the timeframe at which they want to obtain the file transfer progress, i.e., set a refresh threshold.

[0127] Step S32: Obtain the refresh point based on the user-specified refresh threshold, and identify the file transfer progress based on the refresh point.

[0128] Understandably, the refresh point is obtained based on the user-specified refresh threshold, and the file transfer progress is identified based on the refresh point. This can be a sliding I / O, which allows the refresh threshold to slide synchronously with the transfer I / O, thereby achieving real-time updates of the refresh point.

[0129] It should be understood that identifying file transfer progress based on refresh points allows for a reduction in refresh frequency as needed by the user, improving transfer efficiency while accurately reflecting the current transfer progress and meeting user requirements.

[0130] In practical implementation, a schematic diagram of sliding I / O progress monitoring can be referenced. Figure 10 Users can specify a refresh threshold, which slides synchronously with the transmission I / O. When the tail of the refresh threshold leaves the previous refresh point, the position corresponding to the head of the refresh threshold at this time is the new refresh point. The refresh point position is updated and a progress refresh is performed.

[0131] Step S33: When the file transfer progress indicates that the transfer is complete, call the target node interface to perform file verification.

[0132] It should be noted that file verification for calling the target interface can be analyzed in both PUT and GET modes. PUT mode can be simply understood as file sending, and GET mode can be simply understood as file receiving. When sending a file, the local node is the sending node. The digest of the file uploaded by the user at the sending node is extracted, and then the digest of the file received by the target node is extracted. By comparing the digests extracted by the two nodes, it can be determined whether a file error occurred during the file transmission process.

[0133] Furthermore, when receiving a file, the local node acts as the receiving node, extracts a summary from the received file, and then calls the target node's interface to extract a summary from the user-uploaded file at the target node. The summaries extracted by the two nodes are compared, and if they are the same or the similarity meets the standard, the file verification is completed.

[0134] In one feasible implementation, step S33 may include steps A331 to A333:

[0135] Step A331: When the file transfer progress indicates that the transfer is complete, the target node interface is invoked to identify the file to be transferred.

[0136] Understandably, regardless of whether it is GET or PUT mode, after the file transfer is completed, the target node's interface needs to be called to extract the digest of the target file at the target node.

[0137] It should be understood that the file to be identified in PUT mode can be the target file to be transmitted to the target node; in GET mode, the file to be identified in PUT mode can be the target file uploaded by the user.

[0138] It should be noted that before sending the file to be transmitted to the target node based on the transmission mode and calling the target node interface for file verification, the process further includes: obtaining the target node authentication information configured locally; establishing a file transmission connection with the target node based on the target node authentication information according to a preset communication protocol, and sending the file to be transmitted to the target node based on the file transmission connection.

[0139] Understandably, the security of each target node needs to be verified before transferring files between the local node and the target node.

[0140] It should be understood that the local node can pre-configure the verification information of the target node for file transfer. When the user selects the target node, the verification information is sent to the target node at the same time as the file to be transferred for verification, thus ensuring the security of file transfer.

[0141] It should be noted that target node authentication is required before the actual transfer to ensure node security. Authentication information comes from the target node information configured on the local node during the preparation phase. After successful authentication, an SSH connection will be established between the nodes. Then, a directory for storing files will be created locally or on the target node, and encrypted file push or pull will be performed via the SSH protocol.

[0142] Step A332: Classify the files to be transferred according to their size to obtain file categories.

[0143] It should be noted that file categories can include Category 1 files and Category 2 files. A file size threshold is set. Files smaller than the threshold are considered Category 1 files, and files larger than or equal to the threshold are considered Category 2 files. Category 1 files can be simply understood as small files, and Category 2 files can be simply understood as large files.

[0144] Understandably, when transmitting data, especially large files, the transmission process is prone to errors, leading to file transfer failures.

[0145] It should be noted that in traditional file transfer processes, users can see and manipulate the data being transferred, which can easily lead to transfer failures. This invention employs a triple control mechanism to avoid this problem: Data name control: Data in transfer uses a random temporary name instead of a formal name. Data access control: Based on ACL (Access Control List), the data in transfer is placed in a special state where it is unreadable, unwriteable, and unexecutable by other users. Data visibility control: The data in transfer is made invisible, hiding the data during transmission. After the transfer is complete, the temporary data is renamed to the formal data, and its access control permissions and visibility are modified. This minimizes user error and ensures stable and reliable transmission.

[0146] Step A333: Extract the file digest of the file to be transmitted based on the file category, and perform verification based on the file digest.

[0147] It should be noted that the step of extracting the file digest of the file to be transmitted based on the file category and performing verification based on the file digest includes: when the file category is a type I file, obtaining a file digest based on the file to be transmitted and performing verification based on the file digest; when the file category is a type II file, sampling data from the file to be transmitted to obtain sample data, extracting sample digests from each sample data and merging the sample digests sequentially to obtain a file digest and a sample merge position; and performing verification based on the file digest and the sample merge position.

[0148] Furthermore, a schematic diagram of a uniformly randomized sampling summary can be found in [reference]. Figure 11 , Figure 11 Abstract extraction methods can include data sampling, employing a uniform random sampling strategy to obtain representative data samples, dividing the data into multiple intervals, randomly selecting a segment of random length from each interval as a sample, and recording the sample position; using the national cryptographic abstracting algorithm SM3 to calculate an abstract for each sample; concatenating all sample abstracts sequentially into a single sample, and then calculating the abstract for that sample, with the final deliverables being the sample abstract and sample position; using the same sample position and method to calculate abstracts for the data to be compared, and determining whether the data are consistent by comparing whether the abstracts are the same.

[0149] In this embodiment, by comprehensively considering efficiency and user needs, the sliding I / O progress monitoring model can accurately reflect the transmission progress while ensuring efficient transmission.

[0150] The above are merely feasible implementation methods for step S33 provided in this embodiment. This embodiment does not specifically limit the specific implementation method of step S33.

[0151] This embodiment provides a file transfer method that improves transmission stability and provides efficient file integrity verification through a triple control mechanism. Files are layered according to size. For small files, a complete file digest is directly calculated; for large files, a uniform random sampling digest algorithm is used to calculate the digest, which can significantly improve efficiency while ensuring accuracy.

[0152] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the file transmission method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0153] This application also provides a file transfer device, please refer to... Figure 12 The file transfer device includes:

[0154] The acquisition module 10 is used to acquire the target file uploaded by the user and generate a transmission task based on the target file. The transmission task includes the file to be transmitted, target node information, and transmission mode.

[0155] Node election module 20 is used to elect a target node based on the target node information when there is an idle thread.

[0156] The file transfer module 30 is used to send the file to be transferred to the target node based on the transfer mode, and to call the target node interface to perform file verification;

[0157] The file transfer module 30 is also used to complete the transfer task when the file verification is successful.

[0158] The node election module 20 is also used to determine the target cluster based on the target node information when there are idle threads.

[0159] Randomly hash and filter multiple nodes in the target cluster to obtain initial nodes;

[0160] When the initial node is in a connectable state, the initial node is used as the target node.

[0161] The file transfer module 30 is also used to send the file to be transferred to the target node based on the transfer mode, and to obtain the refresh threshold specified by the user.

[0162] A refresh point is obtained based on the user-specified refresh threshold, and the file transfer progress is identified based on the refresh point.

[0163] When the file transfer progress indicates that the transfer is complete, the target node interface is invoked to perform file verification.

[0164] The file transfer module 30 is also used to obtain the target node authentication information configured locally;

[0165] Based on a preset communication protocol, a file transfer connection is established with the target node according to the target node authentication information, so as to send the file to be transferred to the target node based on the file transfer connection.

[0166] The file transfer module 30 is also used to call the target node interface to identify the file to be transferred when the file transfer progress is complete.

[0167] The files to be transmitted are classified according to their size to obtain file categories;

[0168] The file digest of the file to be transmitted is extracted based on the file category, and the file digest is used for verification.

[0169] The file transfer module 30 is further configured to, when the file category is a type of file, obtain a file digest based on the file to be transferred, and perform verification based on the file digest;

[0170] When the file category is Class II, data sampling is performed on the file to be transmitted to obtain sample data. Sample summaries are extracted from each sample data and merged sequentially to obtain a file summary and sample merge position. Verification is performed based on the file summary and the sample merge position.

[0171] The file transfer module 30 is also used to obtain the callback configuration based on the callback name;

[0172] Generate a spliced ​​message based on the task to be transmitted and the callback configuration;

[0173] The concatenated message is callbackd according to a preset number of callbacks.

[0174] The file transfer device provided in this application, employing the file transfer method described in the above embodiments, can solve the technical problem that current file transfer requires synchronous calls, wasting thread resources. Compared with the prior art, the beneficial effects of the file transfer device provided in this application are the same as those of the file transfer method provided in the above embodiments, and other technical features in the file transfer device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0175] This application provides a file transfer device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the file transfer method in Embodiment 1 above.

[0176] The following is for reference. Figure 13 The diagram illustrates a structural schematic of a file transfer device suitable for implementing embodiments of this application. The file transfer device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 13 The file transfer device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0177] like Figure 13 As shown, the file transfer device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the file transfer device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the file transfer device to communicate wirelessly or wiredly with other devices to exchange data. Although file transfer devices with various systems are shown in the figures, it should be understood that it is not required to implement or possess all of the systems shown. More or fewer systems may be implemented alternatively.

[0178] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0179] The file transfer device provided in this application, employing the file transfer method described in the above embodiments, can solve the technical problem that current file transfer requires synchronous calls, wasting thread resources. Compared with the prior art, the beneficial effects of the file transfer device provided in this application are the same as those of the file transfer method provided in the above embodiments, and other technical features of this file transfer device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0180] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0181] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0182] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the file transfer method described in the above embodiments.

[0183] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0184] The aforementioned computer-readable storage medium may be included in a file transfer device; or it may exist independently and not be assembled into a file transfer device.

[0185] The aforementioned computer-readable storage medium carries one or more programs. When the aforementioned one or more programs are executed by the file transfer device, the file transfer device causes the file transfer device to: acquire a target file uploaded by the user; generate a transfer task based on the target file, the transfer task including the file to be transferred, target node information, and transfer mode; when an idle thread exists, elect a target node based on the target node information; send the file to be transferred to the target node based on the transfer mode, and call the target node interface to perform file verification; and complete the transfer task when the file verification is successful.

[0186] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0187] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0188] The modules described in the embodiments of the present application may be implemented in software or hardware, wherein the name of a module does not necessarily limit the unit itself.

[0189] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described file transfer method. This solves the technical problem that current file transfer methods require synchronous calls, wasting thread resources. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the file transfer method provided in the above embodiments, and will not be repeated here.

[0190] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the file transfer method described above.

[0191] The computer program product provided in this application can solve the technical problem that current file transfer requires synchronous calls, which wastes thread resources. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the file transfer method provided in the above embodiments, and will not be repeated here.

[0192] The above description is only part of the embodiments of the present application and does not limit the patent scope of the present application. All equivalent structural transformations made by using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A file transfer method, characterized in that, The file transfer method includes: Obtain the target file uploaded by the user, and generate a transmission task based on the target file. The transmission task includes the file to be transmitted, target node information, and transmission mode. When an idle thread exists, a target node is selected based on the target node information. Based on the transmission mode, the file to be transmitted is sent to the target node, and the user-specified refresh threshold is obtained; A refresh point is obtained based on the user-specified refresh threshold, and the file transfer progress is identified based on the refresh point. When the file transfer progress indicates that the transfer is complete, the target node interface is invoked to identify the file to be transferred; The files to be transmitted are classified according to their size to obtain file categories; Extract the file digest of the file to be transmitted based on the file category, and perform verification based on the file digest; Upon successful file verification, the transmission task is completed.

2. The file transfer method as described in claim 1, characterized in that, The step of selecting a target node based on the target node information when an idle thread exists includes: When idle threads exist, the target cluster is determined based on the target node information; Randomly hash and filter multiple nodes in the target cluster to obtain initial nodes; When the initial node is in a connectable state, the initial node is used as the target node.

3. The file transfer method as described in claim 1, characterized in that, Before sending the file to be transmitted to the target node based on the transmission mode and calling the target node interface to perform file verification, the method further includes: Obtain the target node authentication information configured locally; Based on a preset communication protocol, a file transfer connection is established with the target node according to the target node authentication information, so as to send the file to be transferred to the target node based on the file transfer connection.

4. The file transfer method as described in claim 1, characterized in that, The step of extracting a file digest of the file to be transmitted based on the file category and performing verification based on the file digest includes: When the file category is a type of file, a file digest is obtained based on the file to be transmitted, and verification is performed based on the file digest. When the file category is Class II, data sampling is performed on the file to be transmitted to obtain sample data. Sample summaries are extracted from each sample data and merged sequentially to obtain a file summary and sample merge position. Verification is performed based on the file summary and the sample merge position.

5. The file transfer method as described in claim 1, characterized in that, The task to be transmitted also includes a callback name; After the file verification is successful and the transmission task is completed, the process further includes: Get the callback configuration based on the callback name; Generate a spliced ​​message based on the task to be transmitted and the callback configuration; The concatenated message is callbackd according to a preset number of callbacks.

6. A file transfer device, characterized in that, The file transfer device includes: The acquisition module is used to acquire the target file uploaded by the user and generate a transmission task based on the target file. The transmission task includes the file to be transmitted, target node information, and transmission mode. The node election module is used to elect a target node based on the target node information when there is an idle thread. The file transfer module is used to send the file to be transferred to the target node based on the transfer mode, and obtain a user-specified refresh threshold; obtain a refresh point based on the user-specified refresh threshold, and identify the file transfer progress based on the refresh point; when the file transfer progress indicates that the transfer is complete, call the target node interface to identify the file to be transferred; classify the file to be transferred according to its size to obtain a file category; extract a file digest of the file to be transferred based on the file category, and perform verification based on the file digest; The file transfer module is also used to complete the transfer task when the file verification is successful.

7. A file transfer device, characterized in that, The device includes: a memory, a processor, and a file transfer program stored on the memory and executable on the processor, the file transfer program being configured to implement the file transfer method as described in any one of claims 1 to 5.

8. A storage medium, characterized in that, The storage medium stores a file transfer program, which, when executed by a processor, implements the file transfer method as described in any one of claims 1 to 5.

Images