A data transmission method, system, device, computing device and storage medium
By using multiple coroutines on the worker side and splitting tasks on the control side, the problem of low data transmission efficiency in the existing technology is solved, efficient transmission and synchronization processing of different data storage methods is realized, and the machine is improved The efficiency of learning.
Patent Information
- Application Number
- CN202110851374.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-07-27
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2041-07-27
AI Technical Summary
In the prior art, the data transmission method is single and inefficient, which affects the efficiency of machine learning, especially when a large amount of data is transmitted across regions.
By setting up multiple coroutines on the worker side, using file attribute information to determine the appropriate file reading method, file transfer of different data storage methods is realized, and tasks are split on the control side for parallel processing, improving transmission efficiency.
It improves the efficiency and flexibility of file transfer, improves the efficiency of machine learning, and realizes synchronous transmission through the diversity of coroutines and parallel processing, reducing the probability of transmission failure.
Smart Images

Figure CN113596148B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the technical field of data transmission, and in particular, to a data transmission method, system, apparatus, computing device and storage medium. Background Art
[0002] With the rapid development of computer technology, the application of machine learning is becoming more and more extensive. Data is an important foundation of machine learning. The richer the types and quantity of data, the better the learning effect of machine learning.
[0003] Large amounts of data are often stored in different areas, and different types of data have different data storage methods. Therefore, in order to ensure the effectiveness of machine learning, it is often necessary to first obtain data from different areas and then transfer the data to the target address for machine learning.
[0004] However, the data transmission method in the existing technology has the problems of single transmission method and low transmission efficiency, which affects the efficiency of machine learning. Summary of the invention
[0005] The embodiments of the present disclosure at least provide a data transmission method, system, apparatus, computing device and storage medium.
[0006] In a first aspect, an embodiment of the present disclosure provides a data transmission method, which is applied to a worker, including:
[0007] Acquire a task to be transmitted sent by the control end Master; the task to be transmitted is obtained by the control end Master splitting the acquired target task, and the task to be transmitted includes file attribute information of at least one file to be transmitted;
[0008] Based on the file attribute information, determine the file reading information of each file to be transferred corresponding to the task to be transferred;
[0009] Using a plurality of pre-set coroutines based on the file reading information, respectively obtain the to-be-transferred files matching each of the file reading information;
[0010] Each of the files to be transferred in the tasks to be transferred is transferred to a corresponding target address by using the multiple coroutines.
[0011] This implementation method can determine the file reading method that matches the storage method of each file to be transmitted based on the file reading information corresponding to each file to be transmitted, so that the transmission of files to be transmitted stored in different data storage methods can be realized, thereby improving the transmission efficiency of files or data, and is conducive to improving the efficiency of machine learning. In addition, a coroutine can transmit a file to be transmitted, and each coroutine is configured with multiple file reading methods. Therefore, by setting multiple coroutines in the worker end Worker, it is not only possible to use the file reading method that matches different data storage methods to read the file to be transmitted, but also to realize the synchronous transmission of multiple files to be transmitted after reading the file to be transmitted, which improves the diversity and flexibility of file reading on the one hand, and improves the efficiency of data transmission on the other hand, thereby improving the execution speed and efficiency of the task to be transmitted. In addition, since the task to be transmitted is obtained by splitting the target task obtained by the control end Master, by splitting the target task, it is possible to send the split multiple tasks to be transmitted to multiple worker end Workers, and use multiple worker end Workers to synchronously process multiple tasks to be transmitted, further improving the speed and efficiency of file transmission.
[0012] In a possible implementation, determining the file reading information of the file to be transferred based on the file attribute information includes:
[0013] Based on the file attribute information, determining a file storage mode and a file location of the file to be transferred;
[0014] Determining a file reading method for the file to be transmitted based on a file storage method for the file to be transmitted;
[0015] The file reading mode and the file position are used as the file reading information of the file to be transmitted.
[0016] Different file storage methods correspond to different file reading methods. By determining the file storage method, the file reading method that matches the method can be accurately determined. Based on the determined file location, the file to be transferred can be accurately obtained. Then, the file reading method and the file location are used as file reading information, which enables the worker-side worker to use the information to accurately read the file to be transferred, thereby completing the transmission of the file to be transferred.
[0017] In a possible implementation, the using the multiple coroutines to transfer each of the to-be-transferred files in the to-be-transferred tasks to a corresponding target address includes:
[0018] For each coroutine in the multiple coroutines, using the coroutine to transmit the corresponding file to be transmitted to the target address;
[0019] In the case where the transmission of the file to be transmitted fails, determining the number of transmission failures of the file to be transmitted;
[0020] When the number of transmission failures is less than a preset failure threshold, the file to be transmitted is transmitted again.
[0021] By setting a preset failure threshold, it is possible to actively retransmit files that have failed to be transmitted, thereby reducing the probability of file transmission failure.
[0022] In a possible implementation manner, after determining the number of transmission failures of the to-be-transmitted file, the method further includes:
[0023] When the number of transmission failures is greater than the preset failure threshold, generating a transmission failure record of the file to be transmitted;
[0024] The transmission failure record is sent to the client corresponding to the control end Master, so that the client displays the transmission failure record.
[0025] When the number of transmission failures is greater than the preset failure threshold, it means that the active retransmission of the file to be transmitted cannot complete its transmission, and it is necessary to make certain adjustments manually to complete the transmission, such as adjusting the file to be transmitted, replacing the transmission worker, etc. Therefore, by generating a transmission failure record and sending it to the client, the user can obtain the result of the transmission failure in time on the client, and make certain adjustments based on the result to complete the retransmission of the file to be transmitted, thereby ensuring that the file to be transmitted can be finally transmitted to the target address.
[0026] In a possible implementation manner, before sending the transmission failure record to the client corresponding to the control end Master, the method further includes:
[0027] Determine the file transmission information of each file to be transmitted; the file transmission information includes at least one of the transmission time information, the file name, and the transmission task creator information corresponding to the file to be transmitted;
[0028] Determine a file transfer result for each of the files to be transferred;
[0029] Determining a task execution result of the task to be transferred based on a file transfer result of each of the files to be transferred;
[0030] The step of sending the transmission failure record to the client corresponding to the control end Master comprises:
[0031] The transmission failure record, the file transmission information of each file to be transmitted and the task execution result are sent to the client corresponding to the control end Master.
[0032] By sending transmission failure records, file transfer information and task execution results to the client, not only can the execution results of the task to be transferred be displayed on the client, but also the transmission details of the task to be transferred, such as transmission time information, information about the creator of the task to be transferred, etc. Therefore, the transfer task can be managed based on the above information, for example, a certain file to be transferred can be re-transmitted, the file to be transferred corresponding to the task to be transferred can be downloaded, etc., which is conducive to improving the efficiency of data or file transmission and improving the efficiency of machine learning.
[0033] In a possible implementation manner, after determining the task execution result of the task to be transmitted, the method further includes:
[0034] Treat the task to be transmitted as an executed task;
[0035] Determine a task execution result of each task to be transmitted corresponding to the target task except the executed task;
[0036] Based on the task execution results, determining the task completion order of the executed tasks;
[0037] In a case where the task completion order matches the number of tasks to be transmitted corresponding to the target task, the task status of the target task is updated to a completion status.
[0038] Based on the determined task completion order, the completion progress of the target task corresponding to the task to be executed can be accurately determined, and the status of the target task can be updated in a timely manner.
[0039] In a possible implementation, the method further includes:
[0040] Obtaining a retransmission task instruction sent by the control end Master, and determining a retransmission file from the to-be-transmitted files corresponding to the to-be-transmitted task based on the retransmission task instruction;
[0041] The retransmission file is transmitted.
[0042] Based on the received retransmission task instruction, the retransmission of the to-be-transmitted file (ie, the retransmission file) that has failed in transmission can be completed, thereby ensuring that the retransmission file can be successfully transmitted to the target address in the end.
[0043] In a possible implementation, the method further includes:
[0044] Obtaining a transmission suspension instruction for the file to be transmitted;
[0045] Based on the transmission suspension instruction, determining a transmission suspension file from the to-be-transmitted files corresponding to the to-be-transmitted tasks;
[0046] The transmission of the transmission-pause file is suspended.
[0047] The transmission pause instruction is generated according to the transmission pause request actively initiated by the user on the client. Based on the obtained transmission pause instruction, the worker can realize the transmission control of the files to be transmitted, that is, the user's transmission control of the files to be transmitted is realized.
[0048] In a possible implementation, the method further includes:
[0049] Obtaining a transmission cancellation instruction for the task to be transmitted;
[0050] In response to the cancel transmission instruction, the transmission operation of the to-be-transmitted file corresponding to the to-be-transmitted task is stopped.
[0051] The cancel transfer instruction is also generated based on the cancel transfer request actively initiated by the user on the client. Based on the obtained cancel transfer instruction, the transmission control of the to-be-transferred files corresponding to the transfer task can be realized, that is, the execution control of the transfer task is realized, and then, the user's control over the transfer task is indirectly realized.
[0052] In a second aspect, an embodiment of the present disclosure further provides a data transmission system, including the control end Master described in any one of the first aspects above, and at least one worker end Worker described in the first aspect above;
[0053] The control end Master is used to obtain a target task; split the target task into multiple tasks to be transmitted; send each of the tasks to be transmitted to each of the worker ends Worker in at least one of the worker ends Worker; the tasks to be transmitted include file attribute information of at least one file to be transmitted;
[0054] The worker end Worker is used to obtain the task to be transferred sent by the control end Master; and based on the file attribute information, determine the file reading information of each file to be transferred corresponding to the task to be transferred; and based on the file reading information, respectively obtain the files to be transferred that match each file reading information; and transfer each file to be transferred in the task to be transferred to the corresponding target address.
[0055] In a possible implementation, the data transmission system further includes a task listener, wherein:
[0056] The task listener is used to obtain the task status of the target task during the process of the worker-side Worker executing the target task; and send the task status to the client corresponding to the control-side Master so that the client displays the task status.
[0057] In a possible implementation manner, the task listener is further used to obtain a file retransmission request initiated by the client for the task to be transmitted; based on the file retransmission request, determine a retransmission file corresponding to the file retransmission request; based on the retransmission file, generate a file retransmission task and send it to the control end Master;
[0058] The control end Master is used to receive the file retransmission task; generate a retransmission task instruction based on the file retransmission task; and send the retransmission task instruction to the worker end Worker that matches the retransmission file;
[0059] The worker end Worker is used to receive the retransmission task instruction; based on the retransmission task instruction, determine the retransmission file from the to-be-transmitted files corresponding to the to-be-transmitted task; and transmit the retransmission file.
[0060] In a possible implementation manner, the task listener is further used to obtain a transmission pause request initiated by the client for the task to be transmitted; based on the transmission pause request, determine a transmission pause file corresponding to the transmission pause request; based on the transmission pause file, generate a transmission pause instruction; and send the transmission pause instruction to the control end Master and the worker end Worker matching the transmission pause file;
[0061] The control end Master is used to receive the transmission suspension instruction; in response to the transmission suspension instruction, suspend sending new tasks to be transmitted to the worker end Worker matching the transmission suspension file;
[0062] The worker end Worker is used to receive the transmission suspension instruction; based on the transmission suspension instruction, determine the transmission suspension file from the to-be-transmitted files corresponding to the to-be-transmitted tasks; and suspend the transmission of the transmission suspension file.
[0063] In a possible implementation manner, the task listener is further used to obtain a transmission cancellation request initiated by the client for the task to be transmitted; determine a transmission cancellation task that matches the transmission cancellation request from multiple tasks to be transmitted; generate a transmission cancellation instruction based on the transmission cancellation task; and send the transmission cancellation instruction to the control end Master and the worker end Worker that matches the transmission cancellation task;
[0064] The control end Master is used to receive the cancel transmission instruction; in response to the cancel transmission instruction, suspend sending new tasks to be transmitted to the worker end Worker matching the cancel transmission task;
[0065] The worker end Worker is used to receive the cancel transmission instruction; in response to the cancel transmission instruction, stop the transmission operation of the to-be-transmitted file corresponding to the to-be-transmitted task.
[0066] In a possible implementation manner, the control end Master is used to parse the target task after acquiring the target task, and determine the file attribute information of each to-be-transferred file corresponding to the target task;
[0067] Based on the file attribute information of each file to be transmitted, the files to be transmitted are divided into a plurality of file groups to be transmitted;
[0068] Based on each of the to-be-transmitted file groups, a to-be-transmitted task corresponding to each of the to-be-transmitted file groups is generated, and the to-be-transmitted task is sent to the worker end Worker.
[0069] In a third aspect, an embodiment of the present disclosure further provides a data transmission device, including:
[0070] The first acquisition module is used to acquire the task to be transmitted sent by the control end Master; the task to be transmitted is obtained by the control end Master splitting the acquired target task, and the task to be transmitted includes file attribute information of at least one file to be transmitted;
[0071] A first determining module, configured to determine, based on the file attribute information, file reading information of each file to be transmitted corresponding to the task to be transmitted;
[0072] A second acquisition module is used to respectively acquire a file to be transferred that matches each file reading information based on the file reading information by using a plurality of pre-set coroutines;
[0073] The transmission module is used to utilize the multiple coroutines to transmit each of the files to be transmitted in the tasks to be transmitted to the corresponding target address.
[0074] In a possible implementation manner, the first determining module is used to determine a file storage mode and a file location of the file to be transferred based on the file attribute information;
[0075] Determining a file reading method for the file to be transmitted based on a file storage method for the file to be transmitted;
[0076] The file reading mode and the file position are used as the file reading information of the file to be transmitted.
[0077] In a possible implementation, the transmission module is used to transmit the corresponding file to be transmitted to the target address by using each coroutine in the multiple coroutines;
[0078] In the case where the transmission of the file to be transmitted fails, determining the number of transmission failures of the file to be transmitted;
[0079] When the number of transmission failures is less than a preset failure threshold, the file to be transmitted is transmitted again.
[0080] In a possible implementation manner, the transmission module is further configured to, after determining the number of transmission failures of the file to be transmitted, generate a transmission failure record of the file to be transmitted if the number of transmission failures is greater than the preset failure threshold;
[0081] The transmission failure record is sent to the client corresponding to the control end Master, so that the client displays the transmission failure record.
[0082] In a possible implementation manner, the device further includes a second determining module:
[0083] The second determination module is used to determine the file transmission information of each file to be transmitted before sending the transmission failure record to the client corresponding to the control end Master; the file transmission information includes at least one of the transmission time information, file name, and transmission task creator information corresponding to the file to be transmitted;
[0084] Determine a file transfer result for each of the files to be transferred;
[0085] Determining a task execution result of the task to be transferred based on a file transfer result of each of the files to be transferred;
[0086] The transmission module is further used to send the transmission failure record, the file transmission information of each file to be transmitted and the task execution result to the client corresponding to the control end Master.
[0087] In a possible implementation, the device further includes:
[0088] An updating module, configured to, after determining the task execution result of the task to be transmitted, treat the task to be transmitted as an executed task;
[0089] Determine a task execution result of each task to be transmitted corresponding to the target task except the executed task;
[0090] Based on the task execution results, determining the task completion order of the executed tasks;
[0091] In a case where the task completion order matches the number of tasks to be transmitted corresponding to the target task, the task status of the target task is updated to a completion status.
[0092] In a possible implementation manner, the first acquisition module is further configured to acquire a retransmission task instruction sent by the control end Master, and determine a retransmission file from the to-be-transmitted file corresponding to the to-be-transmitted task based on the retransmission task instruction;
[0093] The transmission module is also used to transmit the retransmission file.
[0094] In a possible implementation manner, the first acquisition module is further configured to acquire a transmission suspension instruction for the file to be transmitted;
[0095] Based on the transmission suspension instruction, determining a transmission suspension file from the to-be-transmitted files corresponding to the to-be-transmitted tasks;
[0096] The transmission module is further used to suspend the transmission of the suspended transmission file.
[0097] In a possible implementation manner, the first acquisition module is further configured to acquire a transmission cancellation instruction for the task to be transmitted;
[0098] The transmission module is further configured to stop the transmission operation of the to-be-transmitted file corresponding to the to-be-transmitted task in response to the cancel transmission instruction.
[0099] In a fourth aspect, an optional implementation of the present disclosure further provides a computer device, a processor, and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the processor is used to execute the machine-readable instructions stored in the memory, and when the machine-readable instructions are executed by the processor, the machine-readable instructions perform the steps of the above-mentioned first aspect, or any possible implementation of the first aspect.
[0100] In a fifth aspect, an optional implementation of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed, the steps of the above-mentioned first aspect, or any possible implementation of the first aspect are executed.
[0101] For a description of the effects of the above-mentioned data transmission system, apparatus, computer device, and computer-readable storage medium, please refer to the description of the above-mentioned data transmission method, which will not be repeated here.
[0102] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0103] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following is a brief introduction to the drawings required for use in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can also be obtained based on these drawings without creative work.
[0104] Figure 1 A flow chart of a data transmission method applied to a worker provided by an embodiment of the present disclosure is shown;
[0105] Figure 2 A specific implementation flow chart of each file to be transmitted corresponding to a transmission target task provided by an embodiment of the present disclosure is shown;
[0106] Figure 3 A schematic diagram of a data transmission system provided by an embodiment of the present disclosure is shown;
[0107] Figure 4 A schematic diagram of a data transmission device provided by an embodiment of the present disclosure is shown;
[0108] Figure 5 A schematic diagram of the structure of a computer device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0109] In order to make the purpose, technical scheme and advantages of the embodiments of the present disclosure clearer, the technical scheme in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure is not intended to limit the scope of the present disclosure claimed for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present disclosure.
[0110] In addition, the terms "first", "second", etc. in the description and claims of the embodiments of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein.
[0111] The "multiple or several" mentioned in this article refers to two or more. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.
[0112] Research has found that with the rapid development of computer technology, the application of machine learning is becoming more and more extensive. Data is an important foundation for machine learning. The richer the types and quantity of data, the better the learning effect of machine learning.
[0113] Large amounts of data are often stored in different areas, and different types of data have different data storage methods. Therefore, in order to ensure the effectiveness of machine learning, it is often necessary to first obtain data from different areas and then transfer the data to the target address for machine learning.
[0114] However, the data transmission method in the existing technology has the problems of single transmission method and low transmission efficiency, which affects the efficiency of machine learning.
[0115] Based on the above research, the present disclosure provides a data transmission method, system, device, computing device and storage medium. Based on the determined file reading information corresponding to each file to be transmitted, the file reading method matching the storage method of each file to be transmitted can be determined. Therefore, the transmission of files to be transmitted stored in different data storage methods can be realized, thereby improving the transmission efficiency of files or data, and is conducive to improving the efficiency of machine learning. In addition, a coroutine can transmit a file to be transmitted, and each coroutine is configured with multiple file reading methods. Therefore, by setting multiple coroutines in the worker end Worker, not only can the file to be transmitted be read using the file reading method matching different data storage methods, but also after reading the file to be transmitted, the synchronous transmission of multiple files to be transmitted can be realized, which improves the diversity and flexibility of file reading on the one hand, and improves the efficiency of data transmission on the other hand, thereby improving the execution speed and efficiency of the task to be transmitted. In addition, since the task to be transmitted is obtained by splitting the target task obtained by the control end Master, by splitting the target task, it is possible to send the split multiple tasks to be transmitted to multiple worker end Workers, and use multiple worker end Workers to synchronously process multiple tasks to be transmitted, further improving the speed and efficiency of file transmission.
[0116] The defects existing in the above solutions are the results obtained by the inventor after practice and careful research. Therefore, the discovery process of the above problems and the solutions proposed by the present disclosure for the above problems below should be the contributions made by the inventor to the present disclosure during the disclosure process.
[0117] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.
[0118] It should be noted that the specific terms mentioned in the embodiments of the present disclosure include:
[0119] Master-Worker Framework: The controller-worker framework is a structure that uses multi-threading for data processing. Multiple worker-side Worker (hereinafter referred to as Worker) processes collaborate to process user requests. The control-side Master (hereinafter referred to as Master) process is responsible for maintaining the Worker process and integrating the final processing results.
[0120] Worker: Used to actually process a task.
[0121] Master: used for task allocation and synthesis of final results.
[0122] Lustre: A parallel distributed file storage system commonly used in large computer clusters and supercomputers.
[0123] Ceph: is a distributed file storage system that adds replication and fault tolerance capabilities while maintaining POSIX compatibility.
[0124] FTP: File Transfer Protocol, a set of standard protocols for transferring files on the Internet.
[0125] To facilitate understanding of this embodiment, a data transmission method disclosed in the embodiment of the present disclosure is first introduced in detail. The executor of the data transmission method provided in the embodiment of the present disclosure is generally a computer device with certain computing capabilities. In some possible implementation methods, the data transmission method can be implemented by a processor calling computer-readable instructions stored in a memory.
[0126] The data transmission method provided by the embodiment of the present disclosure is described below by taking the execution subject as the worker end Worker as an example.
[0127] like Figure 1 As shown, it is a flowchart of a data transmission method applied to a worker end Worker provided in an embodiment of the present disclosure, which may include the following steps:
[0128] S101: Obtain the tasks to be transmitted sent by the control end Master.
[0129] The task to be transmitted is obtained after the control end Master splits the acquired target task, and the task to be transmitted includes file attribute information of at least one file to be transmitted.
[0130] The data transmission method provided by the embodiment of the present disclosure can be implemented based on a pre-built Master-Worker framework, and can be applied to cross-cluster data transmission, wherein the Master is a control terminal, which is used to receive a target task from a client, and the target task is a task for file transfer initiated by a user on the client, and is used to transfer the file to be transferred to the target address. The file to be transferred may include various types of data, and different types of data have different functions. Specifically, the file to be transferred is a file that needs to be transferred and is stored in a certain file area according to a certain file storage method, for example, a file stored in area C of network disk A.
[0131] Worker is a worker end, which is used to execute the pending tasks sent by Master. The pending tasks can be obtained by Master splitting the target task using the to-be-transferred files corresponding to the target task after receiving the target task. A target task can be split into multiple pending tasks, and each pending task can be sent to different Workers for execution, that is, each to-be-transferred task is a subtask corresponding to the target task. A target task can correspond to multiple to-be-transferred files, and the to-be-transferred files corresponding to all to-be-transferred tasks are equivalent to the to-be-transferred files corresponding to the target task.
[0132] Specifically, a Master can correspond to multiple Workers, and multiple Workers can be set in at least one machine for file transfer. Here, although the transmission speed of small files is fast, the IO occupied by each file is fixed. Therefore, for small files, most of the time spent on IO is spent on transmission. The existing technology uses a single machine for file transmission, and the number of IO is limited. The transmission speed of multiple small files is also limited, which reduces the efficiency of file transmission. By setting up multiple machines, the number of available IO is increased, and the IO of multiple machines can be used for file transmission synchronously, avoiding the problem that the transmission of multiple small files is limited by the number of IO of a single machine, which affects the speed and efficiency of file transmission, thereby improving the speed and efficiency of file transmission.
[0133] The file attribute information is used to characterize the basic attribute information of the file to be transferred, such as the file location, file type, file source, file name, file size, etc. of the file to be transferred. A task to be transferred may include multiple files to be transferred, and the file attribute information of each file to be transferred may be the same or different, which may be determined by the file to be transferred corresponding to the actual target task.
[0134] In specific implementation, after the user initiates a target task for multiple files to be transferred on the client, the target task will be added to the pre-set task execution queue. Then, the Master can obtain the target task in the task execution queue and parse the target task, taking a single file to be transferred as the basic parsing unit, and using a concurrent method to quickly parse each file to be transferred corresponding to the target task, determine the file location, file source, file storage method, file size and other information of each file to be transferred, and use it as the file attribute information of each file to be transferred.
[0135] Afterwards, the multiple files to be transferred corresponding to the target task can be grouped according to the transmission capacity of each Worker and the file attribute information of each file to be transferred, and multiple groups of files to be transferred can be obtained. Among them, the transmission capacity of each Worker is used to reflect the file size of the files to be transferred that the Worker can transfer, the number of files to be transferred that can be transferred synchronously, and other information. The transmission capacity of different Workers can be different. For example, the number of files to be transferred that a Worker can transfer synchronously is 10, and the file size of each file to be transferred cannot exceed 20MB. Then the Master can filter out 10 files to be transferred with a file size not exceeding 20MB according to the file attribute information of each file to be transferred, and use them as a group of files to be transferred.
[0136] Alternatively, in the process of dividing the to-be-transmitted file groups, the to-be-transmitted files in the same position may be grouped as a to-be-transmitted file group based on the file position corresponding to each to-be-transmitted file; or, based on the file storage mode corresponding to each to-be-transmitted file, the to-be-transmitted files with the same file storage mode may be grouped as a to-be-transmitted file group. The disclosed embodiment does not limit the method of dividing the to-be-transmitted file groups, and may be set according to actual development needs.
[0137] Furthermore, the Master can package the files to be transferred in each file group to be transferred, and after completing the file packaging, it can generate a task to be transferred corresponding to each file group to be transferred based on the file attribute information of the files to be transferred in the file group to be transferred and the packaged files corresponding to each file group to be transferred. In addition, after generating the task to be transferred, the Master can also initialize each task to be transferred, specifically, it can initialize the task status and task execution progress of each task to be transferred, and determine the total file size and total number of files of the files to be transferred corresponding to each task to be transferred.
[0138] Afterwards, each generated task to be transmitted can be sent to different Workers respectively, and then the Worker can obtain the task to be transmitted sent by the Master.
[0139] S102: Based on the file attribute information, determine the file reading information of each file to be transmitted corresponding to the task to be transmitted.
[0140] Here, the file reading information is information used to read the file to be transmitted, wherein the file reading information may include information such as the file reading method and the file location.
[0141] In the specific implementation, for any Worker corresponding to the Master, after obtaining the task to be transferred, the coroutine set in the Worker for parsing the task to be transferred can be used to parse the task to be transferred, and each file to be transferred corresponding to the task to be transferred can be determined. Then, according to the file attribute information of each file to be transferred, the file reading method, file location and other information of the file to be transferred can be determined, and then it can be used as the file reading information of the file to be transferred.
[0142] S103: using a plurality of pre-set coroutines based on the file reading information, respectively obtaining the to-be-transferred files that match each file reading information.
[0143] Here, a coroutine pool may be set in a Worker, and the coroutine pool includes multiple coroutines. Different coroutines may have different functions, and specifically, may include the coroutine for parsing the task to be transmitted in the above S102, and the coroutine for obtaining the file to be transmitted and transmitting the file to be transmitted.
[0144] In specific implementation, after determining the file reading information of each file to be transferred, the file reading information can be sent to the data storage channel. The data storage channel is a channel for storing file reading information, and the coroutine in the coroutine pool for transferring the file to be transferred needs to obtain the file reading information in the data storage channel.
[0145] Then, for each coroutine in the coroutine pool for transmitting the file to be transferred, the coroutine can obtain file reading information corresponding to a file to be transferred in the data storage channel, and then, based on the file reading information, the file location and file reading method of the file to be transferred that match the file reading information can be determined, and then, the determined file reading method can be used to obtain the file to be transferred from the file location.
[0146] Furthermore, each coroutine can obtain the file to be transferred that matches the file reading information it obtains.
[0147] S104: Utilize multiple coroutines to transfer each file to be transferred in the task to be transferred to a corresponding target address.
[0148] Here, the target address is the location where the file to be transferred needs to be transferred. In addition, the target addresses corresponding to each file to be transferred can be the same or different, and are not limited here. For example, the target address corresponding to the file to be transferred A is the A1 area corresponding to the network disk B, the target address corresponding to the file to be transferred B is the A1 area corresponding to the network disk B, and the target address corresponding to the file to be transferred C is the C1 area corresponding to the network disk C.
[0149] In this step, for each of the multiple coroutines, the coroutine can respond to the acquired task to be transferred and determine the target address corresponding to the file to be transferred corresponding to the coroutine based on the task to be transferred. Then, the coroutine can transfer the acquired file to be transferred to the corresponding target address.
[0150] Based on the above steps, each of the multiple coroutines can determine the target address corresponding to the file to be transferred it obtains, and transfers the corresponding file to be transferred to the target address. Thus, each file to be transferred in the task to be transferred can be transferred to the corresponding target address.
[0151] In addition, after transferring the file to be transferred to the corresponding target address, the Worker can also download the file to be transferred to the database corresponding to the target address. Specifically, the database can be the database corresponding to the cloud server. In this way, the file to be transferred can be stored locally, and then, it is convenient for subsequent users to manage the files to be transferred stored locally.
[0152] In addition, when the number of files to be transferred is greater than the number of Workers corresponding to the Master, each Worker can return to execute S103 and S104 after transferring the acquired files to be transferred to the target address until each file to be transferred in the task to be transferred is transferred to the corresponding target address.
[0153] In this way, based on the file reading information corresponding to each determined file to be transferred, the file reading method that matches the storage method of each file to be transferred can be determined, so that the transmission of files to be transferred stored in different data storage methods can be realized, thereby improving the transmission efficiency of files or data, and is conducive to improving the efficiency of machine learning. In addition, a coroutine can transmit a file to be transferred, and each coroutine is configured with multiple file reading methods. Therefore, by setting multiple coroutines in the Worker, it is not only possible to use the file reading method that matches different data storage methods to read the file to be transferred, but also possible to realize the synchronous transmission of multiple files to be transferred after reading the file to be transferred, which improves the diversity and flexibility of file reading on the one hand, and improves the efficiency of data transmission on the other hand, thereby improving the execution speed and efficiency of the task to be transferred. In addition, since the task to be transferred is obtained by splitting the target task obtained by the Master, by splitting the target task, it is possible to send the split multiple tasks to be transferred to multiple Workers, and use multiple Workers to synchronously process multiple tasks to be transferred, further improving the speed and efficiency of file transmission.
[0154] In one embodiment, with respect to S102, the file reading information of the file to be transmitted may be determined according to the following steps:
[0155] Step 1: Based on the file attribute information, determine the file storage method and file location of the file to be transferred.
[0156] Here, the file storage mode is used to reflect how the file to be transferred is stored. Different file storage modes correspond to different file reading modes. Specifically, the file storage modes may include lustre, ceph, ftp, etc. The file location is the location of the file to be transferred.
[0157] It can be seen from the above embodiments that the file attribute information includes the file location and file storage method of the file to be transmitted. Therefore, for any file to be transmitted corresponding to the task to be transmitted, the file attribute information of the file can be directly parsed to determine the file storage method and file location of the file to be transmitted.
[0158] Step 2: Based on the file storage method of the file to be transmitted, determine the file reading method of the file to be transmitted.
[0159] Here, the file reading mode is used to represent the mode of reading the file to be transmitted from the file location, and one file storage mode corresponds to one file reading mode. For any file to be transmitted, only by reading the file to be transmitted using the file reading mode corresponding to the file storage mode of the file to be transmitted, can an accurate and complete file be obtained. Otherwise, the file to be transmitted cannot be obtained, or an inaccurate and incomplete file is obtained.
[0160] In specific implementation, after determining the file storage mode of the file to be transferred, the Worker can directly determine the file reading mode that matches the file storage mode.
[0161] In addition, for the above S103, each coroutine may be provided with a file reading method that matches a plurality of file storage methods. When the coroutine obtains the file reading method of the to-be-transferred file from the data storage channel, the above file reading method may be selected from the preset plurality of file reading methods as the method for reading the to-be-transferred file this time.
[0162] In this way, by pre-setting multiple file reading methods, it can be ensured that after each coroutine determines the file reading method of the file to be transferred, there must be a reading method that matches the file reading method, avoiding the problem that the coroutine can only use one file reading method, resulting in a single working mode and poor flexibility of the coroutine, improving the flexibility of the coroutine in reading files, and then improving the flexibility and applicability of the Worker.
[0163] Step 3: Use the file reading method and the file location as the file reading information of the file to be transferred.
[0164] In specific implementation, the file reading method and file location can be directly used as the file reading information of the file to be transferred, and then it can be stored in the data storage channel, so that after the coroutine obtains the file reading information, it can accurately obtain the file reading method and file location from it, and complete the reading and transmission of the file to be transferred.
[0165] In one embodiment, for S104, for each of the multiple coroutines, the coroutine can be used to obtain the file to be transferred and transfer it to the target address corresponding to the file to be transferred. When the coroutine transmission is completed, the file transfer result of the file to be transferred can be obtained. Specifically, the file transfer result can be two types: file transfer success and file transfer failure.
[0166] Furthermore, when it is determined that the file transfer result of the file to be transferred is a file transfer failure, the number of transmission failures of the coroutine to transfer the file to be transferred can be determined. Then, the size relationship between the number of transmission failures and the preset failure threshold can be determined. When it is determined that the number of transmission failures is less than the preset failure threshold, the coroutine can be used to retransmit the file to be transferred. Specifically, after storing the file read information of each file to be transferred in the data storage channel, the Worker can establish a transmission connection with each file to be transferred. When it is determined that the number of transmission failures of the file to be transferred is less than the preset failure threshold, it can retry to establish the transmission connection corresponding to the file to be transferred, and retransmit it using the coroutine corresponding to the file to be transferred.
[0167] In this way, by setting a preset failure threshold, active retransmission of files that have failed to be transmitted can be achieved, thereby reducing the probability of file transmission failure.
[0168] In addition, when it is determined that the number of transmission failures is greater than the preset failure threshold, it means that the method of actively retransmitting the file to be transmitted cannot complete its transmission. In this transmission, the file to be transmitted has failed to be transmitted. If the transmission of the file to be transmitted needs to be successfully completed, it is necessary to use manual methods to make certain adjustments to the task to be transmitted or the file to be transmitted. Therefore, based on the file to be transmitted, a transmission failure record corresponding to the file to be transmitted can be generated. Afterwards, the transmission failure record can be sent to the client corresponding to the Master. In this way, after receiving the transmission failure record, the client can display it to the user, so that the user can obtain the transmission failure record, and when it is determined that the file to be transmitted corresponding to the transmission failure record needs to be transmitted to its corresponding target address, the task to be transmitted or the file to be transmitted can be adjusted to a certain extent, and then the transmission task for the file to be transmitted can be resent to complete the transmission of the file to be transmitted.
[0169] Among them, the step of sending the generated transmission failure record to the client can be performed after the Worker completes the transmission of each file to be transmitted corresponding to the acquired task to be processed. Specifically, after determining that each file to be transmitted has been transmitted, the generated transmission failure records can be merged to generate a final failure record, and sent to the client so that the client can display it to the user. Alternatively, for any file to be transmitted, after generating its corresponding transmission failure record, it can be sent directly to the client, which is not specifically limited here.
[0170] In another implementation, after generating the final failure record, the Worker may send it to the Master, and then the Master sends the final failure record to the client.
[0171] In another embodiment, before sending the transmission failure record to the client corresponding to the Master, the file transmission information of each file to be transmitted may also be determined. Here, the file transmission information includes at least one of the transmission time information of the file to be transmitted, the file name, the transmission task creator information corresponding to the file to be transmitted, and the hash check code corresponding to the file to be transmitted.
[0172] Among them, the transmission time information may include the transmission start time, transmission end time and transmission duration information, etc.; the transmission task creator information may include the transmission task creator, transmission task modifier, etc.; the hash check code is used to check the correctness of the file to be transmitted.
[0173] In addition, the Worker can also determine the file transfer result of each coroutine transmitting the file to be transferred, that is, determine whether each file to be transferred is successfully transferred. Furthermore, the task execution result corresponding to the task to be transferred can be determined based on the file transfer result of each file to be transferred. Specifically, the task execution result can include three types: successful task execution, task execution failure, and partial task failure.
[0174] Among them, after the file transfer result of each file to be transferred, the files to be transferred that were successfully transferred and the files to be transferred that failed to be transferred can be determined. Furthermore, when it is determined that each file to be transferred is successfully transferred, the task execution result can be determined as successful task execution; when it is determined that each file to be transferred fails to be transferred, the task execution result can be determined as task execution failure; when it is determined that some files to be transferred fail to be transferred, the task execution result can be determined as partial task failure. And the transmission failure records of the files to be transferred that failed to be transferred can be merged and used together as the task execution result of the task to be transferred.
[0175] In addition, the task execution result corresponding to the target task can also be determined based on the task execution result corresponding to each Worker. And, for each Worker, the file transfer information of each file to be transferred corresponding to the Worker can be merged after it is determined, and after it is determined that the tasks to be transferred corresponding to each Worker have been completed, the merged file transfer information corresponding to each Worker can be combined to obtain the file transfer information corresponding to the target task.
[0176] Afterwards, the transmission failure record, the file transmission information of each file to be transmitted (and / or the file transmission information corresponding to the target task) and the task execution result (and / or the task execution result corresponding to the target task) can be sent to the client together, so that the client can display all the above information to the user. Furthermore, the user can manage the files to be transmitted based on the above information.
[0177] Exemplarily, after determining the file transfer information corresponding to the target task and the task execution result corresponding to the target task, a file directory corresponding to the file to be transferred can be generated based on the file transfer information, and an execution result directory corresponding to the file to be transferred can be generated based on the task execution result, and then sent to the client. The client can display it to the user, so that each file to be transferred can be managed through the file directory and / or the execution result directory, for example, viewing the file to be transferred, deleting the file to be transferred, etc.
[0178] In one embodiment, after determining the task execution result of a task to be transmitted corresponding to a certain Worker, the task to be transmitted can be regarded as an executed task, and then the task execution result of each Worker corresponding to the Master executing the acquired task to be transmitted can be obtained, that is, the task execution result of each task to be transmitted corresponding to the target task except the executed task is determined.
[0179] Then, the task completion order corresponding to the executed tasks can be determined based on the task execution results of each task to be transmitted. The task completion order is used to represent the task to be transmitted corresponding to the executed task, and is the first completed task among all tasks to be transmitted corresponding to the target task.
[0180] Specifically, based on the task execution result of each task to be transmitted, the number of successfully executed tasks to be transmitted can be determined, and then the number can be increased by 1 as the task completion order of the executed tasks.
[0181] Then, the task completion progress of the target task can also be determined based on the task completion order of the executed tasks and the number of tasks to be transmitted corresponding to the target task. For example, if the number of tasks to be transmitted corresponding to the target task is 20 and the task completion order of the executed tasks is 10, then the task completion progress of the target task can be determined to be 50%. Furthermore, the task completion progress can be sent to the client so that the user can obtain the task completion progress of the target task in a timely manner on the client.
[0182] In addition, when it is determined that the task completion order of the executed tasks matches the number of tasks to be transmitted corresponding to the target task, the task status of the target task is updated to the completed state. Specifically, when it is determined that the value corresponding to the task completion order is consistent with the number of tasks to be transmitted, the task status of the target task is updated to the completed state.
[0183] Alternatively, when it is determined that the two match, the information on the completion of the target task can be fed back to the Master, and then, after receiving the information, the Master updates the task status of the target task to the completed status.
[0184] In one embodiment, the Worker may also obtain a retransmission task instruction sent by the Master during the execution of the task to be transmitted or after the task to be transmitted is completed, wherein the retransmission task instruction is generated by the Master according to the file retransmission task sent by the client.
[0185] Specifically, the data transmission method provided in the embodiment of the present disclosure may also include a task listener, which is used to obtain the task status of the target task during the process of the Worker executing the target task. The task status may include an initialization state, an executing state, a completion state, and a failure state, etc. The specific task status setting can be determined according to development needs and is not limited here.
[0186] Furthermore, the task listener can send the acquired task status to the client, so that the client can display the task status of the target task to the user after acquiring the task status of the target task. Thus, the user can grasp the status change of the target task in a timely manner.
[0187] In addition, the task listener can also obtain the file retransmission request initiated by the client for the task to be transferred. The file retransmission request can be a retransmission request initiated by the user on the client for some files to be transferred that failed to be transferred after the task execution result corresponding to the task to be transferred is determined; it can also be a retransmission request initiated by the user on the client for all files to be transferred corresponding to the task to be transferred.
[0188] In addition, the file retransmission request may also be a retransmission request for part of the to-be-transmitted files or all the to-be-transmitted files corresponding to the target task.
[0189] Furthermore, the task listener determines the retransmission file specified by the user that needs to be retransmitted based on the obtained file retransmission request, that is, determines the retransmission file corresponding to the file retransmission request, and then generates a file retransmission task instructing to retransmit the retransmission file according to the determined retransmission file and the file retransmission request and sends it to the Master. For example, a file retransmission task can be generated according to the file position corresponding to each retransmission file.
[0190] The retransmission file may be a to-be-transmitted file corresponding to one to-be-transmitted task, or may be a to-be-transmitted file corresponding to multiple to-be-transmitted tasks.
[0191] Afterwards, the Master can receive the file retransmission task sent by the task listener, parse the file retransmission task, determine the file location, file storage method and other file attribute information corresponding to each retransmitted file, and determine the Worker corresponding to each retransmitted file. The retransmitted files corresponding to the same Worker are taken as a retransmission file group, and at least one retransmission task instruction is generated based on each retransmission file group. Then, each retransmission task instruction can be sent to the Worker that matches the retransmission file.
[0192] Alternatively, after determining the file location, file storage method and other file attribute information corresponding to each retransmitted file, the Master can generate a retransmission task instruction based on the file attribute information corresponding to the retransmitted file, and send it to any Worker, using the Worker to perform the retransmission operation on each retransmitted file.
[0193] Based on the above steps, the Worker can obtain the retransmission task instruction sent by the Master. Then, based on the retransmission task instruction, the Worker can determine the retransmission file from the files to be transmitted corresponding to the task to be transmitted, and retransmit the determined retransmission file to transmit the file to be transmitted to its corresponding target address.
[0194] Alternatively, after receiving the retransmission task instruction, the Worker may reacquire each retransmission file according to steps S101 to S104 and transmit it to the corresponding target address.
[0195] In this way, based on the received retransmission task instruction, the retransmission of the failed retransmission file can be completed, thereby ensuring that the retransmission file can be successfully transmitted to the target address in the end.
[0196] In another embodiment, the Worker may also obtain a transmission suspension instruction during the execution of the task to be transmitted, wherein the task suspension instruction may be directly sent to the Worker by the task listener.
[0197] In specific implementation, the task listener may obtain a transmission suspension request initiated by the client for the task to be transmitted; wherein the transmission suspension request may be a request initiated by the user at the client to suspend transmission of some or all files to be transmitted.
[0198] When the user needs to suspend the transmission of some or all files to be transmitted, a transmission suspension request can be initiated on the client for the task to be transmitted corresponding to the file to be transmitted. After that, the task listener can obtain the transmission suspension request initiated by the client, and based on the transmission suspension request, determine the transmission suspension file corresponding to the transmission suspension request, and then generate a transmission suspension instruction based on the transmission suspension file, and send the generated transmission suspension instruction to the Master and the Worker matching the transmission suspension file.
[0199] Specifically, after determining the suspended transmission file, the Worker corresponding to each suspended transmission file can be determined, and then a suspended transmission instruction matching each Worker and a suspended transmission instruction for instructing the Master to suspend sending new tasks to be transmitted to the Worker matching the suspended transmission file can be generated, and then the generated suspended transmission instruction can be sent to the Worker and the Master respectively.
[0200] Furthermore, after receiving the transmission suspension instruction, the Master may respond to the transmission suspension instruction, determine the Worker corresponding to the transmission suspension instruction, and suspend sending new tasks to be transmitted to the determined Worker.
[0201] Furthermore, for each Worker corresponding to the pause transmission instruction, after receiving the pause transmission instruction, the Worker can determine, based on the received pause transmission instruction, from the files to be transmitted corresponding to the task to be transmitted a file to be transmitted that matches the pause transmission file corresponding to the pause transmission instruction, and use it as the pause transmission file, and then, the Worker can suspend the transmission of the determined pause transmission file.
[0202] In one embodiment, the Worker may also obtain a cancel transmission instruction for the task to be transmitted during the execution of the task to be transmitted. The cancel transmission instruction may be sent directly to the Worker by the task listener, and is generated by the task listener based on the cancel transmission request initiated by the client for the task to be transmitted corresponding to the target task. The cancel transmission request may be for at least one task to be transmitted corresponding to the target task.
[0203] Specifically, after obtaining the cancel transmission request initiated by the client, the task listener can determine the cancel transmission task that matches the cancel transmission request from the multiple tasks to be transmitted corresponding to the target task. Furthermore, based on each cancel transmission task determined, a cancel transmission instruction that matches the Worker corresponding to each cancel transmission task can be generated, and a cancel transmission instruction that is used to instruct the Master to suspend sending new tasks to be transmitted to the Worker that matches the cancel transmission task can be generated. Then, the generated cancel transmission instruction can be sent to the Worker and the Master corresponding to the cancel transmission task.
[0204] Furthermore, after receiving the cancel transmission instruction, the Master may determine each Worker that matches the cancel transmission task in response to the cancel transmission instruction, and suspend sending new tasks to be transmitted to each of the above Workers.
[0205] At the same time, for each Worker corresponding to the cancel transmission instruction, after receiving the cancel transmission instruction, the Worker can directly respond to the cancel transmission instruction and stop the transmission operation of each to-be-transmitted file corresponding to the to-be-transmitted task.
[0206] like Figure 2 As shown, a specific implementation flow chart of each file to be transmitted corresponding to a transmission target task provided by an embodiment of the present disclosure may include the following steps:
[0207] S201: Using the task listener to obtain the task status of the target task.
[0208] Here, in the process of obtaining the task status of the target task, the task listener can also obtain the file retransmission request, transmission suspension request and transmission cancellation request initiated by the user on the client.
[0209] During specific implementation, this step may be carried out throughout the entire implementation process of S201 to S205.
[0210] S202: The control end Master obtains the target task, splits the target task into multiple tasks to be transmitted and sends them to the worker end Worker.
[0211] S203: The Worker obtains the task to be transferred sent by the Master, and determines the file reading information of each file to be transferred corresponding to the task to be transferred.
[0212] S204: The coroutine obtains the to-be-transferred file that matches the file reading information based on the obtained file reading information.
[0213] In this step, the Worker includes using a plurality of pre-set coroutines. For each coroutine, the coroutine can obtain file reading information in the data storage channel, and based on the file reading information, obtain a file to be transferred that matches the file reading information.
[0214] S205: Using the coroutine, the acquired file to be transferred is transferred to the corresponding target address.
[0215] In this step, for each coroutine, after the coroutine obtains the file to be transferred, the file to be transferred can be transferred to its corresponding target address, thereby achieving the transmission of each file to be transferred in the task to be transferred to the corresponding target address.
[0216] Regarding the specific implementation steps of S201 to S205, reference may be made to the above embodiments, which will not be described in detail here.
[0217] In addition, the present disclosure also provides a data transmission system corresponding to the data transmission method, including the Master mentioned in the above embodiments, at least one Worker mentioned in the above embodiments; Figure 3 FIG. 1 is a schematic diagram of a data transmission system provided by an embodiment of the present disclosure.
[0218] The control end Master is used to obtain a target task; split the target task into multiple tasks to be transmitted; send each task to be transmitted to each of the worker ends Worker in at least one worker end Worker; the task to be transmitted includes file attribute information of at least one file to be transmitted;
[0219] The worker end Worker is used to obtain the task to be transferred sent by the control end Master; and based on the file attribute information, determine the file reading information of each file to be transferred corresponding to the task to be transferred; and based on the file reading information, respectively obtain the files to be transferred that match each file reading information; and transfer each file to be transferred in the task to be transferred to the corresponding target address.
[0220] In a possible implementation, the data transmission system further includes a task listener, wherein:
[0221] The task listener is used to obtain the task status of the target task during the process of the worker-side Worker executing the target task; and send the task status to the client corresponding to the control-side Master so that the client displays the task status.
[0222] In a possible implementation manner, the task listener is further used to obtain a file retransmission request initiated by the client for the task to be transmitted; based on the file retransmission request, determine a retransmission file corresponding to the file retransmission request; based on the retransmission file, generate a file retransmission task and send it to the control end Master;
[0223] The control end Master is used to receive the file retransmission task; generate a retransmission task instruction based on the file retransmission task; and send the retransmission task instruction to the worker end Worker matching the retransmission file;
[0224] The worker end Worker is used to receive the retransmission task instruction; based on the retransmission task instruction, determine the retransmission file from the to-be-transmitted files corresponding to the to-be-transmitted task; and transmit the retransmission file.
[0225] In a possible implementation manner, the task listener is further used to obtain a transmission pause request initiated by the client for the task to be transmitted; based on the transmission pause request, determine a transmission pause file corresponding to the transmission pause request; based on the transmission pause file, generate a transmission pause instruction; and send the transmission pause instruction to the control end Master and the worker end Worker matching the transmission pause file;
[0226] The control end Master is used to receive the transmission suspension instruction; in response to the transmission suspension instruction, suspend sending new tasks to be transmitted to the worker end Worker matching the transmission suspension file;
[0227] The worker end Worker is used to receive the transmission suspension instruction; based on the transmission suspension instruction, determine the transmission suspension file from the to-be-transmitted files corresponding to the to-be-transmitted tasks; and suspend the transmission of the transmission suspension file.
[0228] In a possible implementation manner, the task listener is further used to obtain a transmission cancellation request initiated by the client for the task to be transmitted; determine a transmission cancellation task that matches the transmission cancellation request from multiple tasks to be transmitted; generate a transmission cancellation instruction based on the transmission cancellation task; and send the transmission cancellation instruction to the control end Master and the worker end Worker that matches the transmission cancellation task;
[0229] The control end Master is used to receive the cancel transmission instruction; in response to the cancel transmission instruction, suspend sending new tasks to be transmitted to the worker end Worker matching the cancel transmission task;
[0230] The worker end Worker is used to receive the cancel transmission instruction; in response to the cancel transmission instruction, stop the transmission operation of the to-be-transmitted file corresponding to the to-be-transmitted task.
[0231] In a possible implementation manner, the control end Master is used to parse the target task after acquiring the target task, and determine the file attribute information of each to-be-transferred file corresponding to the target task;
[0232] Based on the file attribute information of each file to be transmitted, the files to be transmitted are divided into a plurality of file groups to be transmitted;
[0233] Based on each of the to-be-transmitted file groups, a to-be-transmitted task corresponding to each of the to-be-transmitted file groups is generated, and the to-be-transmitted task is sent to the worker end Worker.
[0234] Regarding the specific operations performed by the Master, Worker and task listener in the above data transmission system, as well as the specific implementation process of various possible implementation methods, please refer to the introduction of each embodiment in the data transmission method, which will not be repeated here.
[0235] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.
[0236] Based on the same inventive concept, a data transmission device corresponding to the data transmission method is also provided in the embodiment of the present disclosure. Since the principle of solving the problem by the device in the embodiment of the present disclosure is similar to the above-mentioned data transmission method in the embodiment of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.
[0237] like Figure 4 FIG. 1 is a schematic diagram of a data transmission device provided by an embodiment of the present disclosure, including:
[0238] The first acquisition module 401 is used to acquire a task to be transmitted sent by the control end Master; the task to be transmitted is obtained by the control end Master splitting the acquired target task, and the task to be transmitted includes file attribute information of at least one file to be transmitted;
[0239] A first determining module 402, configured to determine file reading information of each file to be transferred corresponding to the task to be transferred based on the file attribute information;
[0240] The second acquisition module 403 is used to respectively acquire the to-be-transferred files matching each of the file reading information based on the file reading information by using a plurality of pre-set coroutines;
[0241] The transmission module 404 is used to use the multiple coroutines to transmit each of the files to be transmitted in the tasks to be transmitted to the corresponding target address.
[0242] In a possible implementation manner, the first determining module 402 is configured to determine a file storage mode and a file location of the file to be transferred based on the file attribute information;
[0243] Determining a file reading method for the file to be transmitted based on a file storage method for the file to be transmitted;
[0244] The file reading mode and the file position are used as the file reading information of the file to be transmitted.
[0245] In a possible implementation, the transmission module 404 is configured to transmit, for each coroutine in the plurality of coroutines, the corresponding file to be transmitted to the target address by using the coroutine;
[0246] In the case where the transmission of the file to be transmitted fails, determining the number of transmission failures of the file to be transmitted;
[0247] When the number of transmission failures is less than a preset failure threshold, the file to be transmitted is transmitted again.
[0248] In a possible implementation, the transmission module 404 is further configured to, after determining the number of transmission failures of the file to be transmitted, generate a transmission failure record of the file to be transmitted if the number of transmission failures is greater than the preset failure threshold;
[0249] The transmission failure record is sent to the client corresponding to the control end Master, so that the client displays the transmission failure record.
[0250] In a possible implementation manner, the device further includes a second determining module 405:
[0251] The second determination module 405 is used to determine the file transmission information of each file to be transmitted before sending the transmission failure record to the client corresponding to the control end Master; the file transmission information includes at least one of the transmission time information, file name, and the transmission task creator information corresponding to the file to be transmitted;
[0252] Determine a file transfer result for each of the files to be transferred;
[0253] Determining a task execution result of the task to be transferred based on a file transfer result of each of the files to be transferred;
[0254] The transmission module 404 is further configured to send the transmission failure record, the file transmission information of each file to be transmitted, and the task execution result to the client corresponding to the control end Master.
[0255] In a possible implementation, the device further includes:
[0256] An updating module 406, configured to, after determining the task execution result of the task to be transmitted, treat the task to be transmitted as an executed task;
[0257] Determine a task execution result of each task to be transmitted corresponding to the target task except the executed task;
[0258] Based on the task execution results, determining the task completion order of the executed tasks;
[0259] In a case where the task completion order matches the number of tasks to be transmitted corresponding to the target task, the task status of the target task is updated to a completion status.
[0260] In a possible implementation manner, the first acquisition module 401 is further configured to acquire a retransmission task instruction sent by the control end Master, and determine a retransmission file from the to-be-transmitted file corresponding to the to-be-transmitted task based on the retransmission task instruction;
[0261] The transmission module is also used to transmit the retransmission file.
[0262] In a possible implementation manner, the first acquisition module 401 is further configured to acquire a transmission suspension instruction for the file to be transmitted;
[0263] Based on the transmission suspension instruction, determining a transmission suspension file from the to-be-transmitted files corresponding to the to-be-transmitted tasks;
[0264] The transmission module is further used to suspend the transmission of the suspended transmission file.
[0265] In a possible implementation manner, the first acquisition module 401 is further configured to acquire a transmission cancellation instruction for the task to be transmitted;
[0266] The transmission module is further configured to stop the transmission operation of the to-be-transmitted file corresponding to the to-be-transmitted task in response to the cancel transmission instruction.
[0267] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference may be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.
[0268] The present disclosure also provides a computer device, such as Figure 5 FIG. 1 is a schematic diagram of a computer device structure provided by an embodiment of the present disclosure, including:
[0269] A processor 51 and a memory 52; the memory 52 stores machine-readable instructions executable by the processor 51, and the processor 51 is used to execute the machine-readable instructions stored in the memory 52. When the machine-readable instructions are executed by the processor 51, the processor 51 performs the following steps: S101: obtaining the task to be transferred sent by the control end Master; S102: based on the file attribute information, determining the file reading information of each file to be transferred corresponding to the task to be transferred; S103: using a plurality of pre-set coroutines based on the file reading information, respectively obtaining the files to be transferred that match each file reading information; and S104: using a plurality of coroutines, transferring each file to be transferred in the task to be transferred to the corresponding target address.
[0270] The above-mentioned memory 52 includes an internal memory 521 and an external memory 522; the internal memory 521 here is also called an internal memory, which is used to temporarily store the calculation data in the processor 51, as well as the data exchanged with the external memory 522 such as a hard disk. The processor 51 exchanges data with the external memory 522 through the internal memory 521.
[0271] The specific execution process of the above instructions can refer to the steps of the data transmission method described in the embodiment of the present disclosure, which will not be repeated here.
[0272] The present disclosure also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the data transmission method described in the above method embodiment are executed. The storage medium may be a volatile or non-volatile computer-readable storage medium.
[0273] The computer program product of the data transmission method provided in the embodiments of the present disclosure includes a computer-readable storage medium storing program code, and the instructions included in the program code can be used to execute the steps of the data transmission method described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.
[0274] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK) and the like.
[0275] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.
[0276] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0277] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0278] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0279] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present disclosure, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure shall be subject to the protection scope of the claims.
Claims
1. A data transmission method, characterized in that: Applied to the Worker side, including: Obtain the task to be transmitted sent by the control end Master; the task to be transmitted is obtained by the control end Master splitting the acquired target task according to the transmission capacity of each of the worker ends Worker and the file attribute information of each file to be transmitted, and the task to be transmitted includes the file attribute information of at least one file to be transmitted; the transmission capacity is at least used to characterize the file size of the file that can be transmitted by the worker end Worker and the number of files that can be transmitted simultaneously; Determine the file storage mode and file location of the file to be transmitted based on the file attribute information; determine the file reading mode of the file to be transmitted based on the file storage mode of the file to be transmitted; and use the file reading mode and the file location as the file reading information of the file to be transmitted; Using a plurality of pre-set coroutines based on the file reading information, respectively obtaining a file to be transferred that matches each of the file reading information; the coroutine is configured with a plurality of file reading modes; Each of the files to be transferred in the tasks to be transferred is transferred to a corresponding target address by using the multiple coroutines.
2. The method according to claim 1, characterized in that The using the multiple coroutines to transfer each of the to-be-transferred files in the to-be-transferred tasks to a corresponding target address includes: For each coroutine in the multiple coroutines, using the coroutine to transmit the corresponding file to be transmitted to the target address; In the case where the transmission of the file to be transmitted fails, determining the number of transmission failures of the file to be transmitted; When the number of transmission failures is less than a preset failure threshold, the file to be transmitted is transmitted again.
3. The method according to claim 2, characterized in that After determining the number of transmission failures of the to-be-transmitted file, the method further includes: When the number of transmission failures is greater than the preset failure threshold, generating a transmission failure record of the file to be transmitted; The transmission failure record is sent to the client corresponding to the control end Master, so that the client displays the transmission failure record.
4. The method according to claim 3, characterized in that Before sending the transmission failure record to the client corresponding to the control end Master, the method further includes: Determine the file transmission information of each file to be transmitted; the file transmission information includes at least one of the transmission time information, the file name, and the transmission task creator information corresponding to the file to be transmitted; Determine a file transfer result for each of the files to be transferred; Determining a task execution result of the task to be transferred based on a file transfer result of each of the files to be transferred; The step of sending the transmission failure record to the client corresponding to the control end Master comprises: The transmission failure record, the file transmission information of each file to be transmitted and the task execution result are sent to the client corresponding to the control end Master.
5. The method according to claim 4, characterized in that After determining the task execution result of the task to be transmitted, the method further includes: Treat the task to be transmitted as an executed task; Determine a task execution result of each task to be transmitted corresponding to the target task except the executed task; Based on the task execution results, determining the task completion order of the executed tasks; In a case where the task completion order matches the number of tasks to be transmitted corresponding to the target task, the task status of the target task is updated to a completion status.
6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: Obtaining a retransmission task instruction sent by the control end Master, and determining a retransmission file from the to-be-transmitted files corresponding to the to-be-transmitted task based on the retransmission task instruction; The retransmission file is transmitted.
7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Obtaining a transmission suspension instruction for the file to be transmitted; Based on the transmission suspension instruction, determining a transmission suspension file from the to-be-transmitted files corresponding to the to-be-transmitted tasks; The transmission of the transmission-pause file is suspended.
8. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: Obtaining a transmission cancellation instruction for the task to be transmitted; In response to the cancel transmission instruction, the transmission operation of the to-be-transmitted file corresponding to the to-be-transmitted task is stopped.
9. A data transmission system, characterized in that: It comprises the control end Master according to any one of claims 1 to 8 and at least one worker end Worker according to any one of claims 1 to 8; The control end Master is used to obtain the target task; split the target task into multiple tasks to be transmitted according to the transmission capability of each of the worker ends Worker and the file attribute information of each file to be transmitted; Sending each of the tasks to be transmitted to each of the worker-end Workers in at least one of the worker-end Workers respectively; the tasks to be transmitted include file attribute information of at least one file to be transmitted; The transmission capability is used to represent at least the file size of the files that can be transmitted by the worker and the number of files that can be transmitted simultaneously; The worker end Worker is used to obtain the task to be transmitted sent by the control end Master; and determine the file storage mode and file location of the file to be transmitted based on the file attribute information; Determining a file reading method for the file to be transmitted based on a file storage method for the file to be transmitted; Using the file reading mode and the file location as the file reading information of the file to be transmitted; And using a plurality of pre-set coroutines based on the file reading information, respectively obtaining the to-be-transferred files matching each of the file reading information; the coroutines are configured with a plurality of file reading modes; and each of the to-be-transferred files in the to-be-transferred tasks is transmitted to the corresponding target address.
10. The system according to claim 9, characterized in that The data transmission system further includes a task listener, wherein: The task listener is used to obtain the task status of the target task during the process of the worker-side Worker executing the target task; and send the task status to the client corresponding to the control-side Master so that the client displays the task status.
11. The system according to claim 10, characterized in that The task listener is further used to obtain a file retransmission request initiated by the client for the task to be transmitted; based on the file retransmission request, determine a retransmission file corresponding to the file retransmission request; Based on the retransmission file, generate a file retransmission task and send it to the control end Master; The control end Master is used to receive the file retransmission task; generate a retransmission task instruction based on the file retransmission task; and send the retransmission task instruction to the worker end Worker that matches the retransmission file; The worker end Worker is used to receive the retransmission task instruction; Based on the retransmission task instruction, determining the retransmission file from the to-be-transmitted files corresponding to the to-be-transmitted task; The retransmission file is transmitted.
12. The system according to claim 10 or 11, characterized in that The task listener is further used to obtain a transmission suspension request initiated by the client for the task to be transmitted; based on the transmission suspension request, determine a transmission suspension file corresponding to the transmission suspension request; Based on the transmission suspension file, a transmission suspension instruction is generated; and the transmission suspension instruction is sent to the control end Master and the worker end Worker matching the transmission suspension file; The control end Master is used to receive the transmission suspension instruction; in response to the transmission suspension instruction, suspend sending new tasks to be transmitted to the worker end Worker matching the transmission suspension file; The worker end Worker is used to receive the transmission suspension instruction; Based on the transmission suspension instruction, a transmission suspension file is determined from the to-be-transmitted files corresponding to the to-be-transmitted tasks; and the transmission of the transmission suspension file is suspended.
13. The system according to any one of claims 10 to 12, characterized in that: The task listener is also used to obtain a transmission cancellation request initiated by the client for the task to be transmitted; determine a transmission cancellation task that matches the transmission cancellation request from multiple tasks to be transmitted; generate a transmission cancellation instruction based on the transmission cancellation task; and send the transmission cancellation instruction to the control end Master and the worker end Worker that matches the transmission cancellation task; The control end Master is used to receive the cancel transmission instruction; in response to the cancel transmission instruction, suspend sending new tasks to be transmitted to the worker end Worker matching the cancel transmission task; The worker end Worker is used to receive the cancel transmission instruction; in response to the cancel transmission instruction, stop the transmission operation of the to-be-transmitted file corresponding to the to-be-transmitted task.
14. The system according to any one of claims 9 to 13, characterized in that The control terminal Master is used to parse the target task after acquiring the target task, and determine the file attribute information of each file to be transferred corresponding to the target task; Based on the file attribute information of each file to be transmitted, the files to be transmitted are divided into a plurality of file groups to be transmitted; Based on each of the to-be-transmitted file groups, a to-be-transmitted task corresponding to each of the to-be-transmitted file groups is generated, and the to-be-transmitted task is sent to the worker end Worker.
15. A data transmission device, characterized in that: Applied to the Worker side, including: The first acquisition module is used to acquire the task to be transmitted sent by the control end Master; the task to be transmitted is obtained by the control end Master after splitting the acquired target task according to the transmission capacity of each of the worker ends Worker and the file attribute information of each file to be transmitted, and the task to be transmitted includes the file attribute information of at least one file to be transmitted; the transmission capacity is used to at least characterize the file size of the file that can be transmitted by the worker end Worker and the number of files that can be transmitted simultaneously; A first determination module is used to determine a file storage mode and a file location of the file to be transmitted based on the file attribute information; determine a file reading mode of the file to be transmitted based on the file storage mode of the file to be transmitted; and use the file reading mode and the file location as file reading information of the file to be transmitted; A second acquisition module is used to respectively acquire the to-be-transferred files matching each of the file reading information by using a plurality of pre-set coroutines based on the file reading information; the coroutines are configured with a plurality of file reading modes; The transmission module is used to utilize the multiple coroutines to transmit each of the files to be transmitted in the tasks to be transmitted to the corresponding target address.
16. A computer device, characterized in that: include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the processor is used to execute the machine-readable instructions stored in the memory. When the machine-readable instructions are executed by the processor, the processor executes the steps of the data transmission method according to any one of claims 1 to 8.
17. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program. When the computer program is executed by a computer device, the computer device executes the steps of the data transmission method according to any one of claims 1 to 8.
