Large file transmission method, device and equipment and computer readable storage medium

Abstract

The invention discloses a large file transmission method, device and equipment and a computer readable storage medium, and the method comprises the steps: enabling an execution module to collect a file from a target host according to a file collection task, packaging and compressing the collected file, and cutting the packaged and compressed file into a plurality of fragment files, enabling the service module to recombine the plurality of fragmented files and store the plurality of fragmented files to a first preset position; the file distribution task is stored in a third task pool of the service module, so that the control module pulls the file distribution task from the third task pool and pulls the to-be-distributed file from the service module, and the service module transmits the to-be-distributed file to the control module in a fragmented mode; and enabling the execution module to pull the file distribution task from the task pool of the control module and pull the file to be distributed after fragmentation from the control module, performing merging and integrity inspection after the execution module receives the file to be distributed after fragmentation, and uploading the file to a second preset position of the target host. By utilizing the method disclosed by the invention, the efficiency and the stability of large file transmission can be improved.

Description

Technical Field

[0001] The present invention relates to the field of file transmission technology, and in particular to a large file transmission method, device, equipment and computer-readable storage medium. Background Art

[0002] With the continuous improvement of the degree of enterprise informatization, the automated operation and maintenance platform plays an increasingly important role in the daily operation of enterprises. In large-scale distributed systems, file transfer is a basic requirement in operation and maintenance work. However, the traditional file transfer method has the following problems when transferring large files: when transferring large files at one time, the system cannot process the file because the file size exceeds the memory capacity, which causes the program to crash; in the process of multi-layer file transfer, the transmission delay is long because it is necessary to wait for the file to be completely received before proceeding to the next step, which affects the overall transmission efficiency; these problems seriously affect the efficiency and stability of large file transmission. Summary of the invention

[0003] The embodiments of the present invention provide a large file transmission method, device, equipment and computer-readable storage medium to solve the technical problem of low efficiency and stability of large file transmission in the prior art.

[0004] The technical solution of the present invention is as follows, which provides a large file transmission method, comprising:

[0005] When the service module receives a file collection task, the file collection task is stored in the first task pool of the service module, the control module is caused to pull the file collection task in the first task pool, the file collection task is stored in the second task pool of the control module, and the execution module is caused to pull the file collection task in the second task pool;

[0006] The execution module collects files from the target host according to the file collection task, packages and compresses the collected files, cuts them into multiple fragment files, and sends them to the control module, so that the control module forwards the multiple fragment files to the service module, so that the service module reassembles the multiple fragment files and stores them in the first preset location;

[0007] When the service module receives a file distribution task, the file distribution task is stored in a third task pool of the service module, so that the control module pulls the file distribution task from the third task pool and pulls the file to be distributed from the service module, and the service module transmits the file fragments to be distributed to the control module;

[0008] The execution module is made to pull the file distribution task from the task pool of the control module and pull the files to be distributed after being segmented from the control module, and after receiving the files to be distributed after being segmented, the execution module is made to merge and check the integrity and upload them to the second preset location of the target host.

[0009] Furthermore, before storing the file collection task in the first task pool of the service module, it also includes determining whether there is a file in the first task pool of the service module, and if there is and the latest file needs to be pulled, deleting the file in the first task pool;

[0010] Before storing the file collection task in the second task pool of the control module, it also includes judging whether there is a file in the temporary space of the control module, and if there is and the latest file needs to be pulled, deleting the file in the temporary space.

[0011] Further, storing the file distribution task into the third task pool of the service module includes:

[0012] Check whether the target file corresponding to the file distribution task exists in the third preset position of the service module, and if so, store the file distribution task in the task pool of the service module.

[0013] Further, enabling the execution module to collect files from the target host according to the file collection task includes:

[0014] The execution module is enabled to collect files from the target host through the file transfer protocol according to the file collection task. If the operation is successful, a success result of the first collection task is reported; otherwise, a failure result of the first collection task is reported.

[0015] Furthermore, after the collected files are packaged and compressed, they are cut into a plurality of fragment files and sent to the control module, the method further includes:

[0016] The control module is made to poll and check whether the plurality of segment files are received within a first preset time, and if so, report a success result of the second collection task, otherwise, report a failure result of the second collection task.

[0017] Further, the control module is made to forward the plurality of segment files to the service module, so that the service module reorganizes the plurality of segment files and stores them in a first preset location, including:

[0018] The control module forwards the multiple fragment files to the service module, and the service module polls to check whether the multiple fragment files are received within a second preset time;

[0019] If so, verify the order and integrity of the slices of the multiple slice files to reorganize the files, store the reorganized files into the first preset location, and report the success result of the third collection task;

[0020] Otherwise, report the failure result of the third collection task.

[0021] Furthermore, after pulling the file to be distributed from the service module, it also includes: judging whether the control module pulls the file to be distributed successfully, and if so, reporting the success result of the first distribution task, otherwise, reporting the failure result of the first distribution task;

[0022] After pulling the to-be-distributed file after being segmented from the control module, it also includes judging whether the execution module pulls the to-be-distributed file successfully, and if so, reporting the success result of the second distribution task, otherwise, reporting the failure result of the second distribution task.

[0023] Another technical solution of the present invention is as follows: a large file transmission device is provided, including a collection task pulling unit, a file collection unit, a first distribution task unit and a second distribution task unit;

[0024] The acquisition task pulling unit is used for, when the service module receives a file acquisition task, storing the file acquisition task in the first task pool of the service module, causing the control module to pull the file acquisition task in the first task pool, storing the file acquisition task in the second task pool of the control module, causing the execution module to pull the file acquisition task in the second task pool;

[0025] The file collection unit is used to enable the execution module to collect files from the target host according to the file collection task, package and compress the collected files, cut them into multiple fragment files, and send them to the control module, so that the control module forwards the multiple fragment files to the service module, so that the service module reorganizes the multiple fragment files and stores them in the first preset location;

[0026] The first task distribution unit is used for storing the file distribution task into a third task pool of the service module when the service module receives the file distribution task, so that the control module pulls the file distribution task from the third task pool and pulls the to-be-distributed file from the service module, and the service module transmits the to-be-distributed file segments to the control module;

[0027] The second distribution task unit is used to enable the execution module to pull the file distribution tasks from the task pool of the control module and pull the files to be distributed after being segmented from the control module, and enable the execution module to merge and check the integrity of the files to be distributed after receiving the segmented files, and upload them to the second preset location of the target host.

[0028] Another technical solution of the present invention is as follows: a computer device is provided, the device comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0029] Memory, used to store computer programs;

[0030] The processor is used to implement the steps of the large file transmission method described in any of the above technical solutions when executing the program stored in the memory.

[0031] Another technical solution of the present invention is as follows: a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the large file transmission method described in any of the above technical solutions are implemented.

[0032] The beneficial effects of the present invention are as follows: when the service module receives a file acquisition task, the file acquisition task is stored in the first task pool of the service module, so that the control module pulls the file acquisition task in the first task pool, and stores the file acquisition task in the second task pool of the control module, so that the execution module pulls the file acquisition task in the second task pool; the execution module collects files from the target host according to the file acquisition task, packages and compresses the collected files, cuts them into multiple fragment files, and sends them to the control module, so that the control module forwards the multiple fragment files to the service module, so that the service module reorganizes the multiple fragment files and stores them in a first preset position; when the service module receives a file distribution task, the file distribution task is stored in the third task pool of the service module, so that the control module pulls the file distribution task from the third task pool and pulls the files to be distributed from the service module and the service module transmits the file segments to be distributed to the control module; the execution module pulls the file distribution task from the task pool of the control module and pulls the files to be distributed after being segmented from the control module, and after receiving the files to be distributed after being segmented, the execution module performs merging and integrity check and uploads them to the second preset position of the target host; through the above-mentioned method, by cutting the large file into segments for transmission, the problem of insufficient memory when transmitting a large file at a single time is avoided, the size of each segment can be flexibly adjusted, and in the multi-layer transmission process, a streaming transmission method is adopted, that is, each segment is forwarded immediately after being received, avoiding a large amount of storage space of the transit node being occupied, reducing the transmission delay, and improving the efficiency of large file transmission; through segmented transmission and parallel transmission, the network bandwidth is fully utilized, the possibility of network congestion is reduced, it can run stably in different network environments, adapt to various network conditions, and improve the stability of large file transmission. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0034] Figure 1 A schematic diagram of a large file transmission method provided by an embodiment of the present invention;

[0035] Figure 2 A system architecture diagram corresponding to the large file transmission method provided by an embodiment of the present invention;

[0036] Figure 3 A schematic diagram of a process for collecting files provided by an embodiment of the present invention;

[0037] Figure 4 A schematic diagram of a process for distributing files provided by an embodiment of the present invention;

[0038] Figure 5 A schematic diagram of the structure of a large file transmission device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] In the description of this application, the words "first", "second", etc. are only used for the purpose of distinguishing descriptions, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying an order. In this specification, the terms "include", "comprise", "have" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0041] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0042] Figure 11 is a flow chart of a large file transmission method according to an embodiment of the present invention. It should be noted that if there is substantially the same result, the large file transmission method of the present invention is not limited to the above two methods. Figure 1 The process sequence shown is limited. Figure 1 As shown, the large file transmission method mainly includes the following steps:

[0043] S101, when a service module receives a file collection task, the service module stores the file collection task in a first task pool of the service module, causes the control module to pull the file collection task in the first task pool, stores the file collection task in a second task pool of the control module, causes the execution module to pull the file collection task in the second task pool;

[0044] In an optional implementation, before storing the file collection task in the first task pool of the service module, it also includes determining whether there is a file in the first task pool of the service module, and if there is and the latest file needs to be pulled, deleting the file in the first task pool;

[0045] Before storing the file collection task in the second task pool of the control module, it also includes judging whether there is a file in the temporary space of the control module, and if there is and the latest file needs to be pulled, deleting the file in the temporary space.

[0046] In a specific embodiment, the service module can be a service layer GoServer, which provides an interface to receive tasks sent by the front end and stores them in the task pool (first task pool) for the control module service to pull. It can also collect the heartbeat data of the control module service, report it to the front end service after aggregation, and collect the task result data reported by the control module service and forward it to the front end service. The service layer GoServer can be developed using the Go language and provide services through a RESTful API. The task pool uses a distributed cache Redis to store task information to ensure high concurrent access performance.

[0047] In a specific embodiment, the control module can be the control layer GoGateway. Each client system can deploy a set of GoGateway, which can be deployed in cluster mode. It can pull the tasks of the client system from GoServer and store them in the GoGateway task pool (second task pool) for the execution module service to pull. The control module can pull files from the service module for transfer and provide them for the execution layer service to pull. It can also summarize the heartbeat data of all execution module services of the client system and report the heartbeat of GoGateway itself to the service module. It can summarize the task execution results of all services of the execution module and finally report them to the service module. The control layer GoGateway can also be developed using the Go language, supporting high availability and load balancing. The second task pool uses distributed cache (Redis) to store task information to ensure high concurrent access performance.

[0048] In a specific embodiment, the execution module can be the execution layer GoProxy. Each client system can be divided into networks and deploy a GoProxy cluster (multiple GoProxy can be deployed in each network environment for load balancing). The execution layer GoProxy can pull the tasks belonging to the client network from the control layer GoGateway of the corresponding client system, and execute remote scripts, file collection, and file distribution tasks on the target host through protocols such as SSH / SFTP / XfreeRDP, and send the task results to the corresponding control layer GoGateway, and report the heartbeat to the corresponding control layer GoGateway regularly. The execution module can be developed using the Go language and support multiple network protocols to connect to the server and client to ensure stable operation in complex network environments.

[0049] In a specific embodiment, the system architecture diagram corresponding to the large file transmission method is as follows: Figure 2 As shown, Figure 2 The local computer room includes an execution layer, a control layer, and a service layer. The service layer and the control layer include the first task pool and the second task pool respectively. The execution layer includes two modes: one is the normal mode, namely the Agentless mode, and the other is the Agent mode. In the Agentless mode, the AIOps Go Proxy is connected to the host to perform normal tasks. In the Agent mode, the host AlOps Agent is directly deployed on the host to perform Agent tasks. The cloud management platform, traditional resource pool or other computer rooms also include an execution layer and a control layer.

[0050] In a specific embodiment, the flowchart of file collection is as follows: Figure 3 As shown, Figure 3 In the example, when a task is issued, the service module (Server) can be split into two tasks and three callbacks can be performed.

[0051] S102, causing the execution module to collect files from the target host according to the file collection task, pack and compress the collected files, cut them into multiple fragment files, and send them to the control module, causing the control module to forward the multiple fragment files to the service module, causing the service module to reorganize the multiple fragment files and store them in a first preset location;

[0052] In an optional implementation, enabling the execution module to collect files from the target host according to the file collection task includes:

[0053] The execution module is enabled to collect files from the target host through the file transfer protocol according to the file collection task. If the operation is successful, a success result of the first collection task is reported; otherwise, a failure result of the first collection task is reported.

[0054] In a specific embodiment, the execution module (GoProxy) collects files from the target host through the SFTP protocol, packages and compresses the files, and then cuts them into multiple fragments. For example, each fragment is 20M, which can be adjusted as needed, and multiple fragment files are sent to the control module (GoGateway); the compression algorithm used in the above compression can be gzip or bzip2, etc., to reduce transmission time and bandwidth consumption.

[0055] In an optional implementation, after the collected files are packaged and compressed, cut into a plurality of fragment files, and sent to the control module, the method further includes:

[0056] The control module is made to poll and check whether the plurality of segment files are received within a first preset time, and if so, report a success result of the second collection task, otherwise, report a failure result of the second collection task.

[0057] In a specific embodiment, the execution module sends multiple fragment files to the control module, which may specifically include the execution module sending the fragments to the control module via multiple HTTP requests. The transmission request of each fragment contains information such as the fragment number, the total number of fragments, and the file identifier, so that the control module can perform verification and reorganization.

[0058] In an optional implementation, enabling the control module to forward the plurality of segment files to the service module, enabling the service module to reorganize the plurality of segment files and store them in a first preset location, includes:

[0059] The control module forwards the multiple fragment files to the service module, and the service module polls to check whether the multiple fragment files are received within a second preset time;

[0060] If so, verify the order and integrity of the slices of the multiple slice files to reorganize the files, store the reorganized files into the first preset location, and report the success result of the third collection task;

[0061] Otherwise, report the failure result of the third collection task.

[0062] In a specific embodiment, after receiving the HTTP request, the control module forwards the segmented file to the service module. The control module can verify the integrity and correctness of the segments to ensure that each segment can be successfully transmitted. After receiving all the segments, the service module reassembles them and places them in the first preset position. The reassembly process includes verifying the order and integrity of the segments to ensure the integrity and consistency of the file.

[0063] S103, when the service module receives a file distribution task, the file distribution task is stored in a third task pool of the service module, so that the control module pulls the file distribution task from the third task pool and pulls the to-be-distributed file from the service module, and the service module transmits the to-be-distributed file segments to the control module;

[0064] In a specific embodiment, the service module takes out the files to be distributed from the specified location of the network disk and packages them. The files are packaged in tar, zip, etc. to facilitate transmission and decompression. After the control module receives the task pulled from the service module, it pulls the file from the service module through HTTP request. The service module transmits the slices to the control module through HTTP request. The transmission request for each slice contains information such as the slice number, the total number of slices, and the file identifier, so that the control module can verify and reorganize. After receiving the slices, the control module first verifies the integrity and correctness of the slices, then merges them and verifies the file integrity, and finally stores them for different execution modules to pull.

[0065] In an optional implementation, storing the file distribution task into the third task pool of the service module includes:

[0066] Check whether the target file corresponding to the file distribution task exists in the third preset position of the service module, and if so, store the file distribution task in the task pool of the service module.

[0067] In a specific embodiment, the target file corresponding to the file distribution task may be a file to be distributed. Figure 4 shown. Figure 4 After the next task is completed, the service module (Server) can be split into two tasks and perform three callbacks.

[0068] In an optional embodiment, after pulling the file to be distributed from the service module, it also includes: judging whether the control module pulls the file to be distributed successfully, and if so, reporting the success result of the first distribution task, otherwise, reporting the failure result of the first distribution task;

[0069] After pulling the to-be-distributed file after being segmented from the control module, it also includes judging whether the execution module pulls the to-be-distributed file successfully, and if so, reporting the success result of the second distribution task, otherwise, reporting the failure result of the second distribution task.

[0070] S104, causing the execution module to pull the file distribution task from the task pool of the control module and pull the files to be distributed after being segmented from the control module, and causing the execution module to merge and check the integrity of the files to be distributed after receiving the segmented files, and upload them to the second preset location of the target host.

[0071] In a specific embodiment, after receiving the task pulled from the control module, the execution module pulls the file from the control module through an HTTP request, and the control module transmits the slice to the execution module through the HTTP request. The transmission request of each slice contains information such as the slice number, the total number of slices, and the file identifier, so that the execution module can perform verification and reorganization. After receiving multiple slice files, the execution module first verifies the order, integrity, and correctness of the slices, then merges them and verifies the file integrity, and then uploads them to the specified location of the target host through the SFTP protocol.

[0072] The large file transmission method provided by the embodiment of the present invention comprises the following steps: when the service module receives a file collection task, the file collection task is stored in the first task pool of the service module, the control module pulls the file collection task in the first task pool, and stores the file collection task in the second task pool of the control module, so that the execution module pulls the file collection task in the second task pool; the execution module collects files from the target host according to the file collection task, packages and compresses the collected files, cuts them into multiple fragment files, and sends them to the control module, so that the control module forwards the multiple fragment files to the service module, so that the service module reorganizes the multiple fragment files and stores them in a first preset position; when the service module receives a file distribution task, the file distribution task is stored in the third task pool of the service module, so that the control module pulls the file distribution task from the third task pool and distributes the file to the service module; Pull the file to be distributed, and make the service module transmit the file to be distributed in segments to the control module; make the execution module pull the file distribution task from the task pool of the control module and pull the file to be distributed after segmentation from the control module, and make the execution module merge and check the integrity of the file to be distributed after receiving the segmentation, and upload it to the second preset position of the target host; by cutting the large file into segments for transmission, the problem of insufficient memory when transmitting a large file at a time is avoided, the size of each segment can be flexibly adjusted, and in the multi-layer transmission process, the streaming transmission method is adopted, that is, each segment is forwarded immediately after receiving, so as to avoid the storage space of the transit node being occupied in large quantities, reduce the transmission delay, and improve the efficiency of large file transmission; through segmented transmission and parallel transmission, the network bandwidth is fully utilized, the possibility of network congestion is reduced, it can run stably in different network environments, adapt to various network conditions, and improve the stability of large file transmission.

[0073] The large file transmission method provided by the embodiment of the present invention forwards each shard immediately after receiving it during the multi-layer transmission process, instead of waiting for the file to be completely received before transmitting it, thereby reducing the storage space occupied by the transit node, reducing the waiting time, speeding up the overall transmission speed, and improving the user experience; the storage of the shards on the transit node is temporary, and the storage space is released immediately after the transmission is completed, further reducing the storage burden; the shards are transmitted in parallel through multiple HTTP requests, making full use of the network bandwidth and significantly improving the transmission speed; during the shard transmission process, a check code (such as MD5, SHA-256) is used for verification to ensure the integrity and correctness of each shard and prevent data damage; if a shard fails to be transmitted, it can be automatically retried until all shards are successfully transmitted, ensuring the reliability of the transmission; a multi-layer architecture of the service layer, the control layer and the execution layer is adopted to ensure the high availability and scalability of the system, and the layers communicate through standardized interfaces, making the system easy to maintain and upgrade; it supports the cluster mode deployment of multiple sets of Goproxy and Gogateway to achieve load balancing and improve the overall performance and stability of the system.

[0074] The large file transmission method provided by the embodiment of the present invention can be applied to automated operation and maintenance platforms, large-scale distributed systems, cloud services, data centers, enterprise-level applications, the Internet of Things, the financial industry, the telecommunications industry, the Internet industry, the medical industry, and the education industry.

[0075] Figure 5 is a schematic diagram of the structure of a large file transmission device according to an embodiment of the present invention. Figure 5 As shown, the large file transmission device 50 includes a collection task pulling unit 51, a file collection unit 52, a first distribution task unit 53 and a second distribution task unit 54;

[0076] The acquisition task pulling unit 51 is used for, when the service module receives a file acquisition task, storing the file acquisition task in the first task pool of the service module, causing the control module to pull the file acquisition task in the first task pool, storing the file acquisition task in the second task pool of the control module, causing the execution module to pull the file acquisition task in the second task pool;

[0077] The file collection unit 52 is used to enable the execution module to collect files from the target host according to the file collection task, package and compress the collected files, cut them into multiple fragment files, and send them to the control module, so that the control module forwards the multiple fragment files to the service module, so that the service module reorganizes the multiple fragment files and stores them in the first preset location;

[0078] The first task distribution unit 53 is used for storing the file distribution task into the third task pool of the service module when the service module receives the file distribution task, so that the control module pulls the file distribution task from the third task pool and pulls the to-be-distributed file from the service module, and the service module transmits the to-be-distributed file segments to the control module;

[0079] The second distribution task unit 54 is used to enable the execution module to pull the file distribution task from the task pool of the control module and pull the files to be distributed after being segmented from the control module, and enable the execution module to merge and check the integrity of the files to be distributed after receiving the segmented files, and upload them to the second preset location of the target host.

[0080] In an optional embodiment, the large file transmission device 50 also includes a first judgment unit, which is used to determine whether there is a file in the first task pool of the service module before storing the file acquisition task in the first task pool of the service module; if so, and the latest file needs to be pulled, the file in the first task pool is deleted; the large file transmission device 50 also includes a second judgment unit, which is used to determine whether there is a file in the temporary space of the control module before storing the file acquisition task in the second task pool of the control module; if so, and the latest file needs to be pulled, the file in the temporary space is deleted.

[0081] In an optional implementation, the first task distribution unit 53 stores the file distribution task into the third task pool of the service module, including:

[0082] Check whether the target file corresponding to the file distribution task exists in the third preset position of the service module, and if so, store the file distribution task in the task pool of the service module.

[0083] In an optional implementation, the file collection unit 52 enables the execution module to collect files from the target host according to the file collection task, including:

[0084] The execution module is enabled to collect files from the target host through the file transfer protocol according to the file collection task. If the operation is successful, a success result of the first collection task is reported; otherwise, a failure result of the first collection task is reported.

[0085] In an optional embodiment, the large file transmission device 50 also includes a first checking unit, which, after packaging and compressing the collected files, cuts them into multiple fragment files, and sends them to the control module, causes the control module to poll and check whether the multiple fragment files are received within a first preset time. If so, reports the success result of the second collection task, otherwise, reports the failure result of the second collection task.

[0086] In an optional implementation, the file collection unit 52 enables the control module to forward the multiple segment files to the service module, so that the service module reorganizes the multiple segment files and stores them in the first preset location, including:

[0087] The control module forwards the multiple fragment files to the service module, and the service module polls to check whether the multiple fragment files are received within a second preset time;

[0088] If so, verify the order and integrity of the slices of the multiple slice files to reorganize the files, store the reorganized files into the first preset location, and report the success result of the third collection task;

[0089] Otherwise, report the failure result of the third collection task.

[0090] In an optional embodiment, the large file transmission device 50 also includes a third judgment unit, which is used to judge whether the control module has successfully pulled the file to be distributed after pulling the file to be distributed from the service module. If so, report the success result of the first distribution task, otherwise, report the failure result of the first distribution task; the large file transmission device 50 also includes a third judgment unit, which is used to judge whether the execution module has successfully pulled the file to be distributed after pulling the file to be distributed after being fragmented from the control module. If so, report the success result of the second distribution task, otherwise, report the failure result of the second distribution task.

[0091] Based on the above large file transmission method, an embodiment of the present invention further provides a computer device, which includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus;

[0092] Memory, used to store computer programs;

[0093] The processor is used to implement the steps of the large file transmission method described in any of the above embodiments when executing the program stored in the memory.

[0094] For other details about the processor in the above-mentioned large file transmission device implementing the above-mentioned technical solution, please refer to the description of the large file transmission method provided in the above-mentioned invention embodiment, which will not be repeated here.

[0095] Among them, the processor can also be called CPU (Central Processing Unit), the processor may be an integrated circuit chip with signal processing capabilities; the processor can also be a general-purpose processor, DSP (Digital Signal Process), ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, among which the general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0096] The embodiment of the present invention also provides a computer-readable storage medium, on which a readable computer program is stored; wherein the computer program can be stored in the above storage medium in the form of a software product, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, magnetic disk or optical disk, ROM (Read-Only Memory), RAM (Random Access Memory) and other media that can store program codes, or terminal devices such as computers, servers, mobile phones, and tablets.

[0097] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0098] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0099] In addition, each functional unit in each embodiment of the present application may be integrated into a processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of a software functional unit. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in a computer-readable storage medium.

[0100] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0101] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

[0102] The technical solution provided by the present application is introduced in detail above. The principles and implementation methods of the present application are explained by using specific examples in the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

[0103] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0104] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices and computer program products according to the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0105] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0106] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0107] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A large file transmission method, characterized in that: The method comprises: When the service module receives a file collection task, the file collection task is stored in the first task pool of the service module, the control module is caused to pull the file collection task in the first task pool, the file collection task is stored in the second task pool of the control module, and the execution module is caused to pull the file collection task in the second task pool; The execution module collects files from the target host according to the file collection task, packages and compresses the collected files, cuts them into multiple fragment files, and sends them to the control module, so that the control module forwards the multiple fragment files to the service module, so that the service module reassembles the multiple fragment files and stores them in the first preset location; When the service module receives a file distribution task, the file distribution task is stored in a third task pool of the service module, so that the control module pulls the file distribution task from the third task pool and pulls the file to be distributed from the service module, and the service module transmits the file fragments to be distributed to the control module; The execution module is made to pull the file distribution task from the task pool of the control module and pull the files to be distributed after being segmented from the control module, and after receiving the files to be distributed after being segmented, the execution module is made to merge and check the integrity and upload them to the second preset location of the target host.

2. The large file transmission method according to claim 1, characterized in that: Before storing the file collection task in the first task pool of the service module, it also includes: judging whether there is a file in the first task pool of the service module, and if there is and the latest file needs to be pulled, deleting the file in the first task pool; Before storing the file collection task in the second task pool of the control module, it also includes judging whether there is a file in the temporary space of the control module, and if there is and the latest file needs to be pulled, deleting the file in the temporary space.

3. The large file transmission method according to claim 1, characterized in that: Storing the file distribution task into the third task pool of the service module includes: Check whether the target file corresponding to the file distribution task exists in the third preset position of the service module, and if so, store the file distribution task in the task pool of the service module.

4. The large file transmission method according to claim 1, characterized in that: Enabling the execution module to collect files from the target host according to the file collection task includes: The execution module is enabled to collect files from the target host through the file transfer protocol according to the file collection task. If the operation is successful, a success result of the first collection task is reported; otherwise, a failure result of the first collection task is reported.

5. The large file transmission method according to claim 1, characterized in that: After the collected files are packaged and compressed, they are cut into multiple fragment files and sent to the control module, the process also includes: The control module is made to poll and check whether the plurality of segment files are received within a first preset time, and if so, report a success result of the second collection task, otherwise, report a failure result of the second collection task.

6. The large file transmission method according to claim 1, characterized in that: The control module forwards the plurality of segment files to the service module, and the service module reorganizes the plurality of segment files and stores them in a first preset location, including: The control module forwards the multiple fragment files to the service module, and the service module polls to check whether the multiple fragment files are received within a second preset time; If so, verify the order and integrity of the slices of the multiple slice files to reorganize the files, store the reorganized files into the first preset location, and report the success result of the third collection task; Otherwise, report the failure result of the third collection task.

7. The large file transmission method according to claim 1, characterized in that: After pulling the to-be-distributed file from the service module, the method further includes: determining whether the control module pulls the to-be-distributed file successfully, and if so, reporting a success result of the first distribution task; otherwise, reporting a failure result of the first distribution task; After pulling the to-be-distributed file after being segmented from the control module, it also includes judging whether the execution module pulls the to-be-distributed file successfully, and if so, reporting the success result of the second distribution task, otherwise, reporting the failure result of the second distribution task.

8. A large file transmission device, characterized in that: It includes a collection task pulling unit, a file collection unit, a first distribution task unit and a second distribution task unit; The acquisition task pulling unit is used for, when the service module receives a file acquisition task, storing the file acquisition task in the first task pool of the service module, causing the control module to pull the file acquisition task in the first task pool, storing the file acquisition task in the second task pool of the control module, causing the execution module to pull the file acquisition task in the second task pool; The file collection unit is used to enable the execution module to collect files from the target host according to the file collection task, package and compress the collected files, cut them into multiple fragment files, and send them to the control module, so that the control module forwards the multiple fragment files to the service module, so that the service module reorganizes the multiple fragment files and stores them in the first preset location; The first task distribution unit is used for storing the file distribution task into a third task pool of the service module when the service module receives the file distribution task, so that the control module pulls the file distribution task from the third task pool and pulls the to-be-distributed file from the service module, and the service module transmits the to-be-distributed file segments to the control module; The second distribution task unit is used to enable the execution module to pull the file distribution tasks from the task pool of the control module and pull the files to be distributed after being segmented from the control module, and enable the execution module to merge and check the integrity of the files to be distributed after receiving the segmented files, and upload them to the second preset location of the target host.

9. A computer device, characterized in that: The device includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory, used to store computer programs; The processor is used to implement the steps of the large file transmission method as described in any one of claims 1 to 7 when executing the program stored in the memory.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the large file transmission method according to any one of claims 1 to 7 are implemented.

Images