Netty-based super-large file transmission and breakpoint resume method and Netty-based super-large file transmission and breakpoint resume system

Abstract

The invention provides a Netty-based oversized file transmission and breakpoint resuming method and system, and belongs to the technical field of computer technology and software and oil exploration, and the method comprises the following steps: a Web application end receives a user operation instruction, and transmits an instruction parameter to a client through a REST protocol; when the client side receives the instruction parameters uploaded by the Web application side, a file is subjected to block processing, block information is recorded, and the block information is sent to the server side through an RPC protocol; and the server judges a file uploading state based on the block information, and feeds back the file uploading state to the client. According to the system and the method, the functions of uploading, downloading, breakpoint resuming, transmission progress and state monitoring, file and directory management, client management, task management and the like of oversized files can be realized.

Description

Technical Field

[0001] The present invention relates to the fields of computer technology, software and oil exploration technology, and in particular to a method and system for transmitting and resuming ultra-large files based on Netty. Background Art

[0002] With the rapid development of big data Internet technology, the amount of data in all walks of life is becoming increasingly huge. In the field of oil and gas exploration, seismic data is huge and complex. When transferring TB-level seismic data, it is often copied manually through hard disks, which does not meet the requirements of professionals for lightweight, convenience and transmission efficiency. At the same time, when using hard disks for manual copying, the disk is often damaged due to hard disk collisions, making the data unreadable, which also causes certain hidden dangers and losses to data security; for single ultra-large data, it is often encountered during dumping due to uncontrollable factors. The transmission is interrupted and the file cannot be used after being damaged. In the field of oil exploration, the efficient management of large storage seismic data has gradually become an important way to ensure the quality of operations and the smooth progress of exploration tasks in oil and gas exploration and development. In the prior art, practitioners in the oil exploration industry need to transfer ultra-large files in their daily work, and the existing ultra-large file transmission methods and systems generally have problems such as low transmission efficiency, large network influence, and inability to resume transmission from breakpoints. Summary of the invention

[0003] The purpose of the present invention is to provide a method, system, device and storage medium for large file transmission and breakpoint resumption based on Netty. Aiming at the upload and download requirements of large data bodies in seismic data management in the field of oil exploration, the present invention can support large file upload, download, breakpoint resumption, and real-time monitoring of transmission status, and provide Web-side applications. It is a user-friendly technology with low environmental dependence.

[0004] The present invention provides a method for transmitting and resuming large files based on Netty, the method comprising:

[0005] The Web application receives user operation instructions and transmits instruction parameters to the client through the REST protocol;

[0006] When the client receives the instruction parameter uploaded by the Web application, it processes the file into blocks and records the block information, and sends the block information to the server through the RPC protocol;

[0007] The server determines the file upload status based on the block information, and feeds back the file upload status to the client.

[0008] Preferably, in the step where the Web application accepts user operation instructions and transmits instruction parameters to the client through the REST protocol, the Web application selects a file to be operated according to the user operation instructions and sets instruction parameters. When the instruction parameters are transmitted to the client, the client establishes a connection with the server.

[0009] Preferably, the server determines the file upload status based on the file segmentation information and feeds back the file upload status to the client. Specifically, the server writes the segmented data into a directory in parallel by queue, calculates the size of the segmented data during the writing process to determine the data acceptance status, and sends the file upload status to the client in real time.

[0010] Preferably, after the step of the server determining the file upload status based on the file segmentation information and feeding back the file upload status to the client, the method further includes:

[0011] When the data size is consistent with the size before the file is divided into blocks, it is considered that the reception is complete and data merging is performed. When the client receives the file upload status as completed, the upload is completed and the next task in the queue is carried out, repeating the file transfer work.

[0012] Preferably, when the client receives the instruction parameters uploaded by the Web application, it divides the file into blocks and records the block information, and sends the block information to the server through the RPC protocol. The client records the block information in the client cache in the form of a pointer index.

[0013] Preferably, in the step where the server determines the file upload status based on the block information and feeds back the file upload status to the client, the server creates a temporary file in .fstmp format according to the block information to perform parallel transmission of multiple block data.

[0014] Preferably, the block information includes file transfer location information. When a network anomaly or a manual pause occurs, click the resume function, the client will position the file pointer to the file transfer location information recorded by the client, read the file content after the location information, and continue to transmit the file. The server locates the file breakpoint through the breakpoint position transmitted by the client to write the file, thereby realizing breakpoint resume.

[0015] The present invention also provides a large file transmission and breakpoint resume transmission system based on Netty, comprising:

[0016] Web application: used to receive user operation instructions and transmit instruction parameters to the client through the REST protocol;

[0017] Client: When receiving the instruction parameters uploaded by the Web application, it processes the file into blocks and records the block information, and sends the block information to the server through the RPC protocol.

[0018] Server: deployed on a remote server, used to determine the file upload status based on the block information, and feed back the file upload status to the client, while providing connection number monitoring and traffic monitoring. When overload traffic occurs, a current limiting mechanism will be issued.

[0019] The present invention also provides a data transmission device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, each step of the Netty-based ultra-large file transmission and breakpoint-resume transmission method as described above is implemented.

[0020] The present invention also provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, each step in the above-mentioned Netty-based ultra-large file transmission and breakpoint resume method is implemented.

[0021] The beneficial effects of the present invention include at least:

[0022] In the method and system for transmitting and resuming large files based on Netty of the present invention, the whole service mainly includes three parts, namely, a Web application end, a client end, and a server end. The Web application end accepts user operation instructions, and transmits instruction parameters to the client end through the REST protocol. The client processes the file in blocks based on the instruction parameters and records the block information, and sends the block information to the server end through the RPC protocol. The server end determines the file upload status based on the block information, and feeds back the file upload status to the client end. The present invention realizes high-thread burst transmission by dividing the file into blocks, thereby supporting functions such as uploading, downloading, resuming the transmission from breakpoints, monitoring the progress and status of transmission, file and directory management, client management, and task management of large files, and is equipped with idle detection and keepalive, so that the network is not disturbed when it jitters, and multiple users operate simultaneously; the method and system of the present invention are suitable for the transmission of seismic volume data files by comprehensive researchers in exploration and development.

[0023] The present invention forms a set of standardized, highly automated methods for efficient transmission of large earthquake data bodies that meet industry needs, and develops a Web-side application interface from the perspective of practicality, portability, and interactivity. At the same time, the present invention also provides idle detection and keepalive functions. Idle detection and keepalive refer to that the server will detect in real time that if it does not receive the data request instruction of the client within a period of time, it will automatically disconnect the connection and clean up the idle connection resources in time to ensure that the server is not overused and realize the stable operation of the service, reduce the system necrosis rate, and when the client instruction is initiated, the connection will be reactivated; at the same time, with the keepalive mechanism, it is suitable for situations such as when the transmission time of an ultra-large file is long, it is easily affected by network fluctuations or the client process is dead and disconnected, and the client sends a heartbeat signal if it does not send data within the set time, so as to avoid the connection affected by the network being disconnected, resulting in the termination of transmission; under the double insurance of the two mechanisms, it is achieved that the network is not disturbed when it jitters, the network can be detected when it is disconnected, and the process can be processed in time when it is disconnected. The present invention aims to solve the problems of low efficiency in the transmission of very large files, great network influence, and inability to resume transmission in the daily work of practitioners in the oil exploration industry, and to provide a file transfer service that supports multi-client parallelism, has a high degree of automation, is easy to operate, and is lightweight, and has strong practicality in actual applications.

[0024] The methods and apparatus of the present invention have other features and advantages that will be apparent from or will be described in detail in the accompanying drawings and subsequent detailed descriptions incorporated herein, which together serve to explain the specific principles of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The above and other objects, features and advantages of the present invention will become more apparent through a more detailed description of exemplary embodiments of the present invention with reference to the accompanying drawings.

[0026] Figure 1 The overall structural diagram of the Netty-based ultra-large file transmission and breakpoint-resume transmission system of the present invention is shown;

[0027] Figure 2 The present invention shows a schematic diagram of the process of the method for transferring very large files and resuming breakpoint transfer based on Netty;

[0028] Figure 3 A schematic diagram of file block uploading in the Netty-based ultra-large file transmission and breakpoint resume transmission method of the present invention is shown.

[0029] Figure 4A schematic diagram of a Web application interface in a Netty-based ultra-large file transmission and breakpoint-resume transmission system of the present invention is shown;

[0030] Figure 5 A block diagram of a Netty-based ultra-large file transmission and breakpoint-resume transmission device according to an embodiment of the present invention is shown;

[0031] Figure 6 A block diagram of a storage medium according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in sequences other than those illustrated or described herein.

[0034] It should be understood that in various embodiments of the present invention, the size of the sequence number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0035] It should be understood that in the present invention, "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products or apparatuses.

[0036] It should be understood that in the present invention, "plurality" refers to two or more than two. "And / or" is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "Contains A, B and C", "Contains A, B, C" means that A, B, and C are all included, "Contains A, B or C" means that one of A, B, and C is included, and "Contains A, B and / or C" means that any one, any two, or any three of A, B, and C are included.

[0037] It should be understood that in the present invention, "B corresponding to A", "B corresponding to A", "A corresponds to B" or "B corresponds to A" means that B is associated with A and B can be determined based on A. Determining B based on A does not mean determining B based only on A, but B can also be determined based on A and / or other information. A and B match when the similarity between A and B is greater than or equal to a preset threshold.

[0038] Depending on the context, "if" as used herein may be interpreted as "when" or "when" or "in response to determining" or "in response to detecting."

[0039] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0040] Figure 2 The present invention shows a flow chart of a Netty-based method for transferring very large files and resuming transfers from breakpoints, wherein the method comprises: a Web application receives a user operation instruction and transmits instruction parameters to a client through a REST protocol; when the client receives the instruction parameters uploaded by the Web application, it processes the file in blocks and records the block information, and sends the block information to a server through an RPC protocol; the server determines the file upload status based on the block information, and feeds back the file upload status to the client.

[0041] In a possible implementation, in the process of large file transmission and breakpoint resumption, the present invention is also equipped with idle detection and keepalive, so as to avoid interference when the network jitters and allow multiple users to operate simultaneously. The idle detection and keepalive functions mean that the server will detect in real time that if it does not receive the data request instruction from the client within a period of time, it will automatically disconnect the connection and clean up the idle connection resources in time to ensure that the server is not overused and the service is stable. The necrosis rate of the system is reduced, and when the client instruction is initiated, the connection will be reactivated; at the same time, the keepalive mechanism is used, which is suitable for situations where the large file transmission time is long, it is easily affected by network fluctuations or the client process is dead and disconnected. If the client does not send data within the set time, it will send a heartbeat signal to avoid the connection being disconnected due to the network, resulting in the termination of the transmission; under the double insurance of the two mechanisms, it is possible to avoid interference when the network jitters, detect when the network is disconnected, and handle the process disconnection in time. The present invention solves the problems of low efficiency in super-large file transmission, great network influence, and inability to resume transmission in the daily work of practitioners in the oil exploration industry, and provides a file transmission service that supports multi-client parallelism, has a high degree of automation, is easy to operate, and is lightweight.

[0042] In a possible implementation, in the step of accepting user operation instructions on the Web application side and transmitting instruction parameters to the client through the REST protocol, the Web application side selects the file to be operated according to the user operation instruction and sets the instruction parameters. When the instruction parameters are transmitted to the client, the client establishes a connection with the server. The Web application side directly faces the user and provides the user with a web-based UI operation page, which is mainly used to accept user operation instructions and transmit them to the client. It communicates with the client through the REST protocol. The Web application side is a pure front-end application that supports portability and system integration.

[0043] In a possible implementation, the server determines the file upload status based on the file block information, and the step of feeding back the file upload status to the client specifically includes: the server writes the block data into the directory in parallel by queue, calculates the size of the block data during the writing process to determine the data acceptance status, and sends the file upload status to the client in real time. The server is the core component of the present invention, which is mainly deployed on the remote server to receive the client's operation instructions, perform file transfer operations, and provide connection number monitoring and flow monitoring. When overload flow occurs, the current limiting mechanism will be triggered to ensure the stability of the server and avoid collapse.

[0044] In a possible implementation, after the server determines the file upload status based on the file segmentation information and feeds back the file upload status to the client, it also includes: when the data size is consistent with the size before the file segmentation, it is considered that the reception is complete, and data merging is performed. When the client receives the file upload status as complete, the upload is completed, the next task in the queue is carried out, and the file transfer work is repeated.

[0045] In a possible implementation, when the client receives the instruction parameter uploaded by the Web application, the file is processed into blocks and the block information is recorded, and in the step of sending the block information to the server through the RPC protocol, the client records the block information in the client cache in the form of pointer index. The client is mainly oriented to large file data, which is manifested as an application process opened locally by the client to operate local files. When receiving the upload operation instruction from the Web application, the file is processed into blocks and the block information is recorded and sent to the server. It communicates with the server through the RPC protocol.

[0046] In a possible implementation, in the step where the server determines the file upload status based on the block information and feeds back the file upload status to the client, the server creates a temporary file in .fstmp format according to the block information to perform parallel transmission of multiple block data. Figure 3 The schematic diagram of file block uploading in the Netty-based super-large file transmission and breakpoint resume transmission method of the present invention is shown. When the client receives the transmission instruction, it will scan the size of the file and perform file block according to the set rules. At the same time, the block information is recorded in the client cache in the form of pointer index, and the block information is transmitted to the server. The server creates a temporary file in the .fstmp format according to the fragmentation information, and performs parallel transmission of multiple block data. The fragmentation information also includes the location of the file transmission. When a network abnormality or manual suspension occurs, click the resume transmission function, the client will locate the file pointer to the transmission location recorded by the client, read the file content after the location and continue to transmit, and the server locates the file breakpoint through the breakpoint location transmitted by the client to write the file, thereby realizing breakpoint resume transmission.

[0047] In one possible implementation, the block information includes file transfer location information. When a network anomaly or manual pause occurs, click the resume function, the client will position the file pointer to the file transfer location information recorded by the client, read the location information and continue to transfer the file content. The server locates the file breakpoint through the breakpoint position sent by the client to write the file, thereby realizing breakpoint resume.

[0048] In the Netty-based ultra-large file transmission and breakpoint resume transmission method of the present invention, the entire process must first ensure that the client and the server are interconnected. Starting from the Web application end, the user proposes an operation instruction according to the demand, selects the file to be operated, and sets the instruction parameters. When the instruction reaches the client, the client establishes a connection with the server and sends a file upload request. After the server accepts the request, it will create a file or file directory in the corresponding storage according to the instruction parameters, and feedback the upload instruction to the client. At this time, the client finds the corresponding file and reads it, then executes file segmentation, and sends the segmentation information to the server, and records the segmentation information at the same time. The server writes the segmented data into the directory in parallel according to the queue. During the writing process, the size of the segmented data will be calculated to determine the data acceptance status, and the file upload status will be sent to the client in real time. When the data size is consistent with the size before segmentation, it is considered that the acceptance is complete, and data merging is performed. When the client receives the file upload status as complete, the upload is completed, and the next task in the queue is performed, and the transmission work is repeated.

[0049] Figure 1 The overall structural diagram of the Netty-based ultra-large file transmission and breakpoint resume transmission system of the present invention is shown, and the system includes: Web application end: used to accept user operation instructions and transmit instruction parameters to the client through the REST protocol; client: used to block the file and record the block information when receiving the instruction parameters uploaded by the Web application end, and send the block information to the server through the RPC protocol; server: deployed on the remote server, used to judge the file upload status based on the block information, and feedback the file upload status to the client, while providing connection number monitoring and flow monitoring, and when overload flow occurs, a current limiting mechanism will be issued. The present invention is a large file transmission technical solution composed of three components: file server, client, and Web application end, and realizes large file transmission service based on the high-performance, asynchronous event-driven NIO framework Netty. By realizing high-thread burst transmission by dividing the file into blocks, it supports the functions of uploading, downloading, breakpoint resume, monitoring of transmission progress and status, file and directory management, client management, task management, etc. of ultra-large files.

[0050] Embodiment 1

[0051] According to an embodiment of the present invention, a method for transmitting and resuming a large file based on Netty is provided, and the method includes:

[0052] The Web application receives user operation instructions and transmits instruction parameters to the client through the REST protocol;

[0053] When the client receives the instruction parameter uploaded by the Web application, it processes the file into blocks and records the block information, and sends the block information to the server through the RPC protocol;

[0054] The server determines the file upload status based on the block information, and feeds back the file upload status to the client.

[0055] In some implementations, in the step where the Web application accepts user operation instructions and transmits instruction parameters to the client via the REST protocol, the Web application selects a file to be operated according to the user operation instructions and sets instruction parameters. When the instruction parameters are transmitted to the client, the client establishes a connection with the server.

[0056] In some embodiments, the server determines the file upload status based on the file segmentation information and feeds back the file upload status to the client. Specifically, the server writes the segmented data into a directory in parallel by queue, calculates the size of the segmented data during the writing process to determine the data acceptance status, and sends the file upload status to the client in real time.

[0057] In some implementations, after the server determines the file upload status based on the file segmentation information and feeds back the file upload status to the client, the following step further includes:

[0058] When the data size is consistent with the size before the file is divided into blocks, it is considered that the reception is complete and data merging is performed. When the client receives the file upload status as completed, the upload is completed and the next task in the queue is carried out, repeating the file transfer work.

[0059] In some implementations, when the client receives the instruction parameters uploaded by the Web application, it processes the file into blocks and records the block information, and sends the block information to the server through the RPC protocol. The client records the block information in the client cache in the form of a pointer index.

[0060] In some implementations, in the step where the server determines the file upload status based on the block information and feeds back the file upload status to the client, the server creates a temporary file in .fstmp format according to the block information to perform parallel transmission of multiple block data.

[0061] In some embodiments, the block information includes file transfer location information. When a network abnormality or manual pause occurs, the resume function is clicked, and the client will position the file pointer to the file transfer location information recorded by the client, and continue to transmit the file content after reading the location information. The server locates the file breakpoint through the breakpoint position transmitted by the client to write the file, thereby realizing breakpoint resume.

[0062] Embodiment 2

[0063] According to an embodiment of the present invention, a large file transmission and breakpoint resume transmission system based on Netty is provided, including:

[0064] Web application: used to receive user operation instructions and transmit instruction parameters to the client through the REST protocol;

[0065] Client: When receiving the instruction parameters uploaded by the Web application, it processes the file into blocks and records the block information, and sends the block information to the server through the RPC protocol.

[0066] Server: deployed on a remote server, used to determine the file upload status based on the block information, and feed back the file upload status to the client, while providing connection number monitoring and traffic monitoring. When overload traffic occurs, a current limiting mechanism will be issued.

[0067] In some implementations, in the step where the Web application accepts user operation instructions and transmits instruction parameters to the client via the REST protocol, the Web application selects a file to be operated according to the user operation instructions and sets instruction parameters. When the instruction parameters are transmitted to the client, the client establishes a connection with the server.

[0068] In some embodiments, the server determines the file upload status based on the file segmentation information and feeds back the file upload status to the client. Specifically, the server writes the segmented data into a directory in parallel by queue, calculates the size of the segmented data during the writing process to determine the data acceptance status, and sends the file upload status to the client in real time.

[0069] In some implementations, after the server determines the file upload status based on the file segmentation information and feeds back the file upload status to the client, the following step further includes:

[0070] When the data size is consistent with the size before the file is divided into blocks, it is considered that the reception is complete and data merging is performed. When the client receives the file upload status as completed, the upload is completed and the next task in the queue is carried out, repeating the file transfer work.

[0071] In some implementations, when the client receives the instruction parameters uploaded by the Web application, it processes the file into blocks and records the block information, and sends the block information to the server through the RPC protocol. The client records the block information in the client cache in the form of a pointer index.

[0072] In some implementations, in the step where the server determines the file upload status based on the block information and feeds back the file upload status to the client, the server creates a temporary file in .fstmp format according to the block information to perform parallel transmission of multiple block data.

[0073] In some embodiments, the block information includes file transfer location information. When a network abnormality or manual pause occurs, the resume function is clicked, and the client will position the file pointer to the file transfer location information recorded by the client, and continue to transmit the file content after reading the location information. The server locates the file breakpoint through the breakpoint position transmitted by the client to write the file, thereby realizing breakpoint resume.

[0074] Embodiment 3

[0075] According to another aspect of the present invention, there is also provided a data transmission device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, a method for transmitting a very large file and resuming the transmission at breakpoint based on Netty is implemented, and the method comprises:

[0076] The Web application receives user operation instructions and transmits instruction parameters to the client through the REST protocol;

[0077] When the client receives the instruction parameter uploaded by the Web application, it processes the file into blocks and records the block information, and sends the block information to the server through the RPC protocol;

[0078] The server determines the file upload status based on the block information, and feeds back the file upload status to the client.

[0079] In some implementations, in the step where the Web application accepts user operation instructions and transmits instruction parameters to the client via the REST protocol, the Web application selects a file to be operated according to the user operation instructions and sets instruction parameters. When the instruction parameters are transmitted to the client, the client establishes a connection with the server.

[0080] In some embodiments, the server determines the file upload status based on the file segmentation information and feeds back the file upload status to the client. Specifically, the server writes the segmented data into a directory in parallel by queue, calculates the size of the segmented data during the writing process to determine the data acceptance status, and sends the file upload status to the client in real time.

[0081] In some implementations, after the server determines the file upload status based on the file segmentation information and feeds back the file upload status to the client, the following step further includes:

[0082] When the data size is consistent with the size before the file is divided into blocks, it is considered that the reception is complete and data merging is performed. When the client receives the file upload status as completed, the upload is completed and the next task in the queue is carried out, repeating the file transfer work.

[0083] In some implementations, when the client receives the instruction parameters uploaded by the Web application, it processes the file into blocks and records the block information, and sends the block information to the server through the RPC protocol. The client records the block information in the client cache in the form of a pointer index.

[0084] In some implementations, in the step where the server determines the file upload status based on the block information and feeds back the file upload status to the client, the server creates a temporary file in .fstmp format according to the block information to perform parallel transmission of multiple block data.

[0085] In some embodiments, the block information includes file transfer location information. When a network abnormality or manual pause occurs, the resume function is clicked, and the client will position the file pointer to the file transfer location information recorded by the client, and continue to transmit the file content after reading the location information. The server locates the file breakpoint through the breakpoint position transmitted by the client to write the file, thereby realizing breakpoint resume.

[0086] Embodiment 4

[0087] According to another aspect of the present invention, there is also provided a storage medium on which a computer program is stored. When the computer program is executed by a processor, each step in the method for transferring very large files and resuming breakpoint transfer based on Netty is implemented. The steps include:

[0088] The Web application receives user operation instructions and transmits instruction parameters to the client through the REST protocol;

[0089] When the client receives the instruction parameter uploaded by the Web application, it processes the file into blocks and records the block information, and sends the block information to the server through the RPC protocol;

[0090] The server determines the file upload status based on the block information, and feeds back the file upload status to the client.

[0091] In some implementations, in the step where the Web application accepts user operation instructions and transmits instruction parameters to the client via the REST protocol, the Web application selects a file to be operated according to the user operation instructions and sets instruction parameters. When the instruction parameters are transmitted to the client, the client establishes a connection with the server.

[0092] In some embodiments, the server determines the file upload status based on the file segmentation information and feeds back the file upload status to the client. Specifically, the server writes the segmented data into a directory in parallel by queue, calculates the size of the segmented data during the writing process to determine the data acceptance status, and sends the file upload status to the client in real time.

[0093] In some implementations, after the server determines the file upload status based on the file segmentation information and feeds back the file upload status to the client, the following step further includes:

[0094] When the data size is consistent with the size before the file is divided into blocks, it is considered that the reception is complete and data merging is performed. When the client receives the file upload status as completed, the upload is completed and the next task in the queue is carried out, repeating the file transfer work.

[0095] In some implementations, when the client receives the instruction parameters uploaded by the Web application, it processes the file into blocks and records the block information, and sends the block information to the server through the RPC protocol. The client records the block information in the client cache in the form of a pointer index.

[0096] In some implementations, in the step where the server determines the file upload status based on the block information and feeds back the file upload status to the client, the server creates a temporary file in .fstmp format according to the block information to perform parallel transmission of multiple block data.

[0097] In some embodiments, the block information includes file transfer location information. When a network abnormality or manual pause occurs, the resume function is clicked, and the client will position the file pointer to the file transfer location information recorded by the client, and continue to transmit the file content after reading the location information. The server locates the file breakpoint through the breakpoint position transmitted by the client to write the file, thereby realizing breakpoint resume.

[0098] Embodiment 5

[0099] The present invention has been applied many times in actual production and its feasibility has been verified. In actual work, a single file of about 140G of seismic data has been transmitted, such as Figure 4It is the web application interface of the file when it is transmitted. The server is configured as CentOS Linux release 7.5.1804, 64G memory, 12 cores, x86_64-bit architecture, 10Gbps network bandwidth between server and storage, and the client is configured as Windows11 Professional Edition, Intel(R)Core(TM)i7-9700 CPU@3.00GHz, 16G memory, 8 cores, 64 bits. The usage scenario is to upload local files to server storage, start the file transfer server and client, and start the web application in the same network segment to control data transmission. Select the corresponding local file and remote server directory, click upload to wake up the transmission task, when multiple files are uploaded, the service will queue in queue order to implement the first-in-first-out mechanism. If the network is disconnected during the transmission process, the transmission process will be suspended. When the network is restored, click to continue the transmission, and the transmission process will continue until it is completed. In this embodiment, the average upload speed measured by the speed test tool is 300MB / s, and the total time for the file transmission is about 7 minutes. The size of the file after transmission is consistent with the original size and can be read normally.

[0100] Embodiment 6

[0101] The present invention can also be used as a remote file and directory management tool. When file management is required for remote distributed storage, the distributed storage only needs to be mounted on the corresponding server in the NTFS protocol format. After the client initiates a request, it will connect to the server. At this time, the user can access the file data mounted on the disk and add, delete, modify and query files.

[0102] In other embodiments, the seismic data body is mainly SEGY. In actual use, large files other than seismic data can also be transmitted, such as software installation packages, documents, compressed packages, etc. The transmission process is basically the same and will not be repeated here.

[0103] The Netty-based ultra-large file transmission and breakpoint resume transmission method and system of the present invention are based on the network communication framework: Netty, and front-end technologies: Javascript, vue3.0; the above technologies are relatively mature and can be directly used for the level achieved by the present invention.

[0104] The present invention can be directly used in the field of oil and gas exploration or other industries for large file transmission needs, especially single file upload and download of hundreds of GB of large files, as well as breakpoint resume transmission needs, effectively solving some problems in the daily dumping of huge seismic data, greatly improving the work efficiency of business personnel, reducing time costs, ensuring the security of data transmission, and having certain practicality; and the service also provides a front-end Web application interface, which can be seamlessly transplanted to commonly used data management systems, avoiding the cumbersomeness of desktop-level software tools. The present invention can realize the back-end upload, download, breakpoint resume, transmission progress and status monitoring of large files, as well as Web front-end file and directory management, task management and other functions, and is a user-friendly technology with low environmental dependence.

[0105] Figure 5 A block diagram of a device 800 for transmitting and resuming large files based on Netty according to an embodiment of the present invention is shown. For example, the data transmission device 800 may be a terminal device such as a mobile phone, a computer, a digital broadcast terminal, a message transceiver device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc. The data transmission device includes a memory 804, a processor 802, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, each step in the above-mentioned method for transmitting and resuming large files based on Netty is implemented.

[0106] The data transmission device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output interface 812 , a sensor component 814 , and a communication component 816 .

[0107] The processing component 802 generally controls the overall operation of the data transmission device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0108] The memory 804 is configured to store various types of data to support operations on the data transmission device 800. Examples of such data include instructions for any application or method operating on the data transmission device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0109] The power supply component 806 provides power to the various components of the data transmission device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the data transmission device 800.

[0110] The multimedia component 808 includes a screen that provides an output interface between the data transmission device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the edge of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the data transmission device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0111] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the data transmission device 800 is in an operation mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0112] The input / output interface 812 provides an interface between the processing component 802 and the peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0113] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the data transmission device 800. For example, the sensor assembly 814 can detect the open / closed state of the data transmission device 800, the relative positioning of components, such as the display and keypad of the data transmission device 800, and the sensor assembly 814 can also detect the position change of the data transmission device 800 or a component of the data transmission device 800, the presence or absence of user contact with the data transmission device 800, the orientation or acceleration / deceleration of the data transmission device 800, and the temperature change of the data transmission device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0114] The communication component 816 is configured to facilitate wired or wireless communication between the data transmission device 800 and other devices. The data transmission device 800 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0115] In an exemplary embodiment, the data transmission device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above method.

[0116] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 804 including computer program instructions, which can be executed by the processor 820 of the data transmission device 800 to perform the above method.

[0117] Figure 6 1 is a block diagram of an electronic device 1900 according to an embodiment of the present invention. For example, the electronic storage medium 1900 may be provided as a server or a terminal. Figure 6 , the electronic storage medium 1900 includes a processing unit 1922, which further includes one or more processors, and a memory resource represented by a storage unit 1932 for storing instructions executable by the processing unit 1922, such as an application. The application stored in the storage unit 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing unit 1922 is configured to execute the instructions to perform the above method.

[0118] The electronic storage medium 1900 may also include a power supply unit 1926 configured to perform power management of the electronic storage medium 1900, a wired or wireless network interface 1950 configured to connect the electronic storage medium 1900 to a network, and an input / output interface 1958. The electronic storage medium 1900 may operate based on an operating system stored in the storage unit 1932, such as Windows Server 2000. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM or similar.

[0119] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a storage unit 1932 including computer program instructions that can be executed by the processing unit 1922 of the electronic storage medium 1900 to perform the above method.

[0120] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0121] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0122] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0123] The computer program instructions for performing the operation of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present invention.

[0124] Various aspects of the present invention are described herein with reference to the flow charts and / or block diagrams of the methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each box of the flow chart and / or block diagram and the combination of each box in the flow chart and / or block diagram can be implemented by computer-readable program instructions.

[0125] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0126] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0127] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0128] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK) and the like.

[0129] It can be understood that the above embodiments mentioned in the present invention can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, the present invention will not repeat them. It can be understood by those skilled in the art that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0130] Note that, unless otherwise directly stated, all features disclosed in this specification (including any attached claims, abstracts and drawings) may be replaced by alternative features for achieving the same, equivalent or similar purposes. Therefore, unless otherwise explicitly stated, each feature disclosed is only an example of a group of equivalent or similar features. Where used, further, preferably, further and more preferably are simple beginnings for elaborating another embodiment based on the aforementioned embodiment, and the content of the further, preferably, further or more preferably followed by the combination with the aforementioned embodiment constitutes a complete construction of another embodiment. Several further, preferably, further or more preferably settings following the same embodiment can be arbitrarily combined to form another embodiment.

[0131] It should be understood by those skilled in the art that the embodiments of the present invention described above and shown in the accompanying drawings are only examples and do not limit the present invention. The purpose of the present invention has been fully and effectively achieved. The functional and structural principles of the present invention have been demonstrated and explained in the embodiments, and the embodiments of the present invention may be deformed or modified in any way without departing from the principles.

[0132] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for transferring large files and resuming breakpoints based on Netty. It is characterized in that The method comprises: The Web application receives user operation instructions and transmits instruction parameters to the client through the REST protocol; When the client receives the instruction parameter uploaded by the Web application, it processes the file into blocks and records the block information, and sends the block information to the server through the RPC protocol; The server determines the file upload status based on the block information, and feeds back the file upload status to the client.

2. According to the Netty-based method for transferring large files and resuming transfers at breakpoints in claim 1, It is characterized in that In the step where the Web application receives user operation instructions and transmits instruction parameters to the client through the REST protocol, the Web application selects a file to be operated according to the user operation instructions and sets instruction parameters. When the instruction parameters are transmitted to the client, the client establishes a connection with the server.

3. According to the Netty-based large file transmission and breakpoint resume method of claim 1, It is characterized in that The server determines the file upload status based on the file block information and feeds back the file upload status to the client. Specifically, the server writes the block data into a directory in parallel according to the queue, calculates the size of the block data during the writing process to determine the data acceptance status, and sends the file upload status to the client in real time.

4. According to the Netty-based method for transferring large files and resuming transfers at breakpoints as claimed in claim 1, It is characterized in that After the server determines the file upload status based on the file segmentation information and feeds back the file upload status to the client, the following step further includes: When the data size is consistent with the size before the file is divided into blocks, it is considered that the reception is complete and data merging is performed. When the client receives the file upload status as completed, the upload is completed and the next task in the queue is carried out, repeating the file transfer work.

5. According to the Netty-based method for transferring large files and resuming transfers at breakpoints as claimed in claim 1, It is characterized in that When the client receives the instruction parameters uploaded by the Web application, it divides the file into blocks and records the block information, and sends the block information to the server through the RPC protocol. The client records the block information in the client cache in the form of pointer index.

6. According to the Netty-based method for transferring large files and resuming breakpoint transfers, It is characterized in that In the step where the server determines the file upload status based on the block information and feeds back the file upload status to the client, the server creates a temporary file in .fstmp format according to the block information to perform parallel transmission of multiple block data.

7. According to the Netty-based method for transferring large files and resuming transfers at breakpoints as claimed in claim 1, It is characterized in that The block information includes file transfer location information. When a network anomaly or manual pause occurs, click the resume function, the client will position the file pointer to the file transfer location information recorded by the client, read the file content after the location information, and continue to transmit. The server locates the file breakpoint through the breakpoint position transmitted by the client to write the file, thereby realizing breakpoint resume.

8. A large file transfer and breakpoint resume system based on Netty, It is characterized in that include: Web application: used to receive user operation instructions and transmit instruction parameters to the client through the REST protocol; Client: When receiving the instruction parameters uploaded by the Web application, it processes the file into blocks and records the block information, and sends the block information to the server through the RPC protocol. Server: deployed on a remote server, used to determine the file upload status based on the block information, and feed back the file upload status to the client, while providing connection number monitoring and traffic monitoring. When overload traffic occurs, a current limiting mechanism will be issued.

9. A data transmission device, It is characterized in that The invention comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, each step of the method for transferring very large files and resuming breakpoints based on Netty as described in any one of claims 1 to 7 is implemented.

10. A storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, each step of the method for transmitting and resuming large files based on Netty as described in any one of claims 1 to 7 is implemented.

Images