Method and system for realizing real-time transmission processing of large file based on redis

Abstract

The invention relates to the technical field of computers, and discloses a redis-based large file real-time transmission processing method and system. The method mainly comprises the steps that a sending unit detects a to-be-processed file list of a local directory, after a large file is detected, the file is virtually paged, and the sending unit allocates uploading tasks of all thread units, reads specified page content and sends the specified page content to a redis server to be stored; a receiving unit detects whether a first ordered set serving as a large file name list in the redis serveris updated or not: if yes, detecting newly-added pages in a second ordered set serving as the content of each page of the large file, and putting the content of the newly-added pages into a cache queue; acquiring cache queue data, and performing real-time processing until the data of the cache queue is completely processed; monitoring the whole file transmission state, and if the transmission state is abnormal, carrying out retransmission. The method and the system provided by the invention not only can realize real-time transmission of large files, but also can carry out real-time processing.

Description

technical field

[0001] The invention relates to the field of computer technology, in particular to a method and system for real-time transmission and processing of large files based on redis. Background technique

[0002] There are two traditional cross-network file processing methods: 1. Use the shared network method to directly write large files into the shared directory, and both the sender and the sender can access through the shared method, but this method is low in security; 2. Use the FTP method , upload large files to the FTP server through FTP, and the receiving end judges whether there are new files by periodically checking the FTP directory, and checks whether the upload is complete. Only when the upload is completed can the processing begin. This method is more secure than method 1, but the processing timeliness is relatively low, especially when the file is very large (tens of hundreds of gigabytes or even larger), it will take a long time to wait until the enti...

Examples

Embodiment Construction

[0040] The present invention will be further described below in conjunction with the accompanying drawings.

[0041] Such as figure 1 As shown, the embodiment of the present invention provides a method for real-time transmission of large files based on redis, including several steps of uploading large files, detecting redis data updates, sorting data, and monitoring file transmission status. details as follows:

[0042] Step 1, uploading large files: the sending unit detects the list of files to be transmitted in the local directory, and when the files are detected, the files are virtually paged, and the page size is determined according to the actual network bandwidth;

[0043] The sending unit reads the content of the specified page in a memory offset mode through multithreading+multiprocess and sends it to the redis server for storage, so as to maximize the transmission efficiency;

[0044] The sending unit allocates resource usage among multi-threads and multi-processes ...

technical field

[0001] The invention relates to the field of computer technology, in particular to a method and system for real-time transmission and processing of large files based on redis. Background technique

[0002] There are two traditional cross-network file processing methods: 1. Use the shared network method to directly write large files into the shared directory, and both the sender and the sender can access through the shared method, but this method is low in security; 2. Use the FTP method , upload large files to the FTP server through FTP, and the receiving end judges whether there are new files by periodically checking the FTP directory, and checks whether the upload is complete. Only when the upload is completed can the processing begin. This method is more secure than method 1, but the processing timeliness is relatively low, especially when the file is very large (tens of hundreds of gigabytes or even larger), it will take a long time to wait until the enti...

Examples

Embodiment Construction

[0040] The present invention will be further described below in conjunction with the accompanying drawings.

[0041] Such as figure 1 As shown, the embodiment of the present invention provides a method for real-time transmission of large files based on redis, including several steps of uploading large files, detecting redis data updates, sorting data, and monitoring file transmission status. details as follows:

[0042] Step 1, uploading large files: the sending unit detects the list of files to be transmitted in the local directory, and when the files are detected, the files are virtually paged, and the page size is determined according to the actual network bandwidth;

[0043] The sending unit reads the content of the specified page in a memory offset mode through multithreading+multiprocess and sends it to the redis server for storage, so as to maximize the transmission efficiency;

[0044] The sending unit allocates resource usage among multi-threads and multi-processes ...

Images