A method and system for real-time transmission and processing of large files based on redis

A technology of real-time transmission and processing methods, applied in the computer field, can solve problems such as low security, inability to resume transmission from breakpoints, and low processing timeliness, so as to achieve the effect of transmission efficiency

CN112437160BActive Publication Date: 2022-06-07SOUTHWEST CHINA RES INST OF ELECTRONICS EQUIP
8 Cites 0 Cited by

Patent Information

Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Publication Date
2022-06-07

Smart Images

  • Figure 1
    Figure 1
  • Figure 2
    Figure 2
  • Figure 3
    Figure 3
Patent Text Reader

Abstract

The invention relates to the field of computer technology, and discloses a method and system for real-time transmission and processing of large files based on redis. The method in the present invention mainly includes: the sending unit detects the file list to be processed in the local directory, and when a large file is detected, the file is virtual-paged, the sending unit assigns each thread unit to upload tasks, and reads the content of the specified page and sends it to the redis server Store; the receiving unit detects whether there is an update in the first ordered set of the list of large file names in the redis server, and if there is an update, then detects the newly added page in the second ordered set as the content of each page of the large file, and puts the new The content of the added page is put into the cache queue; the data in the cache queue is obtained and processed in real time until the data in the cache queue is processed; the entire file transmission status is monitored, and if the transmission status is abnormal, retransmission is performed. The method and system provided by the invention can realize real-time transmission of large files and real-time processing.
Need to check novelty before this filing date? Find Prior Art

Description

technical field

[0001] The invention relates to the field of computer technology, in particular to a method and system for realizing real-time transmission and processing of large files based on redis. Background technique

[0002] There are two main methods of traditional cross-network file processing: First, use the shared network method, directly write large files into the shared directory, and both the sender and the receiver can access through the shared method, but this method has low security; Second, use the FTP method , upload the large file to the FTP server through FTP, and the receiving end determines whether there is a new file by periodically detecting the FTP directory, and detects whether the file is uploaded. The security of this method is better than that of the first method, but the processing time is relatively low, especially when the file is very large (tens of hundreds of gigabytes or even larger), it takes a long time to wait until the entire file upl...

Examples

Embodiment Construction

[0040] The present invention will be further described below with reference to the accompanying drawings.

[0041] like figure 1 As shown, an embodiment of the present invention provides a method for realizing real-time transmission and processing of large files based on redis, including the steps of uploading large files, detecting redis data updates, data sorting and processing, and monitoring file transmission status. details as follows:

[0042] Step 1, uploading large files: the sending unit detects the list of files to be transmitted in the local directory, and after detecting the files, virtual paging the files, wherein the paging size is determined according to the actual network bandwidth;

[0043] The sending unit reads the content of the specified page in a memory offset manner through multi-threading + multi-process and sends it to the redis server for storage, so as to maximize the transmission efficiency;

[0044] The sending unit allocates resource usage betwe...