A decentralized file transfer method

Abstract

The present invention belongs to the field of computer software and network technology, and is specifically a decentralized file transmission method. The present invention adopts a method combining in-network cache and network broadcast request to realize decentralized file transmission service, solves the resource location problem existing in BitTorrent by dynamically detecting producers in the network, and effectively manages network resources by analyzing and recording network conditions; it is manifested in that file data is transmitted between file producers and consumers in a "request first, then respond" manner; the file consumer issues a request and broadcasts it within a certain range of the network; all producers who can respond to the corresponding request can return data; the user acts as both a consumer and a producer, and can monitor the network status in real time and dynamically adjust the request sending strategy, thereby making full use of the user's uplink bandwidth, alleviating the load problem of popular servers, and also providing a solution for resource location and management to a certain extent.

Description

Technical Field

[0001] The present invention belongs to the field of computer software and network technology, and in particular relates to a decentralized file transmission method. Background Art

[0002] Traditional file transfer protocols such as FTP and SFTP are both end-to-end transmission services based on the C / S architecture. Servers with popular resources are prone to become performance bottlenecks, and decentralization has become a trend. BitTorrent, a file distribution protocol based on P2P, fully utilizes the user's upstream bandwidth, alleviates the server load, and improves the efficiency of file transfer in multi-user scenarios. However, how to locate and manage resources in a large-scale distributed environment, and how to choose an effective resource request routing strategy, are very challenging issues. Summary of the invention

[0003] The purpose of the present invention is to provide a centralized file transmission method to solve the resource location problem existing in BitTorrent and to manage network resources more effectively.

[0004] The decentralized file transmission method proposed in the present invention realizes the decentralized file transmission service by combining the in-network cache and the network broadcast request, solves the resource location problem existing in BitTorrent by dynamically detecting the producer in the network, and effectively manages the network resources by analyzing and recording the network status; the network is composed of an intermediate router, producer and consumer nodes, and the nodes are connected by a network link for transmitting data. The producer is the party that provides the file data, and the consumer is the party that requests the file data. The intermediate router manages the routing information and forwarding process of the network; the producer and the consumer need to inform the intermediate router by an announcement, so as to become a node in the network and form a network topology; the file data transmission is performed between the file producer and the consumer in a "request first, then response" manner, and the specific steps are as follows.

[0005] 1. The file consumer sends a request to the producer

[0006] (1) The request issued by the file consumer can be broadcast within a certain range of the network. Figure 2 is a diagram of the process of a file consumer requesting a file. The file consumer first broadcasts the file request to the network. The request is a data packet issued by the consumer (referred to as the request data packet), which contains a structured name used to identify the file content (such as A / video / mp4 / a / 1, which identifies "segment 1 of video file a in mp4 format published by application A"), information about the request broadcast range, consumer identification information, and legitimacy verification information.

[0007] (2) After the consumer sends a request to the network, it waits for a response from the producer. If no response is received, it repeats the broadcast file request. If there is still no response after repeating it three times, it is determined that there is no producer in the network that can provide the corresponding file transfer service, and the request process ends.

[0008] (3) After receiving the response from the producer, record all the producer information and file information of the response, including the producer identification, file size, and the number of segments when the file is transferred (since the file may be larger than the maximum capacity of a single packet of the underlying network protocol, the file needs to be transferred in segments); initialize the request queue according to the number of segments, and for each responding producer, allocate a send window entity, which contains a value (send window) used to control the rate at which requests are sent; find a producer whose send window is not full from all send window entities; if no producer is found, enter the waiting phase;

[0009] (4) Select a file segment request from the request queue, then add the producer's identification information to the request, and then send it to the network and listen for the corresponding data arrival event; repeat the process of sending the request until all data segments are received and the file request process ends.

[0010] In the present invention, a file consumer can specify a specific file producer so that the request will only reach the producer along a certain path. This is a request strategy generally used when the consumer already knows the request monitoring pool of the producer.

[0011] (II) Producer’s response to consumer’s request

[0012] All producers that can respond to the corresponding request can return data; the file producer receives the request data packet arriving from the network, and searches for the corresponding file data from the local request listening pool based on the file content identification information in the request data packet. If a hit is found, the corresponding data is packaged into a response data packet, and the destination address of the response data packet is specified based on the file consumer identification information in the request data packet. The response data packet is then transmitted to the corresponding consumer through the underlying network protocol; the response data packet also contains the producer-related verification information, network status and the transmission service quality promised by the producer.

[0013] In the present invention, a file consumer can record a request item being monitored by a file producer and the status of a network link to a producer, and manage network resources.

[0014] In the present invention, the file consumer can also act as a producer, cache the received data, and then monitor the request for the data; after the legal data is verified by the consumer, the data and its corresponding structured naming identifier can be placed in the consumer's request monitoring pool, so that the consumer can respond to the corresponding data as the producer after receiving the same request.

[0015] In the present invention, the consumer mainly includes three modules: a buffer module, a sending window module and an event monitoring module, which transfer files between the application and the network; the buffer module is used to cache the data packets that arrive out of order, restore the complete file after sorting, and finally submit it to the user; the sending window module is responsible for controlling the rate of the entire sending process and the transmission task quota allocated to each producer, and sends the request with the producer's identification to the event monitoring module; the event monitoring module is responsible for processing the data arrival and response timeout events of each request, sending requests and receiving data to the network, caching the received file fragments into the buffer module, and adjusting the behavior of the sending window module through the network status information extracted during the monitoring process; in particular, the application submits the broadcast file transfer request to the event monitoring module to explore the producer in the network that is suitable for providing transmission services. The data flow of the file consumer can be found in Figure 4 shown.

[0016] In the present invention, the requests sent by the file consumer are bound to the two events of "data packet arrival" and "response timeout"; the processing process of the event monitoring module is as follows: when data arrives at the file consumer or the monitored request packet response times out, different processing logics are triggered; if the event of data packet arrival is monitored, the correctness of the data is first verified, and if correct, the data is written into the buffer, the network status information of the producer providing the data is updated, and the value of the maximum sending window is adjusted according to the delay-bandwidth product; if the event of request response timeout is monitored, the request is re-added to the request queue, and it is considered that the request rate to the producer is too high, and the value of the maximum sending window to the producer should be reduced accordingly; see Figure 3.

[0017] In the present invention, regarding the joining of producer and consumer networks:

[0018] When a host wants to become a producer that can provide transmission services, it needs to first send a data packet to the network to announce its own identity and its request listening pool; for the consumer, it only needs to announce its own identity to join the network; the intermediate router builds a routing table by recording the information announced by the producer and consumer; the routing table not only records the relationship between the producer and consumer identities and the forwarding path, but also records the relationship between the structured naming identifier prefix and the forwarding path; the process of forwarding request packets and response packets needs to refer to the routing table information, and match it in the routing table according to the structured naming identifier or receiver identifier contained in the packet (when the packet specifies the receiver of the packet), so as to know which interfaces the packet should be forwarded to, thereby building the network topology.

[0019] Beneficial Effects

[0020] The file transmission method of the present invention can make full use of the user's uplink bandwidth, alleviate the load of popular servers, and have higher file transmission efficiency in multi-user scenarios; it can monitor the network status in real time during the transmission process, dynamically adjust the data request strategy, and have a certain degree of adaptability to changes in network status; it can locate and manage network resources on the consumer side, thereby improving the utilization rate of network resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] FIG. 1 is a diagram showing the deployment of a centralized file transfer method.

[0022] FIG. 2 is a diagram showing the process of a file consumer requesting party issuing a request in the present invention.

[0023] FIG. 3 is a diagram showing the monitoring process of the event monitoring module in the present invention.

[0024] FIG. 4 is a diagram showing the file consumer module structure and file transfer in the present invention. DETAILED DESCRIPTION

[0025] Refer to Figure 1, which is a flowchart of the file transmission method of the present invention. Host A, as a file consumer, requests file F, and the identifier of file F is "xx / xx / F / *", "*" represents an arbitrary number, which is used as the segment number of file F; servers 1, 2, and 3 can all provide services for transmitting file F; host B, as a file consumer, requests and successfully receives file F earlier than host A, and will add file F to the listening pool. When host A requests file F, host A first broadcasts a request identified as "xx / xx / F" to explore the producers in the network that can provide file F transfer services; servers one, two, three and host B will respond and send a data packet containing file information, verification information and respond to host A; host A verifies each producer according to the information contained in the received data packet, and assigns a sending window to each producer and assigns transmission tasks. For example, if the topological distance between host B and host A is closer, then host B's sending window will be larger and it will undertake more transmission tasks; during the file fragment data transmission process, the sending window size of each producer is adjusted according to the actual transmission rate; the received data fragments will be cached in the buffer first, sorted and organized according to the segment number in the data packet identifier, and then submitted to the corresponding application.

Claims

1. A decentralized file transfer method, characterized in that: The decentralized file transfer service is realized by combining the in-network cache and the network broadcast request. The resource location problem of BitTorrent is solved by dynamically detecting the producers in the network, and the network resources are effectively managed by analyzing and recording the network status. The network is composed of an intermediate router, producer and consumer nodes, and the nodes are connected by a network link for transmitting data. The producer is the party that provides the file data, and the consumer is the party that requests the file data. The intermediate router manages the routing information and forwarding process of the network. The producer and the consumer need to inform the intermediate router by announcing, so as to become a node in the network and form a network topology. The file data transmission is performed between the file producer and the consumer in a "request first, then response" manner, and the specific steps are as follows:

1. The file consumer sends a request to the producer (1) The request sent by the file consumer is broadcast within a certain range of the network. The file consumer first broadcasts the file request to the network. The request is a data packet sent by the consumer, called a request data packet, which contains a structured name used to identify the file content, information about the request broadcast range, the consumer's identification information, and legitimacy verification information. (2) After the consumer sends a request to the network, it waits for a response from the producer. If no response is received, it repeats the broadcast file request. If there is still no response after repeating it three times, it is determined that there is no producer in the network that can provide the corresponding file transfer service, and the request process ends. (3) After receiving the response from the producer, record all the producer information and file information of the response, including the producer ID, file size, and the number of segments when the file is transferred. Initialize the request queue according to the number of segments. For each responding producer, assign a send window entity. The entity contains a value, namely the send window, which is used to control the rate at which requests are sent. Find a producer whose send window is not full from all the send window entities. If no producer is found, enter the waiting phase. (4) Select a file segment request from the request queue, add the producer's identification information to the request, send it to the network and listen for the corresponding data arrival event; repeat the process of sending the request until all data segments are received and the file request process ends; (II) Producer’s response to consumer’s request All producers that can respond to the corresponding request can return data; the file producer receives the request data packet arriving from the network, and searches for the corresponding file data from the local request monitoring pool according to the file content identification information in the request data packet. If a match is found, the corresponding data is packaged into a response data packet, and the destination address of the response data packet is specified according to the file consumer identification information in the request data packet. The response data packet is then transmitted to the corresponding consumer through the underlying network protocol; the response data packet also contains the producer's related verification information, network status, and the transmission service quality promised by the producer; The consumer includes three modules, namely, a buffer module, a sending window module and an event monitoring module, which transfer files between the application and the network; wherein the buffer module is used to cache the data packets that arrive out of order, restore the complete file after sorting, and finally deliver it to the user; the sending window module is responsible for controlling the rate of the entire sending process and the transmission task quota allocated to each producer, and sending the request with the producer's identification to the event monitoring module; the event monitoring module is responsible for processing the data arrival and response timeout events of each request, sending requests and receiving data to the network, caching the received file fragments into the buffer module, and adjusting the behavior of the sending window module through the network status information extracted during the monitoring process; the application submits the broadcast file transfer request to the event monitoring module to explore the producer in the network that is suitable for providing transmission services; The requests sent by the file consumer are bound to the two events of "data packet arrival" and "response timeout"; the processing process of the event monitoring module is as follows: when data arrives at the file consumer or the monitored request packet response times out, different processing logics are triggered; if the event of data packet arrival is monitored, the correctness of the data is first verified, and if correct, the data is written into the buffer, the network status information of the producer providing the data is updated, and the value of the maximum sending window module is adjusted according to the delay-bandwidth product; if the event of request response timeout is monitored, the request is re-added to the request queue, and it is considered that the request rate to the producer is too high, and the maximum sending window value to the producer should be reduced accordingly.

2. The decentralized file transfer method according to claim 1, characterized in that: The file consumer records the request items being monitored by a file producer and the status of the network link to a producer, and manages network resources.

3. The decentralized file transfer method according to claim 1, characterized in that: The file consumer can also act as a producer, cache the received data, and then monitor requests for the data; after the consumer verifies the legal data, the data and its corresponding structured naming identifier are placed in the consumer's request monitoring pool, so that the consumer can respond to the corresponding data as a producer after receiving the same request.

Images