A method and system for improving transmission efficiency of network distributed file system
By establishing a shared space in the server kernel space and optimizing network card queue selection, the problem of slow transmission speed of network distributed file systems is solved, and more efficient data transmission is achieved.
Patent Information
- Application Number
- CN202411380067.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-30
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-09-30
AI Technical Summary
The existing network distributed file system has a slow transmission speed when responding to client data requests, and the overall transmission speed needs to be improved.
Establish an independent shared space in the server's kernel space, optimize data transmission through network card queues, determine network card queues based on data size and historical transmission logs, optimize data storage and transmission paths, simplify server data copying process, and reduce data transfer in user space .
It significantly reduces the time-consuming data transmission to the client, improves the overall transmission speed, simplifies the data copying process, and improves the transmission efficiency of the network distributed file system.
Smart Images

Figure CN118890352B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of network communication technology, and in particular to a method and system for improving the transmission efficiency of a network distributed file system. Background Art
[0002] A network distributed file system (DFS) is a file system that manages physical storage resources that are not directly connected to a local server, but are connected to a local server through a computer network. It can be seen as a complete, hierarchical file system that combines multiple different logical hard disk partitions or volumes. A network distributed file system provides a logical tree-like file system structure for any resource distributed on the network, making it easier for users to access shared files on the network.
[0003] The network distributed file system expands a file system fixed at a certain location to any number of locations or file systems, so that many network nodes together form an overall network file system; network file transmission efficiency is an indicator that affects the user experience of the network distributed file system. The data access process of the existing network distributed file system is that the server reads the file on the hard disk and then sends it to the client through the network protocol. The existing network distributed file system takes a long time to respond to the client's data request and transmit the data to the client, and the overall transmission speed needs to be improved. Summary of the invention
[0004] The main purpose of the present invention is to provide a method and system for improving the transmission efficiency of a network distributed file system, aiming to solve the problem that the existing network distributed file system takes a long time to transmit data to a client in response to a data request from the client, and the overall transmission speed needs to be improved.
[0005] The technical solution proposed by the present invention is:
[0006] A method for improving the transmission efficiency of a network distributed file system is applied to a system for improving the transmission efficiency of a network distributed file system; a server comprises a processing module, a memory unit, a storage module and a network card module; the memory unit, the storage module and the network card module are all communicatively connected to the processing module; a client is communicatively connected to the server via the network card module; the memory unit comprises a kernel space; the network card module comprises a plurality of network card queues; the method comprises:
[0007] The processing module establishes an independent shared space in the kernel space;
[0008] When the processing module obtains a data read request instruction from the client through the network card module, the data corresponding to the data read request instruction is marked as the first target data, and the client sending the data read request instruction is marked as the target end;
[0009] The processing module writes the first target data in the storage module into the shared space;
[0010] The processing module sends the first target data in the shared space to the cache space of the network card module;
[0011] The processing module determines the network card queue corresponding to the first target data based on the size of the first target data;
[0012] The network card queue in the network card module sends the corresponding first target data to the target end.
[0013] Preferably, the processing module determines the network card queue corresponding to the first target data based on the size of the first target data, including:
[0014] The processing module marks at least one network card queue as a first network card queue, and marks other network card queues except the first network card queue as second network card queues;
[0015] The processing module determines whether the size of the first target data is greater than a preset value;
[0016] If so, the processing module uses the first network card queue as the network card queue corresponding to the first target data;
[0017] If not, the processing module uses the second network card queue as the network card queue corresponding to the first target data.
[0018] Preferably, the storage module stores a historical transmission log, which includes the size of data sent from the server to the client each time within a preset time period in the past; the processing module marks at least one network card queue as a first network card queue, and marks other network card queues except the first network card queue as second network card queues, including:
[0019] The processing module obtains the total number of times the server sends data to the client in the past preset time period based on the historical transmission log, and the number of times the size of data sent by the server to the client in the past preset time period is greater than a preset value, and marks them as target times;
[0020] The processing module obtains the target number based on the target number, the total number, and the total number of network card queues:
[0021] ,
[0022] In the formula, is the target number, is the total number of times; is the target number of times; is the total number of network card queues;
[0023] The processing module marks the target number of network card queues as first network card queues, and marks other network card queues except the first network card queue as second network card queues.
[0024] Preferably, the processing module determines the network card queue corresponding to the first target data based on the size of the first target data, and then further includes:
[0025] The processing module determines whether there is an idle second network card queue in the network card module;
[0026] If so, the processing module determines whether the network card queue corresponding to the first target data is the first network card queue;
[0027] If so, the processing module changes the network card queue corresponding to the first target data to an idle second network card queue, and sends the first target data to the target end through the idle second network card queue;
[0028] If not present, the step of executing the network card queue in the network card module to send the corresponding first target data to the target end.
[0029] Preferably, the storage module includes a plurality of storage units that are communicatively connected to the processing module; the method further includes:
[0030] When the processing module obtains the data storage request instruction from the client through the network card module, the data corresponding to the data storage request instruction is marked as the second target data;
[0031] The processing module determines whether there is a storage unit with a remaining space greater than the size of the second target data;
[0032] If so, the processing module marks the storage unit with a remaining space larger than the size of the second target data as the first target unit;
[0033] The processing module stores the second target data as a whole into the first target unit.
[0034] Preferably, the processing module stores the second target data as a whole into the first target unit, including:
[0035] The processing module obtains the number of files stored in each first target unit and the amount of space occupied by each first target unit;
[0036] The processing module calculates the average file size corresponding to each first target unit based on the number of files stored in each first target unit and the amount of space occupied by each first target unit:
[0037] ,
[0038] In the formula, is the average file size corresponding to the i-th first target unit, 1≤i≤N, N is the total number of first target units; is the amount of occupied space corresponding to the i-th first target unit; is the number of files stored in the i-th first target unit;
[0039] The processing module obtains the number of files of the second target data and the size of the second target data, and determines a first target unit for storing the second target data based on the number of files of the second target data, the size of the second target data, and the average file size corresponding to each first target unit, and marks the first target unit as the second target unit;
[0040] The processing module stores the second target data into the corresponding second target unit.
[0041] Preferably, the processing module obtains the number of files of the second target data and the size of the second target data, and determines the first target unit for storing the second target data based on the number of files of the second target data, the size of the second target data, and the average file size corresponding to each first target unit, and marks it as the second target unit, including:
[0042] The processing module calculates the current overall average file size based on the average file size corresponding to each first target unit:
[0043] ,
[0044] In the formula, is the current overall average file size;
[0045] The processing module calculates the data storage impact value corresponding to each first target unit when the second target data is stored in each first target unit respectively based on the number of files of the second target data, the size of the second target data, the number of files stored in each first target unit, the amount of space occupied by each first target unit, and the current overall average file size:
[0046] ,
[0047] In the formula, When the second target data is stored in the i-th first target unit, the data corresponding to the i-th first target unit is stored in the impact value; The number of files for the second target data, is the size of the second target data;
[0048] The processing module determines the first target unit with the smallest impact value where the data is stored as the first target unit for storing the second target data, and marks it as the second target unit.
[0049] Preferably, it also includes:
[0050] The processing module obtains the storage duration and the number of reads of the files stored in each first target unit to calculate the read frequency corresponding to the files stored in each first target unit:
[0051] ,
[0052] In the formula, is the reading frequency corresponding to the jth stored file in the i-th first target unit; is the storage duration of the jth stored file in the i-th first target unit; is the number of times the jth stored file in the i-th first target unit is read;
[0053] The processing module marks the stored files whose reading frequency is greater than the preset frequency as hot files;
[0054] The processing module marks a storage unit whose remaining space is larger than the size of the second target data as a first target unit, and then further includes:
[0055] The processing module obtains the number of hot files in each first target unit, and marks the first target unit with the least number of hot files as the third target unit;
[0056] The processing module stores the second target data as a whole into the third target unit.
[0057] Preferably, it also includes:
[0058] The processing module establishes a hot file prediction model and obtains the size, type, IP address of the sender corresponding to each stored file in the storage module, and whether the stored file is a hot file;
[0059] The processing module uses the size and type of each stored file and the IP address of the sender as input parameters of the hot file prediction model, and uses whether each stored file is a hot file as an output parameter of the hot file prediction model, thereby training the hot file prediction model;
[0060] The processing module obtains the number of hot files in each first target unit, and marks the first target unit with the least number of hot files as the third target unit, and then further includes:
[0061] The processing module estimates the second target data based on the trained hot file estimation model to obtain the estimated hot file in the second target data;
[0062] The processing module stores the estimated hot spot file in the second target data into the third target unit, and marks the first target unit with the largest remaining space among the other first target units except the third target unit as the fourth target unit;
[0063] The processing module stores other files in the second target data except the estimated hot spot files into the fourth target unit.
[0064] The present invention also proposes a system for improving the transmission efficiency of a network distributed file system, and applies a method for improving the transmission efficiency of a network distributed file system; the system includes a server and a client; the server includes a processing module, a memory unit, a storage module and a network card module; the memory unit, the storage module and the network card module are all communicatively connected to the processing module; the client is communicatively connected to the server via the network card module; the memory unit includes a kernel space; and the network card module includes multiple network card queues.
[0065] Through the above technical solution, the following beneficial effects can be achieved:
[0066] The method for improving the transmission efficiency of a network distributed file system proposed in the present invention can improve the data transmission speed; the first target data read by the server is no longer transferred through the user space, but an independent shared space is divided in the kernel space, the first target data read in the storage module is transferred to the shared space, and then the first target data in the shared space is sent to the cache space of the network card module, and then the network card queue corresponding to the first target data is determined based on the size of the first target data; in this solution, a simplified design is performed on the server data copy process of the network distributed file system. Compared with the data transmission process of the traditional network distributed file system, this solution can greatly reduce the time consumption of transmitting data to the client, thereby improving the overall transmission speed. BRIEF DESCRIPTION OF THE DRAWINGS
[0067] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.
[0068] Figure 1 This is a flowchart of the steps of a first embodiment of a method for improving the transmission efficiency of a network distributed file system proposed by the present invention. DETAILED DESCRIPTION
[0069] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.
[0070] The invention provides a method for improving the transmission efficiency of a network distributed file system.
[0071] As attached Figure 1 As shown, in a first embodiment of a method for improving the transmission efficiency of a network distributed file system proposed by the present invention, the method is applied to a system for improving the transmission efficiency of a network distributed file system; the system includes a server and a client; the server includes a processing module, a memory unit, a storage module (hard disk group) and a network card module; the memory unit, the storage module and the network card module are all communicatively connected to the processing module; the client is communicatively connected to the server through the network card module; the memory unit includes a kernel space (the server runs a Linux operating system, and the Linux operating system divides the memory unit into a user space and a kernel space); the network card module includes a plurality of network card queues; this embodiment includes the following steps:
[0072] Step S110: The processing module establishes an independent shared space in the kernel space.
[0073] Specifically, the existing network distributed file system data access is to read the file on the hard disk and then send it to the client through the network protocol. File reading uses I / O access, and the working method of I / O is that data reading and writing is copied back and forth from user space to kernel space, and the data in kernel space is read and written from the hard disk through the API interface provided by the I / O library of the operating system.
[0074] Specifically, during the process of client-server file access, the server will copy data four times: hard disk → kernel cache → user space cache → kernel socket cache → network card cache. During the above data copy process, no data processing will be performed on the copied data, only the data will be simply transferred and stored. Repeated data copying consumes a lot of processing module resources and seriously affects the speed of network file transmission.
[0075] Step S120: When the processing module obtains a data read request instruction from the client through the network card module, the data corresponding to the data read request instruction is marked as the first target data, and the client sending the data read request instruction is marked as the target end.
[0076] Step S130: the processing module writes the first target data in the storage module into the shared space.
[0077] Step S140: the processing module sends the first target data in the shared space to the cache space of the network card module.
[0078] Step S150: the processing module determines the network card queue corresponding to the first target data based on the size of the first target data.
[0079] Specifically, first target data of different sizes correspond to different network card queues, thereby avoiding data congestion caused by first target data of different sizes being transmitted from the same network card queue.
[0080] Step S160: the network card queue in the network card module sends the corresponding first target data to the target end.
[0081] The method for improving the transmission efficiency of a network distributed file system proposed in the present invention can improve the data transmission speed; the first target data read by the server is no longer transferred through the user space, but an independent shared space is divided in the kernel space, the first target data read in the storage module is transferred to the shared space, and then the first target data in the shared space is sent to the cache space of the network card module, and then the network card queue corresponding to the first target data is determined based on the size of the first target data; in this solution, a simplified design is performed on the server data copy process of the network distributed file system. Compared with the data transmission process of the traditional network distributed file system, this solution can greatly reduce the time consumption of transmitting data to the client, thereby improving the overall transmission speed.
[0082] In a second embodiment of a method for improving the transmission efficiency of a network distributed file system proposed by the present invention, based on the first embodiment, step S150 includes the following steps:
[0083] Step S210: the processing module marks at least one network card queue as a first network card queue, and marks other network card queues except the first network card queue as second network card queues.
[0084] Step S220: The processing module determines whether the size of the first target data is greater than a preset value (eg, 1 GB).
[0085] If so, executing step S230: the processing module uses the first network card queue as the network card queue corresponding to the first target data.
[0086] If not, executing step S240: the processing module uses the second network card queue as the network card queue corresponding to the first target data.
[0087] Specifically, this embodiment provides a specific solution for determining the network card queue corresponding to the first target data based on the size of the first target data; if the first target data is larger than the preset value, it means that the first target data requires a larger space, and the larger first target data is transmitted over the network through the first network card queue, and the remaining smaller first target data is transmitted over the network through the second network card queue, so that differentiated transmission can be performed and the waiting time for the smaller first target data to be transmitted through the network card queue can be reduced; that is, this solution can optimize the network card queue, and large files and small files are transmitted in different queues respectively to avoid congestion, thereby improving the overall data transmission efficiency.
[0088] In a third embodiment of a method for improving the transmission efficiency of a network distributed file system proposed by the present invention, based on the second embodiment, a storage module stores a historical transmission log, and the historical transmission log includes the size of data sent from the server to the client each time within a preset time period (for example, the past month); step S210 includes the following steps:
[0089] Step S310: The processing module obtains the total number of times the server sent data to the client in the past preset time period based on the historical transmission log, and the number of times the size of data sent by the server to the client in the past preset time period was greater than a preset value, and marks them as target times.
[0090] Step S320: The processing module obtains the target number based on the target number, the total number and the total number of network card queues:
[0091] ,
[0092] In the formula, is the target number, is the total number of times; is the target number of times; The total number of network card queues.
[0093] Step S330: the processing module marks the target number of network card queues as first network card queues, and marks the other network card queues except the first network card queue as second network card queues.
[0094] Specifically, this embodiment provides a calculation solution for determining the number of the first network card queue and the second network card queue.
[0095] In a fourth embodiment of a method for improving the transmission efficiency of a network distributed file system proposed by the present invention, based on the third embodiment, step S140 further includes the following steps:
[0096] Step S410: The processing module determines whether there is an idle second network card queue in the network card module.
[0097] Step S420: If it exists, the processing module determines whether the network card queue corresponding to the first target data is the first network card queue.
[0098] If so, execute step S430: the processing module changes the network card queue corresponding to the first target data to an idle second network card queue, and sends the first target data to the target end through the idle second network card queue.
[0099] Specifically, if there is an idle second network card queue (for sending smaller data) and the first target data is larger data, the first target data can be transmitted over the network through the second network card queue to fully utilize the network card queue and further ensure data transmission efficiency.
[0100] Step S440: If not present, execute step S160.
[0101] In a fifth embodiment of a method for improving the transmission efficiency of a network distributed file system proposed by the present invention, based on the first embodiment, the storage module includes a plurality of storage units that are communicatively connected to the processing module, for example, the hard disk group (storage module) includes a plurality of hard disks (storage units); this embodiment further includes the following steps:
[0102] Step S510: when the processing module obtains the data storage request instruction from the client through the network card module, the data corresponding to the data storage request instruction is marked as the second target data.
[0103] Step S520: The processing module determines whether there is a storage unit with a remaining space larger than the size of the second target data.
[0104] If yes, execute step S530: the processing module marks the storage unit whose remaining space is larger than the size of the second target data as the first target unit.
[0105] Step S540: the processing module stores the second target data as a whole into the first target unit.
[0106] Specifically, the purpose of this embodiment is to store the second target data that needs to be stored as a whole into one storage unit (i.e., the first target unit), that is, the second target data is not sliced and stored separately in different storage units. This can reduce the process of jumping between different storage units when subsequently reading the second target data (jumping when reading data will increase the reading time of the data), thereby further improving data transmission efficiency.
[0107] In a sixth embodiment of a method for improving the transmission efficiency of a network distributed file system proposed by the present invention, based on the fifth embodiment, step S540 includes the following steps:
[0108] Step S610: The processing module obtains the number of files stored in each first target unit and the amount of space occupied by each first target unit.
[0109] Step S620: The processing module calculates the average file size corresponding to each first target unit based on the number of files stored in each first target unit and the amount of space occupied by each first target unit:
[0110] ,
[0111] In the formula, is the average file size corresponding to the i-th first target unit, 1≤i≤N, N is the total number of first target units; is the amount of occupied space corresponding to the i-th first target unit; is the number of files stored in the i-th first target unit.
[0112] Specifically, the average file size here can reflect the average size of each file in the first target unit.
[0113] Step S630: The processing module obtains the number of files of the second target data and the size of the second target data, and determines the first target unit for storing the second target data based on the number of files of the second target data, the size of the second target data, and the average file size corresponding to each first target unit, and marks it as the second target unit.
[0114] Specifically, the solution of this embodiment is to select which first target unit to store the second target data in based on the average size of the stored files in each first target unit; the average size of the files stored in each first target unit should be consistent, so that under the premise of random reading of data, the average time spent reading the files of each first target unit will also be consistent, so that the usage time of each first target unit will be consistent, and the service life will also be consistent, which is more conducive to the maintenance of the storage module hardware.
[0115] Step S640: The processing module stores the second target data into the corresponding second target unit.
[0116] In a seventh embodiment of a method for improving the transmission efficiency of a network distributed file system proposed by the present invention, based on the sixth embodiment, step S630 includes the following steps:
[0117] Step S710: The processing module calculates the current overall average file size based on the average file size corresponding to each first target unit:
[0118] ,
[0119] In the formula, is the current overall average file size.
[0120] Step S720: The processing module calculates the data storage impact value corresponding to each first target unit when the second target data is stored in each first target unit respectively based on the number of files of the second target data, the size of the second target data, the number of files stored in each first target unit, the amount of space occupied by each first target unit, and the current overall average file size:
[0121] ,
[0122] In the formula, When the second target data is stored in the i-th first target unit, the data corresponding to the i-th first target unit is stored in the impact value; The number of files for the second target data, is the size of the second target data.
[0123] Specifically, from the above formula, we can know that is the difference between the average file size of the i-th first target unit and the overall average file size when the second target data is stored in the i-th first target unit, The smaller it is, the closer the average file size of the i-th first target unit is to the overall file size after the second target data is stored, that is, the more it conforms to the aforementioned goal that the average size of the files stored in each first target unit should tend to be consistent, so the second target data is more suitable to be stored in the i-th first target unit.
[0124] Step S730: the processing module determines the first target unit with the smallest impact value into which the data is stored as the first target unit for storing the second target data, and marks it as the second target unit.
[0125] Specifically, the first target unit in which the data is stored and has the smallest impact value is determined as the first target unit for storing the second target data and is marked as the second target unit.
[0126] In an eighth embodiment of a method for improving the transmission efficiency of a network distributed file system proposed by the present invention, based on the fifth embodiment, this embodiment further includes the following steps:
[0127] Step S810: The processing module obtains the storage duration and the number of reads of the files stored in each first target unit, so as to calculate the read frequency corresponding to the files stored in each first target unit:
[0128] ,
[0129] In the formula, is the reading frequency corresponding to the jth stored file in the i-th first target unit; is the storage duration of the jth stored file in the i-th first target unit; is the number of times the jth stored file in the i-th first target unit is read.
[0130] Step S820: The processing module marks the stored files whose reading frequency is greater than a preset frequency (eg, 10 times per day) as hot files.
[0131] Step S530 further includes the following steps:
[0132] Step S830: The processing module obtains the number of hot files in each first target unit, and marks the first target unit with the least number of hot files as the third target unit.
[0133] Specifically, in each storage unit, hot files are files that are read more frequently. Therefore, in order to balance the number of times each storage unit is read, the number of hot files in each storage unit needs to be consistent. Therefore, in this embodiment, the first target unit with the least number of hot files is marked as the third target unit.
[0134] Step S840: the processing module stores the second target data as a whole into the third target unit.
[0135] Specifically, directly storing the second target data in the third target unit can make the number of hot files in each storage unit consistent, thereby balancing the number of times each storage unit is read, and preventing the service life of one or more storage units from being consumed prematurely.
[0136] In a ninth embodiment of a method for improving the transmission efficiency of a network distributed file system proposed by the present invention, based on the eighth embodiment, this embodiment further includes the following steps:
[0137] Step S910: The processing module establishes a hot file prediction model and obtains the size, type (such as text, audio, video and program files, etc.), IP address of the sender (the sender here is the client that sends data to the server) of each stored file in the storage module, and whether the stored file is a hot file.
[0138] Specifically, whether a file stored in the storage module is a hot file is related to the size and type of the file itself, and the IP address of the sender (different IP addresses correspond to different senders, and the probability of files uploaded by different senders becoming hot files is not the same; for example, for uploaded video files, if a video file uploaded by a certain sender is more popular with online viewers, then the probability of the video file uploaded by this sender becoming a hot file will be higher than that of video files uploaded by other senders).
[0139] Step S920: The processing module uses the size, type, and IP address of each stored file as input parameters of the hot file prediction model, and uses whether each stored file is a hot file as an output parameter of the hot file prediction model, thereby training the hot file prediction model.
[0140] Specifically, the size, type, and IP address of the sender corresponding to each stored file are used as input parameters of the hot file prediction model, and whether each stored file is a hot file is used as the output parameter of the hot file prediction model to train the hot file prediction model. The trained hot file prediction model can predict the data just received by the server, thereby determining whether the data just received contains the estimated hot file.
[0141] Step S830, and then further includes the following steps:
[0142] Step S930: The processing module estimates the second target data based on the trained hot file estimation model to obtain the estimated hot file in the second target data.
[0143] Step S940: The processing module stores the estimated hot spot file in the second target data to the third target unit, and marks the first target unit with the largest remaining space among the other first target units except the third target unit as the fourth target unit.
[0144] Step S960: The processing module stores other files in the second target data except the estimated hot spot files into the fourth target unit.
[0145] Specifically, the hot file prediction model can be used to estimate the data just received by the server, thereby determining whether the data just received contains an estimated hot file (the estimated hot file is likely to become a real feature file in the future), and directly store the estimated hot file in the third target unit in advance to balance the number of hot files in each storage unit.
[0146] In a tenth embodiment of a method for improving the transmission efficiency of a network distributed file system proposed by the present invention, based on the ninth embodiment, step S930 includes the following steps:
[0147] Step S1010: The processing module obtains the size, type, and IP address of the sender corresponding to each file in the second target data.
[0148] Step S1020: The processing module inputs the size, type, and IP address of each file in the second target data into the trained hot file prediction model to obtain an output result of whether each file in the second target data is a hot file.
[0149] Specifically, after the size, type, and IP address of each file in the second target data are input into the trained hot file prediction model, the output result of the hot file prediction model can be obtained (i.e., whether the corresponding file is a hot file).
[0150] Step S1030: The processing module uses the files in the second target data that are output as hot files as estimated hot files.
[0151] Specifically, this embodiment provides how to estimate each file in the second target data based on the hot file estimation model to determine whether there is a hot file in the second target data, and if so, obtain the hot file in the second target data.
[0152] The present invention also proposes a system for improving the transmission efficiency of a network distributed file system, and applies a method for improving the transmission efficiency of a network distributed file system; the system includes a server and a client; the server includes a processing module, a memory unit, a storage module and a network card module; the memory unit, the storage module and the network card module are all communicatively connected to the processing module; the client is communicatively connected to the server via the network card module; the memory unit includes a kernel space; and the network card module includes multiple network card queues.
[0153] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.
[0154] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.
Claims
1. A method for improving the transmission efficiency of a network distributed file system. It is characterized in that A system for improving the transmission efficiency of a network distributed file system; the system comprises a server and a client; the server comprises a processing module, a memory unit, a storage module and a network card module; the memory unit, the storage module and the network card module are all communicatively connected to the processing module; The client is connected to the server via the network card module; The memory unit includes a kernel space; the network card module includes a plurality of network card queues; the storage module includes a plurality of storage units that are communicatively connected to the processing module; the method includes: The processing module establishes an independent shared space in the kernel space; When the processing module obtains a data read request instruction from the client through the network card module, the data corresponding to the data read request instruction is marked as the first target data, and the client sending the data read request instruction is marked as the target end; The processing module writes the first target data in the storage module into the shared space; The processing module sends the first target data in the shared space to the cache space of the network card module; The processing module determines the network card queue corresponding to the first target data based on the size of the first target data; The network card queue in the network card module sends the corresponding first target data to the target end; When the processing module obtains the data storage request instruction from the client through the network card module, the data corresponding to the data storage request instruction is marked as the second target data; The processing module determines whether there is a storage unit with a remaining space greater than the size of the second target data; If so, the processing module marks the storage unit with a remaining space larger than the size of the second target data as the first target unit; The processing module stores the second target data as a whole into the first target unit.
2. A method for improving the transmission efficiency of a network distributed file system according to claim 1, It is characterized in that The processing module determines the network card queue corresponding to the first target data based on the size of the first target data, including: The processing module marks at least one network card queue as a first network card queue, and marks other network card queues except the first network card queue as second network card queues; The processing module determines whether the size of the first target data is greater than a preset value; If so, the processing module uses the first network card queue as the network card queue corresponding to the first target data; If not, the processing module uses the second network card queue as the network card queue corresponding to the first target data.
3. A method for improving the transmission efficiency of a network distributed file system according to claim 2, It is characterized in that The storage module stores historical transmission logs, which include the size of data sent from the server to the client each time within a preset time period in the past; The processing module marks at least one network card queue as a first network card queue, and marks other network card queues except the first network card queue as second network card queues, including: The processing module obtains the total number of times the server sends data to the client in the past preset time period based on the historical transmission log, and the number of times the size of data sent by the server to the client in the past preset time period is greater than a preset value, and marks them as target times; The processing module obtains the target number based on the target number, the total number, and the total number of network card queues: , In the formula, is the target number, is the total number of times; is the target number of times; is the total number of network card queues; The processing module marks the target number of network card queues as first network card queues, and marks other network card queues except the first network card queue as second network card queues.
4. A method for improving the transmission efficiency of a network distributed file system according to claim 3, It is characterized in that The processing module determines the network card queue corresponding to the first target data based on the size of the first target data, and then further includes: The processing module determines whether there is an idle second network card queue in the network card module; If so, the processing module determines whether the network card queue corresponding to the first target data is the first network card queue; If so, the processing module changes the network card queue corresponding to the first target data to an idle second network card queue, and sends the first target data to the target end through the idle second network card queue; If not present, the step of executing the network card queue in the network card module to send the corresponding first target data to the target end.
5. A method for improving the transmission efficiency of a network distributed file system according to claim 1, It is characterized in that The processing module stores the second target data as a whole into the first target unit, including: The processing module obtains the number of files stored in each first target unit and the amount of space occupied by each first target unit; The processing module calculates the average file size corresponding to each first target unit based on the number of files stored in each first target unit and the amount of space occupied by each first target unit: , In the formula, is the average file size corresponding to the i-th first target unit, 1≤i≤N, N is the total number of first target units; is the amount of occupied space corresponding to the i-th first target unit; is the number of files stored in the i-th first target unit; The processing module obtains the number of files of the second target data and the size of the second target data, and determines a first target unit for storing the second target data based on the number of files of the second target data, the size of the second target data, and the average file size corresponding to each first target unit, and marks the first target unit as the second target unit; The processing module stores the second target data into the corresponding second target unit.
6. A method for improving the transmission efficiency of a network distributed file system according to claim 5, It is characterized in that The processing module obtains the number of files of the second target data and the size of the second target data, and determines a first target unit for storing the second target data based on the number of files of the second target data, the size of the second target data, and the average file size corresponding to each first target unit, and marks the first target unit as the second target unit, including: The processing module calculates the current overall average file size based on the average file size corresponding to each first target unit: , In the formula, is the current overall average file size; The processing module calculates the data storage impact value corresponding to each first target unit when the second target data is stored in each first target unit respectively based on the number of files of the second target data, the size of the second target data, the number of files stored in each first target unit, the amount of space occupied by each first target unit, and the current overall average file size: , In the formula, When the second target data is stored in the i-th first target unit, the data corresponding to the i-th first target unit is stored in the impact value; The number of files for the second target data, is the size of the second target data; The processing module determines the first target unit with the smallest impact value where the data is stored as the first target unit for storing the second target data, and marks it as the second target unit.
7. A method for improving the transmission efficiency of a network distributed file system according to claim 1, It is characterized in that Also includes: The processing module obtains the storage duration and the number of reads of the files stored in each first target unit to calculate the read frequency corresponding to the files stored in each first target unit: , In the formula, is the reading frequency corresponding to the jth stored file in the i-th first target unit; is the storage duration of the jth stored file in the i-th first target unit; is the number of times the jth stored file in the i-th first target unit is read; The processing module marks the stored files whose reading frequency is greater than the preset frequency as hot files; The processing module marks a storage unit whose remaining space is larger than the size of the second target data as a first target unit, and then further includes: The processing module obtains the number of hot files in each first target unit, and marks the first target unit with the least number of hot files as the third target unit; The processing module stores the second target data as a whole into the third target unit.
8. A method for improving the transmission efficiency of a network distributed file system according to claim 7, It is characterized in that Also includes: The processing module establishes a hot file prediction model and obtains the size, type, IP address of the sender corresponding to each stored file in the storage module, and whether the stored file is a hot file; The processing module uses the size and type of each stored file and the IP address of the sender as input parameters of the hot file prediction model, and uses whether each stored file is a hot file as an output parameter of the hot file prediction model, thereby training the hot file prediction model; The processing module obtains the number of hot files in each first target unit, and marks the first target unit with the least number of hot files as the third target unit, and then further includes: The processing module estimates the second target data based on the trained hot file estimation model to obtain the estimated hot file in the second target data; The processing module stores the estimated hot spot file in the second target data into the third target unit, and marks the first target unit with the largest remaining space among the other first target units except the third target unit as the fourth target unit; The processing module stores other files in the second target data except the estimated hot spot files into the fourth target unit.
9. A system for improving the transmission efficiency of a network distributed file system. It is characterized in that The method for improving the transmission efficiency of a network distributed file system according to any one of claims 1 to 8 is applied; the system comprises a server and a client; the server comprises a processing module, a memory unit, a storage module and a network card module; the memory unit, the storage module and the network card module are all communicatively connected to the processing module; The client is connected to the server via a network card module; the memory unit includes a kernel space; and the network card module includes a plurality of network card queues.
Citation Information
Patent Citations
Storage platform service method and device, equipment and storage medium
CN116910015A
Network file transmission method and system for improving throughput of sending end equipment
CN117376343A
