File transmission method and device, equipment, storage medium and program product

Abstract

The invention provides a file transmission method and device, equipment, a storage medium and a program product, and relates to the technical field of file transmission, and the method comprises the steps: generating a to-be-packaged file directory of a first time period according to a preset packaging rule and a file packaging block scale of the first time period, the to-be-packaged file directory is used for indicating path information of each to-be-packaged file; executing a packaging operation according to the to-be-packaged file directory to obtain a file packaging block of a first time period; transmitting the file packaging block to a receiving end, and acquiring a transmission parameter of the first time period and a transmission result of the first time period fed back by the receiving end in the transmission process of the file packaging block; according to the transmission parameters, the transmission result and the bandwidth prediction result of the second time period, the file packaging block scale of the second time period is obtained, and the second time period is the next time period of the first time period. According to the scheme, the file transmission efficiency can be improved, and resource consumption is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of file transmission, in particular to a file transmission method, device, equipment, storage medium and program product. BACKGROUND

[0002] Data flow presents an explosive growth trend, and the demand for data migration is increasingly strong. Massive small file transmission in scenarios such as Internet of Things detection, financial transactions, and social media is becoming more and more frequent. However, as the demand for massive small file transmission continues to rise, massive small files will face many problems in the transmission process: frequent connection establishment leads to large network resource overhead and easy network congestion; repeated file reading causes increased storage device wear and tear and increased read delay, resulting in low overall transmission efficiency and high resource consumption. SUMMARY

[0003] At least one embodiment of the present application provides a file transmission method, device, equipment, storage medium and program product, which are used to solve the problems of low small file transmission efficiency and large resource consumption in the prior art.

[0004] To solve the above technical problems, the present application is implemented as follows:

[0005] In a first aspect, the embodiments of the present application provide a file transmission method applied to a sending end, and the method comprises:

[0006] According to a preset packing rule and a file packing block size of a first time period, a directory of files to be packed in the first time period is generated, and the directory of files to be packed is used to indicate path information of each file to be packed;

[0007] According to the directory of files to be packed, a packing operation is performed to obtain a file packing block in the first time period;

[0008] The file packing block is transmitted to a receiving end, and transmission parameters of the first time period are obtained during the transmission of the file packing block, and transmission results of the first time period fed back by the receiving end are obtained;

[0009] According to the transmission parameters, the transmission results, and a bandwidth prediction result of a second time period, a file packing block size of the second time period is obtained, and the second time period is a next time period of the first time period.

[0010] Optionally, the file transmission method, wherein the preset packing rule is related to a file packing priority and / or file storage information.

[0011] The file packaging priority is related to at least one of a file size corresponding to the file, a device load state, and a transmission tolerable time, wherein the smaller the file size is, the higher the corresponding file packaging priority is; the higher the device load state is, the higher the corresponding file packaging priority is; and the shorter the transmission tolerable time is, the higher the corresponding file packaging priority is.

[0012] The file storage information is used to indicate whether the at least two files are on the same storage device.

[0013] Optionally, the file transmission method, wherein the packaging operation is performed according to the to-be-packaged file directory, and the file packaging block of the first time period is obtained, including:

[0014] The file metadata and the file data of each file to be packaged are obtained from path information of each file to be packaged indicated by the to-be-packaged file directory.

[0015] The first password hash value is obtained according to the file metadata and the file data.

[0016] The file metadata, the file data, and the first password hash value are spliced to obtain the file packaging block of the first time period.

[0017] Optionally, the file transmission method, wherein the file packaging block size of the second time period is obtained according to the transmission parameter, the transmission result, and the bandwidth prediction result of the second time period, including:

[0018] The reward parameter of the first time period of the packaging block size decision model is obtained according to the transmission parameter and the transmission result.

[0019] The network state evaluation index is obtained according to link historical network data and the bandwidth prediction result of the second time period.

[0020] The adjustment coefficient of the file packaging block size of the second time period is obtained according to the network state evaluation index, the reward parameter, and the packaging block size decision model.

[0021] The file packaging block size of the second time period is obtained according to the adjustment coefficient, the bandwidth prediction result, and the round-trip time of the first time period.

[0022] Optionally, the file transmission method, wherein the method further includes:

[0023] The traffic prediction data of the second time period is obtained according to link historical traffic data and a time series prediction model.

[0024] According to the flow prediction data and the upper limit of the link bandwidth, a bandwidth prediction result of the second time period is obtained.

[0025] Optionally, the file transmission method, wherein the file packing block size of the first time period is obtained according to a bandwidth prediction result of the first time period, a transmission parameter and a transmission result of a file packing block transmitted to the receiving end in a third time period, and the third time period is a last time period of the first time period.

[0026] In a second aspect, the embodiments of the present application further provide a file transmission method applied to a receiving end, and the method comprises:

[0027] receiving a file packing block of a first time period transmitted by a sending end;

[0028] feeding back a transmission result of the file packing block to the sending end.

[0029] Optionally, the file transmission method, wherein the method further comprises:

[0030] obtaining a second password hash value according to file metadata and file transmission data in the file packing block;

[0031] judging whether the first password hash value in the file packing block is consistent with the second password hash value;

[0032] if the first password hash value is consistent with the second password hash value, obtaining a transmission result.

[0033] In a third aspect, the embodiments of the present application further provide a file transmission device applied to a sending end, and the device comprises:

[0034] a generating module configured to generate a to-be-packed file directory of a first time period according to a preset packing rule and a file packing block size of the first time period, and the to-be-packed file directory is used to indicate path information of each file to be packed;

[0035] a packing module configured to perform a packing operation according to the to-be-packed file directory, and obtain a file packing block of the first time period;

[0036] a transmission module configured to transmit the file packing block to a receiving end, obtain a transmission parameter of the first time period in a process of transmitting the file packing block, and obtain a transmission result of the first time period fed back by the receiving end;

[0037] A decision module is configured to obtain a file packing block size of the second time period according to the transmission parameter, the transmission result, and a bandwidth prediction result of the second time period, the second time period being a next time period of the first time period.

[0038] In a fourth aspect, the embodiments of the present application further provide a file transmission device, which is applied to a receiving end, and the device comprises:

[0039] A receiving module is configured to receive a file packing block of a first time period transmitted by a sending end.

[0040] A feedback module is configured to feed back a transmission result of the file packing block to the sending end.

[0041] In a fifth aspect, the embodiments of the present application further provide a file transmission equipment, which comprises a processor, a memory, and a program or instruction stored in the memory and executable on the processor, and the processor executes the program or instruction to implement the file transmission method in the first aspect or the file transmission method in the second aspect.

[0042] In a sixth aspect, the embodiments of the present application further provide a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the file transmission method in the first aspect or the file transmission method in the second aspect.

[0043] In a seventh aspect, the embodiments of the present application further provide a computer program product, which comprises computer instructions, and the computer instructions are executed by a processor to implement the file transmission method in the first aspect or the file transmission method in the second aspect.

[0044] Compared with the prior art, the file transmission method, device, equipment, storage medium and program product provided by the embodiments of the present application, wherein the method is applied to a sending end, comprises: generating a file directory to be packaged in a first time period according to a preset packaging rule and a file packaging block size of the first time period, the file directory to be packaged being used to indicate path information of each file to be packaged; performing a packaging operation according to the file directory to be packaged, to obtain a file packaging block of the first time period; transmitting the file packaging block to a receiving end, and obtaining a transmission parameter of the first time period and a transmission result of the first time period fed back by the receiving end in the process of transmitting the file packaging block; obtaining a file packaging block size of a second time period according to the transmission parameter, the transmission result and a bandwidth prediction result of the second time period, the second time period being a next time period of the first time period, so that the file packaging block size is dynamically adjusted, a plurality of files are packaged and then transmitted, and the file transmission efficiency and the network resource utilization rate are improved. BRIEF DESCRIPTION OF DRAWINGS

[0045] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The accompanying drawings are included to provide a description of the preferred embodiments, and are not intended to limit the scope of the present application. Moreover, like reference numerals designate like parts throughout the several views in the drawings. In the drawings:

[0046] Figure 1 A flowchart of a file transmission method according to an embodiment of the present application;

[0047] Figure 2 A flowchart of an embodiment of a file transmission method according to the present application;

[0048] Figure 3 A flowchart of an embodiment of a file transmission method according to the present application;

[0049] Figure 4 A flowchart of an embodiment of a file transmission method according to the present application;

[0050] Figure 5 A flowchart of a file transmission method according to another embodiment of the present application;

[0051] Figure 6 A module diagram of a file transmission device according to an embodiment of the present application;

[0052] Figure 7 A module diagram of a file transmission device according to another embodiment of the present application;

[0053] Figure 8A hardware block diagram of a file transmission device according to an embodiment of the present application. DETAILED DESCRIPTION

[0054] The terms "first", "second", and the like in the present application are used to distinguish similar objects, and are not used to describe a particular order or sequence. It should be understood that the terms used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than that illustrated or described herein, and the objects distinguished by "first", "second" are generally of a kind and do not limit the number of objects, for example, the first object can be one or more. In addition, "or" in the present application means at least one of the connected objects. For example, "A or B" covers three scenarios, namely, scenario one: including A and not including B; scenario two: including B and not including A; scenario three: including A and B. The character " / " generally represents that the objects before and after are in an "or" relationship.

[0055] Please refer to Figure 1 The embodiments of the present application provide a file transmission method, applied to a sending end, which can be a gateway. The gateway can include a file storage device, or the gateway can be hung with a file storage device, and the gateway can perceive file information and device information stored by the file storage device, wherein the file information includes at least one of file metadata, file size, file data, tolerable transmission time and file storage information; and the device information includes device load state.

[0056] Further, the method includes:

[0057] Step 101, generating a file directory to be packaged in a first time period according to a preset packaging rule and a file packaging block size in the first time period, the file directory to be packaged being used to indicate path information of each file to be packaged;

[0058] The file packaging block size can be understood as a file packaging block size or a file packaging block dimension.

[0059] The first time period can be a current time period.

[0060] It should be noted that the size of each file to be packaged is several KB, several tens of KB or several hundred KB, so in the embodiments of the present application, each file to be packaged can be understood as a small file, and the file transmission method in the embodiments of the present application is a small file transmission method.

[0061] It should be further noted that the file directory to be packaged is not a physical directory in an actual file system, but a path information, for example, a path list, used to store each file to be packaged, so as to realize structured management and accurate indexing of the file to be packaged.

[0062] In the embodiments of the present application, according to the preset packing rule and the file packing block size of the first time period, a file directory of the first time period to be packed is generated, that is, a file directory of the file to be packed to different file packing blocks.

[0063] Optionally, the preset packing rule is related to a file packing priority and / or file storage information.

[0064] The file packing priority is related to at least one of a file size corresponding to the file, a device load state, and a transmission tolerable time, wherein the smaller the file size, the higher the corresponding file packing priority; the higher the device load state, the higher the corresponding file packing priority; and the shorter the transmission tolerable time, the higher the corresponding file packing priority.

[0065] The file storage information is used to indicate whether at least two files are on the same storage device.

[0066] The device load state is a device load state of the sending end. Since the file storage information is used to indicate whether at least two files are on the same storage device, the file storage information is used to indicate whether each two files to be packed are on the same storage device, and it can be understood that the file storage information is used to indicate file storage correlation.

[0067] In the embodiments of the present application, when the file directory to be packed is divided, the preset packing rule needs to be designed in advance, so as to realize fast and efficient division of file packing blocks. The file packing priority is designed based on at least one of the file size, the device load state, the transmission tolerable time, and the file storage information. In addition to considering the influence of at least one of the file size, the device load state, and the transmission tolerable time on the file itself, the file storage information also needs to be considered, so as to preferentially aggregate files stored on the same storage device, thereby reducing I / O switching overhead.

[0068] In combination with the above description, the preset packing rule is constructed, and specifically includes the following formulas (1) to (5).

[0069] Formula (1) is as follows, which is an optimization objective function of a file packing block to be packed:

[0070] (1);

[0071] Wherein, is an optimization objective, and are file and file file packing priority of a file, a binary variable storing information of a file, whether the file is in the same storage device as the file, the maximum size of a file packing block, the size of a file, in KB.

[0072] The file packing priority is defined as formula (2) considering the three dimensions of file size, device load status and transmission tolerable time, which affect the efficiency of file packing. :

[0073] (2).

[0074] wherein, is the file packing priority of a file f, is the file size factor of a file f, indicating the file size, is the device load status factor of a file f, indicating the device load status, is the transmission tolerable time factor of a file f, indicating the transmission tolerable time, is the weight factor, = 1.

[0075] The file size factor is defined as formula (3) to ensure that smaller files can obtain higher priority and improve the overall throughput of the system.

[0076] (3).

[0077] wherein, represents the size of a file f in KB, represents the largest file size among the files to be packed.

[0078] The device load status factor is defined as formula (4) to ensure that files stored on a high-load device can obtain higher priority when packing, thereby reducing the potential impact of high-load devices on other services due to continuous task processing, such as resource preemption or response delay, balancing the workloads of different devices, and improving the stability of overall operation and the rationality of resource allocation.

[0079] (4).

[0080] wherein, is the CPU utilization rate (in percentage) of the device where the file f is located, and ​​respectively the current I / O operation frequency and the maximum I / O operation frequency of the device where the file f is located. With is a weight coefficient, .

[0081] The transmission tolerable time T is calculated as follows: The transmission priority of a file is represented by T, which ensures that files with shorter transmission tolerable time are transmitted first.

[0082] (5).

[0083] wherein, is the transmission tolerable time of the file, and are respectively the minimum transmission tolerable time threshold and the maximum transmission tolerable time threshold specified by the service to which the file belongs (if not specified, the system default value is used).

[0084] Therefore, the preset packaging rule based on the file packaging priority comprehensively considers the file size, the device load state, the transmission tolerable time and the file storage information to determine the file packaging priority and optimize the packaging rule. Under the premise of meeting the upper limit of the file packaging block size, efficient and fast file packaging block division is achieved, ensuring that the file packaging work is orderly and efficient.

[0085] Step 102, performing a packaging operation according to the file list to be packaged to obtain a file packaging block in the first time period;

[0086] In one embodiment, the packaging operation according to the file list to be packaged to obtain a file packaging block in the first time period includes:

[0087] Obtaining file metadata and file data of each file to be packaged from path information of the file indicated by the file list to be packaged;

[0088] Obtaining a first cryptographic hash value according to the file metadata and the file data;

[0089] Splicing the file metadata, the file data and the first cryptographic hash value to obtain the file packaging block in the first time period.

[0090] The file metadata at least includes file size and file name.

[0091] In the embodiment of the present application, the file metadata and the file data of all the files to be packed are spliced to obtain a file packing block, and further, the MD5x message digest algorithm is used to obtain a first cryptographic hash value, i.e., an MD5 value, of the file packing block. The MD5 value is spliced with the file packing block to obtain the file packing block of the first time period.

[0092] In step 103, the file packing block is transmitted to the receiving end, and the transmission parameter of the first time period is obtained during the transmission of the file packing block, and the transmission result of the first time period fed back by the receiving end is obtained.

[0093] The transmission parameter of the first time period includes a packet loss rate, and the transmission result of the first time period includes a data integrity check result and a receiving rate.

[0094] In the embodiment of the present application, the sending end transmits the file packing block to the receiving end. Moreover, during the transmission of the file packing block, the sending end calculates the packet loss rate of the file packing block and records it. The receiving end receives the file packing block, performs data integrity check on the file packing block, calculates the receiving rate, and transmits the data integrity check result and the receiving rate to the sending end.

[0095] It should be noted that after the transmission of the file packing block of the first time period is completed, it is determined whether all the file packing blocks are transmitted, and if there is a file packing block not transmitted, the subsequent steps are continued.

[0096] In step 104, the file packing block size of the second time period is obtained according to the transmission parameter, the transmission result, and the bandwidth prediction result of the second time period, and the second time period is the next time period of the first time period.

[0097] Here, the length of the time period can be determined by a packing block size decision model.

[0098] It should be noted that after the receiving end receives the transmission result, if the data integrity check result in the transmission result indicates that the data integrity check fails, the sending end needs to retransmit the file packing block to the receiving end until the data integrity check result in the transmission result indicates that the data integrity check is successful.

[0099] In one embodiment, the file packing block size of the second time period is obtained according to the transmission parameter, the transmission result, and the bandwidth prediction result of the second time period, which includes:

[0100] Based on the transmission parameters and the transmission results, obtain the reward parameters for the first time period of the packet size decision model;

[0101] Based on historical network data and bandwidth prediction results for the second time period, network status evaluation indicators are obtained.

[0102] Based on the network status evaluation index, the reward parameter, and the block size decision model, the adjustment coefficient for the file block size in the second time period is obtained;

[0103] The file pack block size for the second time period is obtained based on the adjustment coefficient, the bandwidth prediction result, and the round-trip delay of the first time period.

[0104] In this embodiment of the application, the reward parameters for each file block in the first time period of the block size decision model are obtained based on the transmission parameters and the transmission results, as shown in the following formula (6):

[0105] ;

[0106] in, and These are the receiving rate and packet loss rate during the transmission of the file package blocks, respectively.

[0107] Based on the reward parameters of each file block in the first time period of the block size decision model, the average reward parameter of all file blocks in the first time period is obtained. .

[0108] Furthermore, based on the historical bandwidth data of the link in the historical network data, the historical bandwidth variation coefficient is obtained, as shown in the following formula (7):

[0109] (7);

[0110] in, Link historical bandwidth data for +1 time period ; yes The average of historical bandwidth data of the link over a period of time.

[0111] Furthermore, based on the historical round-trip time (RRT) data in the historical network data of the link, the historical RRT variation coefficient is obtained, as shown in the following formula (8):

[0112] (8);

[0113] in, link history RTT data of a time period ; is a mean value of the link history RTT data of a time period.

[0114] And, according to the bandwidth prediction result of the second time period and the bandwidth of the first time period, a bandwidth change rate is obtained, specifically as shown in the following formula (9):

[0115] (9);

[0116] wherein, is the bandwidth prediction result of the second time period; is the bandwidth prediction result of the first time period.

[0117] In combination with the above, the network state evaluation index is obtained, including the historical bandwidth variation coefficient, the historical RRT variation coefficient and the bandwidth change rate described above. Therefore, the network state can be represented by the evaluation index triplet ( , , ).

[0118] The packing block size decision model in the embodiments of the present application is a deep reinforcement learning model, including a state space {S} for representing the link network state, wherein the network state reflects the network data transmission efficiency and carrying capacity in the second time period. The state space {S} can be composed of the evaluation index triplet ( , , ).

[0119] Further, the network state evaluation index ( , , ) and the mean value of the reward parameter of all file packing blocks in the first time period are input into the packing block size decision model, and the adjustment coefficient A of the file packing block size in the second time period is calculated by the deep learning neural network of the packing block decision model, A {A}, {A} is the action space of the packing block size decision model, which is the discrete space of the file packing block size adjustment coefficient.

[0120] According to the bandwidth prediction result of the second time period and the RRT of the first time period, a bandwidth delay product (BDP) of the second time period is obtained, and specifically, the BDP of the second time period is equal to the product of the bandwidth prediction result of the second time period and the average RRT of the first time period.

[0121] Finally, according to the adjustment coefficient and the BDP of the second time period, a file packing block size of the second time period is obtained, and specifically, as shown in the following formula (10):

[0122] (10);

[0123] Wherein, F is the file packing block size of the second time period, BDP is the bandwidth delay product of the second time period, and A is the adjustment coefficient.

[0124] Therefore, by using deep reinforcement learning, the embodiment of the present application realizes dynamic adaptation to network state changes, so that the file packing block size is self-adaptively adjusted, and the data transmission efficiency and link utilization rate are improved.

[0125] Figure 2 The flowchart of the first embodiment of the file transmission method is shown. Next, the decision process of the packing block size decision model will be described. Figure 2 The decision process of the packing block size decision model will be described.

[0126] Step 201, input state: input the network state evaluation index of the second time period into the packing block size decision model;

[0127] Step 202, input reward: input the average of the reward parameters of all file packing blocks of the first time period into the packing block size decision model;

[0128] Step 203, output action: the packing block size decision model outputs the adjustment coefficient of the file packing block size of the second time period;

[0129] Step 204, update size: according to the adjustment coefficient, the bandwidth prediction result and the RRT of the first time period, the file packing block size of the second time period is calculated;

[0130] Step 205, update state: according to the historical network data of the link and the bandwidth prediction result of the third time period, the network state evaluation index of the third time period is obtained.

[0131] In one embodiment, the method further comprises:

[0132] According to the link historical traffic data and the time series prediction model, traffic prediction data of the second time period is obtained;

[0133] According to the traffic prediction data and the link bandwidth upper limit, a bandwidth prediction result of the second time period is obtained.

[0134] In the embodiment of the present application, based on the link historical traffic data, the bandwidth prediction result of the second time period of the link can be predicted, so as to combine the receiving rate of the file packing block of the first time period fed back by the receiving end, the packet loss rate of the file packing block of the first time period calculated by the sending end, and the network state evaluation index of the second time period evaluated by the link historical network data, and finally use the packing block size decision model to complete the file packing block size decision.

[0135] Optionally, the link historical traffic data is input into a time series prediction model to obtain traffic prediction data of the second time period, wherein is a timestamp, indicating the time period, represents the link traffic data at time .

[0136] Optionally, the time series prediction model includes a temporal convolutional network (TCN) or a self-attention model (Transformer).

[0137] Suppose the link bandwidth upper limit is , and the traffic prediction data of the second time period is , then the bandwidth prediction result of the second time period is , realizing the prediction of the available bandwidth of the link.

[0138] Optionally, the file packing block size of the first time period is obtained according to the bandwidth prediction result of the first time period, the transmission parameters and transmission results of the file packing block transmitted to the receiving end in the third time period, and the third time period is the last time period of the first time period.

[0139] It can be understood that according to the bandwidth prediction result of the last time period, the transmission parameters and transmission results of the next time period, the file packing block size of the next time period can be obtained.

[0140] Figure 3 is a flowchart of the second embodiment of the file transmission method according to the present application. As Figure 3As shown, the file transmission method is applied to a sending end, and the sending end comprises a to-be-packed file directory generation module, a file packing module, a file transmission module, a network monitoring module, a data processing module, a traffic prediction module, and a deep reinforcement learning module, and specifically:

[0141] The to-be-packed file directory generation module is configured to generate a to-be-packed file directory of a first time period according to a preset packing rule and a file packing block size of the first time period;

[0142] The file packing module is configured to perform a packing operation according to the to-be-packed file directory to obtain a file packing block of the first time period.

[0143] The file transmission module is configured to transmit the file packing block to a receiving end and record transmission parameters obtained in a transmission process of the file packing block and transmission results fed back by the receiving end; if the transmission of the file packing block is completed, the current file packing block transmission task is ended, and a next process is performed.

[0144] The network monitoring module is configured to monitor link network data including bandwidth, traffic, and RRT and send the link network data to the receiving end.

[0145] The data processing module is configured to perform data processing according to the transmission parameters, the transmission results, and the link network data to obtain a reward parameter of the first time period.

[0146] The traffic prediction module is configured to obtain traffic prediction data of a second time period using a time series prediction model according to link historical traffic data.

[0147] The deep reinforcement learning module is configured to output a file packing block size of the second time period from a packing block size decision model according to the traffic prediction data of the second time period, the reward parameter of the first time period, and link historical network data.

[0148] Figure 4 A flowchart of an embodiment three of the file transmission method is shown in FIG. 4. Figure 4 As shown, the method comprises:

[0149] Step 401, initialization: including initialization (using a default value) of a file packing block size and initialization (using a default value) of a packing block decision model, wherein the initialization of the packing block decision model comprises: a state space {S} and an action space {A}, the state space {S} is configured to represent a link network state, so that the packing block size decision model can perceive network data; the action space {A} is a discrete space of an adjustment coefficient of the file packing block size, and is configured to obtain the file packing block size.

[0150] Step 402, set a preset packing rule:

[0151] Step 403, generating a file directory of a first time period to be packed;

[0152] Step 404, obtaining a file packing block of the first time period;

[0153] Step 405, transmitting the file packing block to a receiving end;

[0154] Step 406, obtaining a transmission result fed back by the receiving end;

[0155] Step 407, judging whether transmission is completed;

[0156] Step 408, if transmission is not completed, obtaining a reward parameter of the first time period according to the transmission result;

[0157] Step 409, obtaining flow prediction data of a second time period using a time series prediction model;

[0158] Step 410, obtaining a file packing block size of the second time period using a packing block size decision model according to the reward parameter and the flow prediction data;

[0159] Step 411, if transmission is completed, ending the transmission task.

[0160] Please refer to Figure 5 The embodiment of the application further provides a file transmission method applied to a receiving end, which can be a gateway.

[0161] Further, the method comprises:

[0162] Step 501, receiving a file packing block of a first time period transmitted by a sending end;

[0163] Step 502, feeding back a transmission result of the file packing block to the sending end.

[0164] The transmission result of the first time period comprises a data integrity check result and a receiving rate, and the data integrity check result is used to indicate whether data integrity check of the file packing block of the first time period is successful.

[0165] In an embodiment, the method further comprises:

[0166] Obtaining a second password hash value according to file metadata and file transmission data in the file packing block;

[0167] Judging whether the first password hash value in the file packing block is consistent with the second password hash value;

[0168] If the first password hash value is consistent with the second password hash value, obtaining a transmission result.

[0169] It should be noted that the number of file packing blocks transmitted in a time period is not limited in the embodiments of the present application.

[0170] In the embodiments of the present application, after the transmission of one file packing block is completed, the receiving end immediately starts a data integrity verification process:

[0171] First, the file metadata and file transmission data in the file packing block are extracted, and the MD5x information digest algorithm is used to obtain the second password hash value, i.e., the MD5 value, of the file packing block. At the same time, the first password hash value in the file packing block transmitted by the sending end is obtained. If the second password hash value is consistent with the first password hash value, it is determined that the data integrity verification result is successful, and further file packing block splitting operation can be performed to obtain all files. If the second password hash value is inconsistent with the first password hash value, it is determined that the data integrity verification result is failed, and the file packing block is discarded.

[0172] Then, after the data integrity verification, the receiving end calculates the receiving rate, which is equal to the file packing block size divided by the receiving time, with the unit of MB / s.

[0173] Finally, according to the data integrity verification result and the receiving rate, the transmission result is obtained, and the receiving end transmits the transmission result to the sending end.

[0174] In summary, the file transmission method described in the embodiments of the present application is essentially a mass small file packing transmission method based on deep reinforcement learning. Based on the collaborative application of time series prediction technology and deep reinforcement learning algorithm, a packing block size decision model is constructed to calculate the file packing block size of the next time period according to real-time network state evaluation indicators. At the same time, considering multiple factors such as file size, device load state, and transmission tolerable time, a preset packing rule is designed to achieve efficient packing of the sending end, which can effectively solve the problems of low transmission efficiency of small files, insufficient utilization of network resources, and the influence of packing strategy on transmission efficiency in the prior art, and achieve efficient and stable transmission of mass small files in complex network environment.

[0175] As shown in Figure 6 The embodiments of the present application also provide a file transmission device applied to a sending end, which comprises:

[0176] The generating module 601 is configured to generate a to-be-packed file directory of a first time period according to a preset packing rule and a file packing block size of the first time period, wherein the to-be-packed file directory is used to indicate path information of each file to be packed.

[0177] The packing module 602 is configured to perform a packing operation according to the file list to be packed, and obtain a file packing block in a first time period;

[0178] The transmission module 603 is configured to transmit the file packing block to a receiving end, and obtain a transmission parameter in the first time period during transmission of the file packing block, and obtain a transmission result in the first time period fed back by the receiving end;

[0179] The decision module 604 is configured to obtain a file packing block size in a second time period according to the transmission parameter, the transmission result, and a bandwidth prediction result in the second time period, the second time period being a next time period of the first time period.

[0180] Optionally, the file transmission device, wherein the preset packing rule is related to a file packing priority and / or file storage information;

[0181] The file packing priority is related to at least one of a file size corresponding to the file, a device load state, and a transmission tolerable time, wherein the smaller the file size, the higher the file packing priority; the higher the device load state, the higher the file packing priority; and the shorter the transmission tolerable time, the higher the file packing priority;

[0182] The file storage information is used to indicate whether at least two files are on a same storage device.

[0183] Optionally, the file transmission device, wherein the packing module 602 is specifically configured to:

[0184] Obtain file metadata and file data of each file to be packed from path information of the file to be packed indicated by the file list to be packed;

[0185] Obtain a first cryptographic hash value according to the file metadata and the file data;

[0186] Splice the file metadata, the file data, and the first cryptographic hash value to obtain the file packing block in the first time period.

[0187] Optionally, the file transmission device, wherein the decision module 604 is specifically configured to:

[0188] Obtain a reward parameter of a packing block size decision model in the first time period according to the transmission parameter and the transmission result;

[0189] Obtain a network state evaluation index according to link historical network data and a bandwidth prediction result in the second time period;

[0190] According to the network state evaluation index, the reward parameter and the packaging block size decision model, an adjustment coefficient of a file packaging block size of the second time period is obtained;

[0191] According to the adjustment coefficient, the bandwidth prediction result and the round-trip delay of the first time period, a file packaging block size of the second time period is obtained.

[0192] Optionally, the file transmission device further comprises:

[0193] The flow prediction module is configured to obtain flow prediction data of the second time period according to historical flow data of the link and a time series prediction model.

[0194] The bandwidth prediction module is configured to obtain a bandwidth prediction result of the second time period according to the flow prediction data and an upper limit of the link bandwidth.

[0195] Optionally, the file packaging block size of the first time period is obtained according to a bandwidth prediction result of the first time period, a transmission parameter and a transmission result of a file packaging block transmitted by the sending end to the receiving end in a third time period, and the third time period is a time period before the first time period.

[0196] It should be noted that the above device provided by the embodiments of the present application can realize all the method steps achieved by the above file transmission method embodiments, and achieve the same technical effects. Therefore, the same parts and beneficial effects of the method embodiments will not be described in detail.

[0197] As shown in Figure 7 The embodiments of the present application further provide a file transmission device applied to a receiving end, and the device comprises:

[0198] The receiving module 701 is configured to receive a file packaging block of a first time period transmitted by a sending end.

[0199] The feedback module 702 is configured to feed back a transmission result of the file packaging block to the sending end.

[0200] Optionally, the file transmission device further comprises:

[0201] The obtaining module is configured to obtain a second cryptographic hash value according to file metadata and file transmission data in the file packaging block.

[0202] The judging module is configured to judge whether the first cryptographic hash value in the file packaging block is consistent with the second cryptographic hash value.

[0203] The verification module is used to obtain the transmission result if the first cryptographic hash value is consistent with the second cryptographic hash value.

[0204] It should be noted that the apparatus provided in this application embodiment can implement all the method steps implemented in the above file transfer method embodiment and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.

[0205] This application also provides a file transfer device, such as... Figure 8 As shown, it includes:

[0206] The processor 801, memory 802, transceiver 803, and a program or instructions stored in the memory 802 and executable on the processor 801; when the processor 801 executes the program or instructions, it implements the various processes of the above-described file transfer method embodiments and achieves the same technical effect. To avoid repetition, these will not be described again here.

[0207] The transceiver 803 is used to receive and send data under the control of the processor 801.

[0208] Among them, Figure 8 In this context, the bus architecture can include any number of interconnected buses and bridges, specifically connecting various circuits of one or more processors represented by processor 801 and memory represented by memory 802. The bus architecture can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface. The transceiver 803 can be multiple elements, including transmitters and receivers, providing a unit for communicating with various other devices over a transmission medium. For different user equipment, the user interface 804 can also be an interface capable of connecting external or internal devices, including but not limited to keypads, displays, speakers, microphones, joysticks, etc.

[0209] The processor 801 is responsible for managing the bus architecture and general processing, while the memory 802 can store the data used by the processor 801 when performing operations.

[0210] The embodiment of the present application further provides a computer readable storage medium, and the computer readable storage medium stores a computer program. The computer program is executed by a processor to implement each process of the file transmission method embodiment, and achieves the same technical effects. To avoid repetition, details are not described herein. The computer readable storage medium includes a read-only memory (ROM), a random access memory (RAM), a magnetic disk, an optical disk, and the like.

[0211] The embodiment of the present application further provides a computer program product, including computer instructions, which are executed by a processor to implement each process of the file transmission method embodiment, and achieve the same technical effects. To avoid repetition, details are not described herein.

[0212] It should be noted that, in this document, the term "comprising" or "comprises" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or apparatus including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such a process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus including the element.

[0213] From the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment method can be realized by means of software and necessary general hardware platform, of course, it can also be realized by hardware, but in many cases, the former is a better embodiment. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a plurality of instructions for making a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) execute the method described in each embodiment of the present application.

[0214] The embodiments of the present application are described above in combination with the drawings, but the present application is not limited to the above specific embodiments, and the above specific embodiments are only illustrative, not restrictive. Those skilled in the art can make many forms under the inspiration of the present application without departing from the scope of the present application and the protection scope of the claims.

[0215] The above is the preferred embodiment of the present application, it should be noted that for those skilled in the art, without departing from the principles described in the present application, can also be made several improvements and refinements, these improvements and refinements should also be considered the scope of protection of the present application.

Claims

1. A file transfer method, characterized in that, Applied to the sending end, the method includes: Based on the preset packaging rules and the file packaging block size of the first time period, a file directory to be packaged for the first time period is generated, wherein the file directory to be packaged is used to indicate the path information of each file to be packaged; Perform a packaging operation based on the directory of files to be packaged to obtain a file packaging block for the first time period; The file package block is transmitted to the receiving end, and during the transmission of the file package block, the transmission parameters of the first time period and the transmission results of the first time period fed back by the receiving end are obtained. Based on the transmission parameters, the transmission results, and the bandwidth prediction results of the second time period, the file packing block size of the second time period is obtained, and the second time period is the next time period after the first time period.

2. The method according to claim 1, characterized in that, The preset packaging rules are related to file packaging priority and / or file storage information; The file packaging priority is based on at least one of the following: file size, device load status, and tolerable transmission time. Specifically, the smaller the file size, the higher the file packaging priority; the higher the device load status, the higher the file packaging priority; and the shorter the tolerable transmission time, the higher the file packaging priority. The file storage information is used to indicate whether at least two of the files are on the same storage device.

3. The method according to claim 1, characterized in that, Perform a packaging operation based on the directory of files to be packaged to obtain a file packaging block for the first time period, including: From the path information of each file to be packaged indicated by the directory of files to be packaged, obtain the file metadata and file data of the file; Based on the file metadata and the file data, a first cryptographic hash value is obtained; The file metadata, the file data, and the first password hash value are concatenated to obtain a file package block for the first time period.

4. The method according to claim 1, characterized in that, Based on the transmission parameters, the transmission results, and the bandwidth prediction results for the second time period, the file packet size for the second time period is obtained, including: Based on the transmission parameters and the transmission results, obtain the reward parameters for the first time period of the packet size decision model; Based on historical network data and bandwidth prediction results for the second time period, network status evaluation indicators are obtained. Based on the network status evaluation index, the reward parameter, and the block size decision model, the adjustment coefficient for the file block size in the second time period is obtained; The file pack block size for the second time period is obtained based on the adjustment coefficient, the bandwidth prediction result, and the round-trip delay of the first time period.

5. The method according to claim 1 or 4, characterized in that, The method further includes: Based on historical link traffic data and time series prediction models, traffic prediction data for the second time period is obtained; Based on the traffic prediction data and the link bandwidth limit, the bandwidth prediction result for the second time period is obtained.

6. The method according to claim 1, characterized in that, The size of the file packet block in the first time period is obtained based on the bandwidth prediction result of the first time period, the transmission parameters and transmission results of the file packet block transmitted to the receiving end in the third time period, and the third time period is the time period preceding the first time period.

7. A file transfer method, characterized in that, Applied to the receiving end, the method includes: Receive the first time period of file packets transmitted by the sending end; The transmission result of the file package block is fed back to the sending end.

8. The method according to claim 7, characterized in that, The method further includes: The second cryptographic hash value is obtained based on the file metadata and file transfer data in the file package block; Determine whether the first cryptographic hash value and the second cryptographic hash value in the file packaging block are consistent; If the first cryptographic hash value matches the second cryptographic hash value, then the transmission result is obtained.

9. A file transfer device, characterized in that, Applied to the transmitting end, the device includes: The generation module is used to generate a directory of files to be packaged for the first time period according to preset packaging rules and the file packaging block size for the first time period. The directory of files to be packaged is used to indicate the path information of each file to be packaged. The packaging module is used to perform a packaging operation based on the directory of files to be packaged, and obtain the file packaging block for the first time period; The transmission module is used to transmit the file package block to the receiving end, and to obtain the transmission parameters of the first time period and the transmission result of the first time period fed back by the receiving end during the transmission of the file package block. The decision module is used to obtain the file packing block size of the second time period based on the transmission parameters, the transmission result, and the bandwidth prediction result of the second time period, wherein the second time period is the next time period after the first time period.

10. A file transfer device, characterized in that, Applied to the receiving end, the device includes: The receiving module is used to receive the file package blocks of the first time period transmitted by the sending end; The feedback module is used to send the transmission result of the file package block back to the sending end.

11. A file transfer device, characterized in that, include: A processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the processor, when executing the program or instructions, implements the file transfer method as described in any one of claims 1 to 6, or implements the file transfer method as described in any one of claims 7 to 8.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the file transfer method as described in any one of claims 1 to 6, or implements the file transfer method as described in any one of claims 7 to 8.

13. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement the file transfer method as described in any one of claims 1 to 6, or implement the file transfer method as described in any one of claims 7 to 8.

Images