File transmission method, electronic equipment and storage medium
By slicing files and optimizing the transmission speed of files in the data exchange platform, the problem of inefficient large file transfer is solved, and a more stable and efficient file transfer process is achieved.
Patent Information
- Application Number
- CN202510250208.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-04
- Publication Date
- 2025-04-11
AI Technical Summary
In a data exchange platform, when external data parties transmit large files, direct uploading may lead to upload failure, inefficiency or waste of resources.
By comparing the target file size with the file maximum transfer threshold, if the threshold is exceeded, the file is sliced and the single slice size is determined based on the maximum transfer speed, and transferred slice by slice to the data exchange platform.
It improves the efficiency of file transfer, avoids the problem of failure of direct upload of large files, and ensures the stability of the transmission process and the effective utilization of resources.
Smart Images
Figure CN119814772A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data transmission, and in particular to a file transmission method, electronic equipment and storage medium. Background Art
[0002] The data exchange platform is a platform that provides services during the data exchange stage of the data life cycle. The data exchange platform exchanges data with external data parties. The external data parties here refer to customers or subsidiaries, etc. Before the birth of the data exchange platform, the management authority of interactive data with customers or subsidiaries was chaotic, and the transmission channels were diverse, such as e-mail, various instant messaging software, etc., and the supervision of data security was relatively chaotic. The use of a data exchange platform can solve this problem well.
[0003] External data parties usually have corresponding storage directories on the data exchange platform. External data parties can transfer files to their corresponding storage directories. When large file data is being transferred, directly uploading the entire large file may cause upload failure, resulting in inefficiency or waste of resources. Summary of the invention
[0004] In view of the above technical problems, the technical solution adopted by the present invention is: a file transmission method, which is used for an external data party to transmit files to a data exchange platform, and the method comprises the following steps:
[0005] S100, obtaining the target file and target file size G0 transmitted by the external data party; wherein the external data party has a corresponding storage directory on the data exchange platform;
[0006] S200, comparing the target file size G0 with the maximum file transfer threshold, if G0 is not greater than the maximum file transfer threshold, directly transferring the target file to the storage directory corresponding to the data exchange platform;
[0007] If G0 is greater than the maximum file transfer threshold, execute S300;
[0008] S300, obtaining the maximum transmission speed v0 transmitted by the external data party to the data exchange platform, and determining the size of a single slice of the target file based on the maximum transmission speed v0, slicing the target file based on the single slice size and transmitting the single slice to a storage directory corresponding to the data exchange platform;
[0009] Among them, when the external used capacity is greater than the external available capacity, a data clearing prompt message is sent to the external data party, the external available capacity is the product of the total capacity of the corresponding storage directory of the external data party on the data exchange platform and the preset clearing ratio, and the external capacity is the used capacity of the corresponding storage directory of the external data party on the data exchange platform.
[0010] According to another aspect of the present invention, a non-transitory computer-readable storage medium is provided, in which at least one instruction or at least one program is stored, and the at least one instruction or the at least one program is loaded and executed by a processor to implement the aforementioned method.
[0011] According to yet another aspect of the present invention, an electronic device is provided, comprising a processor and the aforementioned non-transitory computer-readable storage medium.
[0012] The present invention has at least the following beneficial effects: In summary, the target file and the target file size transmitted by the external data party are obtained, the target file size is compared with the maximum file transmission threshold, if the target file size is not greater than the maximum file transmission threshold, the target file is directly transmitted to the storage directory corresponding to the data exchange platform, if the target file size is greater than the maximum file transmission threshold, the maximum transmission speed transmitted by the external data party to the data exchange platform is obtained, and the single slice size of the target file is determined based on the maximum transmission speed, the target file is sliced based on the single slice size and the single slice is transmitted to the storage directory corresponding to the data exchange platform. The present invention determines whether to slice the target file based on the target file size, thereby improving transmission efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0014] Figure 1 A flowchart of a file transmission method provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0015] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.
[0016] The embodiment of the present invention provides a file transmission method, which is used for an external data source to transmit a file to a data exchange platform. Figure 1 As shown, the method comprises the following steps:
[0017] S100, obtaining the target file and target file size G0 transmitted by the external data party; wherein the external data party has a corresponding storage directory on the data exchange platform.
[0018] S200, compare the target file size G0 with the maximum file transfer threshold, if G0 is not greater than the maximum file transfer threshold, directly transfer the target file to the storage directory corresponding to the data exchange platform. If G0 is greater than the maximum file transfer threshold, execute S300.
[0019] Specifically, the maximum file transfer threshold can be determined according to actual needs. It can be understood that if G0 is not greater than the maximum file transfer threshold, the target file is considered not to be a large file and does not need to be sliced and transferred. The target file can be directly transferred to the storage directory corresponding to the data exchange platform.
[0020] S300, obtaining the maximum transmission speed v0 transmitted by the external data party to the data exchange platform, and determining the single slice size of the target file based on the maximum transmission speed v0, slicing the target file based on the single slice size and transmitting the single slice to the storage directory corresponding to the data exchange platform.
[0021] The maximum transmission speed is the maximum speed at which external data is transmitted to the data exchange platform; the maximum transmission speed in the present invention is a theoretical value.
[0022] Among them, when the external used capacity is greater than the external available capacity, a data clearing prompt message is sent to the external data party, the external available capacity is the product value of the total capacity of the storage directory corresponding to the external data party on the data exchange platform and the preset clearing ratio, and the external used capacity is the used capacity of the storage directory corresponding to the external data party on the data exchange platform. Specifically, when the external used capacity> the product value of the external total capacity and the preset clearing ratio, a data clearing prompt message is sent to the external data party; specifically, the data clearing prompt message can be: your storage capacity has reached the upper limit, please perform a clearing operation.
[0023] Specifically, after S300, the process further includes: if data clearing confirmation information from the external data party is received, clearing the corresponding storage directory of the external data party on the data exchange platform.
[0024] In summary, the target file and target file size transmitted by the external data party are obtained, and the target file size is compared with the maximum file transmission threshold. If the target file size is not greater than the maximum file transmission threshold, the target file is directly transmitted to the storage directory corresponding to the data exchange platform. If the target file size is greater than the maximum file transmission threshold, the maximum transmission speed transmitted by the external data party to the data exchange platform is obtained, and the single slice size of the target file is determined based on the maximum transmission speed. The target file is sliced based on the single slice size and the single slice is transmitted to the storage directory corresponding to the data exchange platform. The present invention determines whether to slice the target file based on the target file size, thereby improving transmission efficiency.
[0025] In S300, the size of a single slice of the target file is determined based on the maximum transmission speed v0, and further includes:
[0026] Get the speed slice function v=F1(x), where the dependent variable v is the maximum transmission speed, the independent variable x is the size of a single slice, and F1() is the speed slice correspondence law.
[0027] Substitute the maximum transmission speed v0 into the speed slicing function v=F1(x) to obtain the single slice size of the target file. Specifically, v0=F1(x0), where x0 is the single slice size.
[0028] Among them, S300 also includes obtaining a preset clearing ratio through the following steps:
[0029] Get the speed-capacity ratio function v=F2(y), where the dependent variable v is the maximum transmission speed, the independent variable y is the clearing ratio, F2() is the speed-capacity ratio correspondence rule, and the clearing ratio is the ratio of the external used capacity to the external total capacity for sending data clearing prompt information to the external data party.
[0030] Substitute the maximum transmission speed v0 into the speed capacity ratio function v=F2(y) to obtain the preset clearing ratio. Specifically, v0=F2(y0), y0 is the preset clearing ratio.
[0031] In summary, the present invention makes the single slice size and the preset clearing ratio more reasonable through the speed slice function v=F1(x) and the speed capacity ratio function v=F2(y).
[0032] Further, it can be understood that when the transmission speed is faster, the used capacity will approach the available capacity faster, so there is a certain functional relationship between the transmission speed and the clearing ratio; when the target file is divided into several single slices, the number of slices transmitted by the transmission channel at a time is limited. The more single slices there are, the longer the transmission time is. There is a certain functional relationship between the transmission speed and the size of a single slice. Therefore, the present invention uses two functional relationships to determine the clearing ratio and the size of a single slice.
[0033] Specifically, the speed slice function v=F1(x) and the speed capacity ratio function v=F2(y) are obtained by the following steps:
[0034] S001, obtain a sample vector list A = {A1, A2, ..., A i , …, A m}, the i-th sample vector A i =(x i ,y i , v i ), x i is the size of a single slice during the transmission of the i-th sample, y i is the clearing ratio in the i-th sample transmission, v i is the maximum transmission speed during the transmission of the i-th sample, the value range of i is 1 to m, and m is the number of sample vectors.
[0035] S002, construct a neural network model, input the sample vector list A into the constructed neural network model, perform training, and obtain a target neural network model.
[0036] Specifically, S002 also includes: obtaining training results, and if the accuracy in the training results is greater than a preset accuracy threshold, using the trained model as the target neural network model.
[0037] Specifically, the preset accurate threshold can be determined according to actual needs.
[0038] S003, obtain the input data list V for training = {V1, V2, ..., V j , …, V n}, the jth sample vector V j is the maximum transmission speed in the jth training transmission, j ranges from 1 to n, and n is the number of training input data.
[0039] S004, traverse the training input data list V, and convert V j Input the target neural network model and obtain the training output vector list C = {C1, C2, ..., C j , …, C n}, B j The corresponding training output vector C j =(x j ,y j ), x j It is B j The corresponding single slice size, y j It is B jThe corresponding training clearing ratio. The training clearing ratio is the ratio of the used capacity of the training external data to the total capacity of the training external data when sending a data clearing prompt message to the training external data party; the used capacity of the training external data is the used capacity of the directory corresponding to the training transmission, and the total capacity of the training external data is the total used capacity of the directory corresponding to the training transmission.
[0040] S005, obtain a list of several intermediate vectors for training E = {E1, E2, ..., E r , …E s}, the rth training intermediate vector E r =(G r , N r ), G r is the size of the file transferred for the rth training transmission, N r is the total capacity of the directory transmitted for the rth training, r ranges from 1 to s, and s is the number of intermediate vectors used for training.
[0041] S006, traverse D, D j , E r Using C j The corresponding data is transmitted to obtain the training transmission score H jr , and based on the training transmission score H jr Get the intermediate score H for training j0 Among them, S006 includes: H j0 =∑ s r＝1 H jr .
[0042] Specifically, it can be understood that when E r Using C j When the corresponding data is transmitted, there may be a problem of too many slices, or the directory may need to be cleared quickly. Scoring is performed based on different preset dimensions to obtain the transmission score for training. The preset dimensions include: directory clearing time, transmission time, etc.
[0043] In one embodiment of the present invention, based on B j , C j and G r Obtain the transmission time; obtain the first score AH based on the transmission time jr ; Based on N r ,y j , get the clearing time, and get the second score BH based on the clearing time jr ;H jr =a1×AH jr +a2×BH jr , where a1 is the first weight factor and a2 is the second weight factor.
[0044] S007, if H j0 If it is greater than the preset transmission threshold H0, H j0 Corresponding C j Marked as the intermediate training vector J g , thereby obtaining the intermediate training vector list J = {J1, J2, ..., J g , …, J z}, the value range of g is 1 to z, and the number of intermediate training vectors z≤n; where J g =(x g ,y g ), x g =x j ,y g =y j .
[0045] S008, based on the training input data list B and the intermediate training vector list J, obtain the velocity slice function v=F1(x) and the velocity capacity ratio function v=F2(y).
[0046] In summary, obtain a list of sample vectors, build a neural network model, input the list of sample vectors into the constructed neural network model, train, obtain the target neural network model, obtain a list of training input data, traverse the list of training input data, input the training input data into the target neural network model, obtain a list of training output vectors, obtain several lists of training intermediate vectors, traverse D, and train D j , E r Using C j The corresponding data is transmitted to obtain the training transmission score H jr , and based on the training transmission score H jr Get the intermediate score H for training j0 , if H j0 If it is greater than the preset transmission threshold H0, H j0 Corresponding C j Marked as the intermediate training vector J g, thereby obtaining an intermediate training vector list, based on the training input data list B and the intermediate training vector list J, obtaining the speed slicing function v=F1(x) and the speed capacity ratio function v=F2(y), the present invention obtains the target neural network model by training the constructed neural network model, inputs the training input vector into the target neural network model, obtains the training output vector, and scores based on the training output vector, fits the coordinate points corresponding to the training output vector that meets the scoring requirements, generates the speed slicing function and the speed capacity ratio function, the present invention judges the score, thereby obtaining high-quality data points, determines the speed slicing function and the speed capacity ratio function based on the high-quality data points, thereby more accurately obtaining the speed slicing function and the speed capacity ratio function.
[0047] Furthermore, S008 also includes:
[0048] S081, Get J g The corresponding training input data, and J g The corresponding training input data is marked as intermediate input data H g .
[0049] S082, obtain the first coordinate point list AJ = {AJ1, AJ2, ..., AJ g , …, AJ z} and the second coordinate point list BJ = {BJ1, BJ2, ..., BJ g , …, BJ z}, the gth first coordinate point AJ g =(x g , H g ), the g-th second coordinate point BJ g =(y g , H g ).
[0050] S083, performing fitting operation based on the first coordinate point list AJ to obtain a velocity slice function v=F1(x).
[0051] S084, performing fitting operation based on the second coordinate point list BJ to obtain the speed capacity ratio function v=F2(y).
[0052] Specifically, those skilled in the art know that any fitting operation in the prior art belongs to the protection scope of the present invention and will not be described in detail here.
[0053] In summary, obtain J g The corresponding training input data, and J g The corresponding training input data is marked as intermediate input data H g, obtain a first coordinate point list and a second coordinate point list, perform a fitting operation based on the first coordinate point list, obtain a speed slicing function, perform a fitting operation based on the second coordinate point list, and obtain a speed capacity ratio function.
[0054] An embodiment of the present invention also provides a non-transitory computer-readable storage medium, which can be set in an electronic device to store at least one instruction or at least one program related to implementing a method in a method embodiment. The at least one instruction or the at least one program is loaded and executed by the processor to implement the method provided in the above embodiment.
[0055] An embodiment of the present invention further provides an electronic device, comprising a processor and the aforementioned non-transitory computer-readable storage medium.
[0056] Although some specific embodiments of the present invention have been described in detail by way of example, it should be understood by those skilled in the art that the above examples are only for illustration, not for limiting the scope of the present invention. It should also be understood by those skilled in the art that various modifications may be made to the embodiments without departing from the scope and spirit of the present invention.
Claims
1. A file transmission method, characterized in that: The file transfer method is used by an external data source to transfer files to a data exchange platform, and the method comprises the following steps: S100, obtaining the target file and target file size G0 transmitted by the external data party; wherein the external data party has a corresponding storage directory on the data exchange platform; S200, comparing the target file size G0 with the maximum file transfer threshold, if G0 is not greater than the maximum file transfer threshold, directly transferring the target file to the storage directory corresponding to the data exchange platform; If G0 is greater than the maximum file transfer threshold, execute S300; S300, obtaining the maximum transmission speed v0 transmitted by the external data party to the data exchange platform, and determining the size of a single slice of the target file based on the maximum transmission speed v0, slicing the target file based on the single slice size and transmitting the single slice to a storage directory corresponding to the data exchange platform; Among them, when the external used capacity is greater than the external available capacity, a data clearing prompt message is sent to the external data party, the external available capacity is the product of the total capacity of the corresponding storage directory of the external data party on the data exchange platform and the preset clearing ratio, and the external capacity is the used capacity of the corresponding storage directory of the external data party on the data exchange platform.
2. The file transmission method according to claim 1, characterized in that: In S300, the size of a single slice of the target file is determined based on the maximum transmission speed v0, including: Obtain the speed slice function v=F1(x), where the dependent variable v is the maximum transmission speed, the independent variable x is the size of a single slice, and F1() is the speed slice correspondence law; Substitute the maximum transmission speed v0 into the speed slicing function v=F1(x) to obtain the single slice size of the target file.
3. The file transmission method according to claim 2, characterized in that: S300 also includes obtaining a preset clearing ratio through the following steps: Obtain the speed capacity ratio function v=F2(y), where the dependent variable v is the maximum transmission speed, the independent variable y is the clearing ratio, and F2() is the speed capacity ratio corresponding rule. The clearing ratio is the ratio of the external used capacity to the external total capacity when sending a data clearing prompt message to the external data party; Substitute the maximum transmission speed v0 into the speed-to-capacity ratio function v=F2(y) to obtain the preset clearing ratio.
4. The file transmission method according to claim 3, characterized in that: The velocity slice function v=F1(x) and the velocity capacity ratio function v=F2(y) are obtained by the following steps: S001, obtain a sample vector list A = {A1, A2, ..., A i , …, A m }, the i-th sample vector A i =(x i ,y i , v i ), x i is the size of a single slice during the transmission of the i-th sample, y i is the clearing ratio in the i-th sample transmission, v i is the maximum transmission speed during the transmission of the i-th sample, the value range of i is 1 to m, and m is the number of sample vectors; S002, constructing a neural network model, inputting the sample vector list A into the constructed neural network model, performing training, and obtaining a target neural network model; S003, obtain the input data list V for training = {V1, V2, ..., V j , …, V n }, the jth sample vector V j is the maximum transmission speed in the jth training transmission, j ranges from 1 to n, and n is the number of training input data; S004, traverse the training input data list V, and convert V j Input the target neural network model and obtain the training output vector list C = {C1, C2, ..., C j , …, C n }, B j The corresponding training output vector C j =(x j ,y j ), x j It is B j The corresponding single slice size, y j It is B j The corresponding training clearing ratio; S005, obtain a list of several intermediate vectors for training E = {E1, E2, ..., E r , …E s }, the rth training intermediate vector E r =(G r , N r ), G r is the size of the file transferred for the rth training transmission, N r is the total capacity of the directory for the rth training transmission, r ranges from 1 to s, and s is the number of intermediate vectors for training; S006, traverse D, D j , E r Using C j The corresponding data is transmitted to obtain the training transmission score H jr , and based on the training transmission score H jr Get the intermediate score H for training j0 ; S007, if H j0 If it is greater than the preset transmission threshold H0, H j0 The corresponding C j Marked as the intermediate training vector J g , thereby obtaining the intermediate training vector list J = {J1, J2, ..., J g , …, J z }, the value range of g is 1 to z, and the number of intermediate training vectors z≤n; where J g =(x g ,y g ), x g =x j ,y g =y j ; S008, based on the training input data list B and the intermediate training vector list J, obtain the velocity slice function v=F1(x) and the velocity capacity ratio function v=F2(y).
5. The file transmission method according to claim 4, characterized in that: S008 also includes: S081, Get J g The corresponding training input data, and J g The corresponding training input data is marked as intermediate input data H g ; S082, obtain the first coordinate point list AJ = {AJ1, AJ2, ..., AJ g , …, AJ z } and the second coordinate point list BJ = {BJ1, BJ2, ..., BJ g , …, BJ z }, the gth first coordinate point AJ g =(x g , H g ), the g-th second coordinate point BJ g =(y g , H g ); S083, performing a fitting operation based on the first coordinate point list AJ to obtain a velocity slice function v=F1(x); S084, performing fitting operation based on the second coordinate point list BJ to obtain the speed capacity ratio function v=F2(y).
6. The file transmission method according to claim 4, characterized in that: S002 also includes: obtaining training results, and if the accuracy in the training results is greater than a preset accuracy threshold, using the trained model as a target neural network model.
7. The file transmission method according to claim 1, characterized in that: After S300, the method further includes: if data clearing confirmation information of the external data party is received, clearing the corresponding storage directory of the external data party on the data exchange platform.
8. The file transmission method according to claim 4, characterized in that: S006 includes:H j0 =∑ s r＝1 H jr .
9. A non-transitory computer-readable storage medium, wherein at least one instruction or at least one program is stored in the storage medium, characterized in that: The at least one instruction or the at least one program is loaded and executed by the processor to implement the file transmission method as described in any one of claims 1-8.
10. An electronic device, characterized in that: The invention comprises a processor and the non-transitory computer-readable storage medium as claimed in claim 9.