High-concurrency file queue transmission system and method and storage medium

Abstract

The invention discloses a high-concurrency file queue transmission system and method and a storage medium, relates to the technical field of network transmission, and solves the technical problem that the transmission efficiency of network transmission is reduced due to the fact that an existing transmission fragmentation method is difficult to adapt to different transmission file types and transmission network states. Comprising the steps of obtaining a file type and a file size of a to-be-transmitted file and network state data of a current transmission network; performing transmission fragmentation on the to-be-transmitted file based on the file type, the file size and the network state data, and estimating the transmission effect of the transmission fragmentation based on the network state data to obtain a fragmentation transmission score; transmitting the to-be-transmitted file based on the fragment transmission score; according to the file type of the to-be-transmitted file and the current network state, targeted fragmentation is performed on the file, so that the transmission quality and the transmission speed during file transmission are improved, and the transmission efficiency of network transmission is further ensured.

Description

Technical Field

[0001] This application belongs to the field of network transmission technology, and specifically relates to a high-concurrency file queue transmission system, method, and storage medium. Background Art

[0002] The network transmission of high-concurrency files refers to the ability to handle a large number of file upload or download requests simultaneously in a short period of time, and is widely used in scenarios such as cloud storage, online video, and enterprise data synchronization. Its implementation relies on an efficient network architecture and optimization techniques to ensure stability and performance under high load.

[0003] High-concurrency file transmission usually adopts a sharding and multi-threaded concurrency mechanism, that is, a large file is divided into multiple small pieces, and multiple threads or connections download or upload in parallel and are finally merged into a complete file. This method can significantly improve the download rate; however, the existing transmission sharding methods often set a fixed sharding size; since the network state changes in real time and the types of transmitted files are also diverse, it is difficult to adapt to different transmitted file types and transmission network states with a unified sharding strategy, resulting in a reduction in the transmission efficiency of network transmission; therefore, a high-concurrency file queue transmission system, method, and storage medium are needed. Summary of the Invention

[0004] This application aims to solve at least one of the technical problems existing in the prior art; for this purpose, this application proposes a high-concurrency file queue transmission system, method, and storage medium to solve the technical problem that the existing transmission sharding method is difficult to adapt to different transmitted file types and transmission network states, resulting in a reduction in the transmission efficiency of network transmission.

[0005] To achieve the above object, the first aspect of this application provides a high-concurrency file queue transmission method, including:

[0006] Obtain the file type and file size of the file to be transmitted, as well as the network state data of the current transmission network;

[0007] Perform transmission sharding on the file to be transmitted based on the file type, file size, and network state data, estimate the transmission effect of the transmission sharding based on the network state data to obtain a sharding transmission score; and perform transmission on the file to be transmitted based on the sharding transmission score.

[0008] This application obtains the file type and file size of the file to be transmitted, as well as the network status data of the current transmission network; performs transmission sharding on the file to be transmitted based on the file type, file size, and current network status data, estimates the transmission effect of the transmission shards based on the network status data to obtain a shard transmission score; transmits the file to be transmitted based on the shard transmission score; and performs targeted sharding on the file according to the file type of the file to be transmitted and the current network status, so as to improve the transmission quality and transmission speed during file transmission, thereby ensuring the transmission efficiency of network transmission.

[0009] Preferably, the performing transmission sharding on the file to be transmitted based on the file type, file size, and network status data includes:

[0010] Obtain the file type of the file to be transmitted, extract the network latency and network bandwidth in the network status data, and generate a shard size corresponding to the file type based on the network latency, network bandwidth, and file type;

[0011] When the file size is greater than the shard; then perform sharding on the file to be transmitted based on the shard size to obtain a number of transmission shards.

[0012] Preferably, the generating a shard size corresponding to the file type based on the network latency, network bandwidth, and file type includes:

[0013] Obtain the standard network latency and standard network bandwidth corresponding to the file type, as well as the standard shard size corresponding to the standard network latency and standard network bandwidth; mark the standard network latency, standard network bandwidth, and standard shard size as BWY, BWD, and BFY respectively; the file type is the form of the file to be transmitted, including text type, video type, program file, etc.; the standard shard size is the optimal shard size of the file corresponding to the file type being transmitted at this time determined by experts according to the standard network latency and standard network bandwidth;

[0014] Obtain the current network latency and network bandwidth, and mark them as WY and WD respectively;

[0015] Through the formula Calculate the shard size FY corresponding to the file type under the current network latency and network bandwidth;

[0016] Among them, ε and δ are the adjustment coefficients corresponding to the network latency and network bandwidth respectively; and 0 < ε ≤ 1 / e, 0 < δ < 1; which are respectively used to adjust the influence of the network latency and network bandwidth on the shard size; the specific values are set according to expert experience.

[0017] Preferably, the performing sharding on the file to be transmitted based on the shard size to obtain a number of transmission shards includes:

[0018] Obtain the file size of the transmission file, and slice the transmission file according to the slice size corresponding to the slice size to obtain a number of transmission slices.

[0019] Preferably, estimating the transmission effect of the transmission slice based on the network status data to obtain a slice transmission score includes:

[0020] Obtain the network latency in the network status data corresponding to a number of set time periods, and integrate the network latency into a network latency sequence in the order of collection time;

[0021] Judge whether the maximum value in the network latency sequence is less than the set network latency threshold; if so, obtain the variance of the network latency sequence. When the variance is less than the set variance threshold, set the slice transmission score to 1; otherwise, generate a slice transmission score based on the network latency sequence.

[0022] In this application, by judging whether the recent network is stable, if so, the generated transmission slices can be transmitted according to the current network status; otherwise, it means that the network status during transmission may be worse than the current network status and differ a lot. When transmitting large files or multi-slice files, the probability of mistransmission is relatively high. Therefore, it is necessary to re-slice.

[0023] In this application, first roughly judge whether the recent network is stable. When it is unstable, then make a detailed judgment on the network status, which can not only ensure the accuracy of network status judgment, but also save a large amount of unnecessary computing power.

[0024] Preferably, generating a slice transmission score based on the network latency sequence includes:

[0025] Equally divide the network latency in the network latency sequence into a set number of network latency sub-sequences, and number each network latency sub-sequence in the order of time; the network latency in the network latency sub-sequence is continuous in time, and the number of network latency in each network latency sub-sequence is the same;

[0026] Perform Fourier transform on the network latency sub-sequence to obtain a spectrogram, obtain the sum of energies corresponding to each frequency below the set frequency threshold in the spectrogram, calculate the ratio of the sum of energies to the total energy of the spectrogram, and perform weighted summation on the ratios corresponding to each delay sub-sequence to obtain the slice transmission score.

[0027] Preferably, the frequency threshold is set based on the slice size, including: obtaining the set maximum slice and its corresponding frequency threshold; calculating the ratio of the slice size to the maximum slice and marking it as the down-regulation coefficient; calculating the difference between the frequency threshold and the product of the down-regulation coefficient and the frequency threshold, and recording it as the current frequency threshold.

[0028] Preferably, transmitting the file to be transmitted based on the shard transmission score includes:

[0029] When the shard transmission score is 1, transmit according to the transmission shards;

[0030] When the shard transmission score is greater than the set score threshold and less than 1; obtain the maximum value in the delay sequence, regenerate the shard size corresponding to the file type according to the maximum value, network bandwidth, and file type; when the file size is greater than the shard; then shard the file to be transmitted based on the shard size; and transmit the shards.

[0031] The second aspect of the present application provides a high-concurrency file queue transmission system, including: a data acquisition module, a data analysis module, a data transmission module, and a database;

[0032] The data acquisition module: is used to acquire network status data, as well as the file type and file size of the file to be transmitted;

[0033] The data analysis module: shards the file to be transmitted based on the file type, file size, and network status data, and estimates the transmission effect of the transmission shards based on the network status data to obtain the shard transmission score;

[0034] The data transmission module: transmits the file to be transmitted based on the shard transmission score.

[0035] The third aspect of the present application provides a computer-readable storage medium, which includes a stored computer program. Wherein, when the computer program runs, it controls the device where the computer-readable storage medium is located to execute any one of the above-mentioned high-concurrency file queue transmission methods.

[0036] Compared with the prior art, the beneficial effects of the present application are:

[0037] 1. The present application obtains the file type and file size of the file to be transmitted, as well as the network status data of the current transmission network; shards the file to be transmitted based on the file type, file size, and current network status data, estimates the transmission effect of the transmission shards based on the network status data to obtain the shard transmission score; transmits the file to be transmitted based on the shard transmission score; shards the file specifically according to the file type of the file to be transmitted and the current network status, so as to improve the transmission quality and transmission speed during file transmission, and thus ensure the transmission efficiency of network transmission.

[0038] 2. The present application first roughly judges whether the network is stable recently, and when it is unstable, then makes a detailed judgment on the network status, which can not only ensure the accuracy of network status judgment, but also save a large amount of unnecessary calculation. Brief Description of the Drawings

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0040] Figure 1 It is a schematic diagram of the method steps of the high-concurrency file queue transmission method in the present application;

[0041] Figure 2 It is a schematic diagram of the module connection of the high-concurrency file queue transmission system in the present application. Detailed Embodiments

[0042] The following will clearly and completely describe the technical solutions of the present application in combination with the embodiments. Obviously, the described embodiments are only some embodiments of the present application, rather than all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0043] Please refer to Figure 1 , the first aspect of the present application provides a high-concurrency file queue transmission method, including:

[0044] Obtain the file type and file size of the file to be transmitted, as well as the network status data of the current transmission network; the file type includes text files, video types, program files, etc.; the file size is the size of the file to be transmitted, and the units are bytes, kilobytes, megabytes, etc. It can be understood that the units of the file size and the transmission shards need to be unified; for subsequent use and calculation; the network status data includes the network latency and network bandwidth of the current network;

[0045] Perform transmission sharding on the file to be transmitted based on the file type, file size, and network status data. Here, the transmission sharding is to segment the file to be transmitted according to the current network status data to obtain shards; estimate the transmission effect of the transmission shards based on the network status data to obtain a shard transmission score; the shard transmission score is the score of the transmission effect of the transmission shard during the transmission process. The more stable the network status, the better the corresponding transmission effect may be, and the higher the corresponding shard transmission score; transmit the file to be transmitted based on the shard transmission score.

[0046] In this embodiment, the file type and file size of the file to be transmitted are obtained, as well as the network status data of the current transmission network; the file to be transmitted is fragmented for transmission based on the file type, file size, and current network status data, and the transmission effect of the fragmented transmission is estimated based on the network status data to obtain a fragmented transmission score; the file to be transmitted is transmitted based on the fragmented transmission score; the file is fragmented specifically according to the file type of the file to be transmitted and the current network status, so as to improve the transmission quality and transmission speed during file transmission, thereby ensuring the transmission efficiency of network transmission.

[0047] Fragmenting the file to be transmitted based on the file type, file size, and network status data includes: obtaining the file type of the file to be transmitted, extracting the network latency and network bandwidth from the network status data, and generating the fragmentation size corresponding to the file type based on the network latency, network bandwidth, and file type; the fragmentation size is the size required to divide the file to be transmitted into fragmented transmissions.

[0048] When the file size is greater than the fragmentation size; then the file to be transmitted is fragmented based on the fragmentation size to obtain a number of fragmented transmissions.

[0049] Generating the fragmentation size corresponding to the file type based on the network latency, network bandwidth, and file type includes:

[0050] Obtaining the standard network latency and standard network bandwidth corresponding to the file type, as well as the standard fragmentation size corresponding to the standard network latency and standard network bandwidth; marking the standard network latency, standard network bandwidth, and standard fragmentation size as BWY, BWD, and BFY respectively; the file type is the form of the file to be transmitted, including text type, video type, program file, etc.; the standard fragmentation size is the optimal fragmentation size of the file corresponding to the file type being transmitted at this time determined by experts according to the standard network latency and standard network bandwidth.

[0051] Obtaining the current network latency and network bandwidth, and marking them as WY and WD respectively;

[0052] Through the formula Calculate the fragmentation size FY corresponding to the file type under the current network latency and network bandwidth;

[0053] Among them, ε and δ are the adjustment coefficients corresponding to the network latency and network bandwidth respectively; and 0 < ε ≤ 1 / e, 0 < δ < 1; which are used to adjust the influence of the network latency and network bandwidth on the fragmentation size; the specific values are set according to expert experience.

[0054] In this embodiment, ε = 1 / e, δ = 0.1. When the file to be transmitted is a text file, its corresponding standard network delay BWY = 50 ms, standard network bandwidth BWD = 1 Mbps, and standard shard size BFY = 10 KB; when the file to be transmitted is a program file, its corresponding standard network delay BWY = 200 ms, standard network bandwidth BWD = 100 Mbps, and standard shard size BFY = 512 KB;

[0055] When the network delay is low, setting smaller shards will increase the number of transmission shards, thereby increasing the overhead caused by establishing and closing connections; when the network delay is too high, setting larger shards will greatly increase the transmission duration in case of transmission failure. In this embodiment, the shard size corresponding to the file type under the current network delay and network bandwidth is calculated through the above formula to ensure a better transmission effect when transmitting each shard under the current network delay and network bandwidth as much as possible; when the network delay is low, appropriately increase the shard size to reduce the number of shards and the overhead caused by establishing and closing connections; when the network delay is high: appropriately reduce the shard size to reduce the retransmission cost; when the network bandwidth is high, a large amount of data can be transmitted quickly. In this embodiment, the shard size is set to be larger to give full play to the bandwidth advantage and improve the transmission efficiency; when the network bandwidth is low, this embodiment sets a smaller shard size to reduce the transmission duration of the shards and reduce the retransmission cost in case of transmission failure. By setting the shard size to be adaptively generated according to the current network environment, this embodiment improves the transmission efficiency of file transmission.

[0056] Then, the file to be transmitted is sharded based on the shard size to obtain a number of transmission shards, including: obtaining the file size of the transmission file, and sharding the file to be transmitted according to the shard size corresponding to the shard size to obtain a number of transmission shards.

[0057] Estimate the transmission effect of the transmission shards based on the network status data to obtain a shard transmission score, including: obtaining the network delay in the network status data corresponding to a number of set time periods, and integrating the network delays into a network delay sequence in the order of collection time;

[0058] Judge whether the maximum value in the network delay sequence is less than the set network delay threshold; if so, obtain the variance of the network delay sequence. When the variance is less than the set variance threshold, set the shard transmission score to 1; otherwise, generate a shard transmission score based on the network delay sequence.

[0059] In this embodiment, by determining whether the recent network is stable, if it is, the generated transmission shards can be transmitted according to the current network state; if not, it indicates that the network state during transmission may be worse than the current network state and the difference is significant. When transmitting large files or multi-shard files, the probability of mistransmission is relatively high. Therefore, re-sharding is required.

[0060] In this embodiment, first, a rough judgment is made on whether the recent network is stable. When it is unstable, a detailed judgment on the network state is then carried out, which can ensure the accuracy of network state judgment while saving a large amount of unnecessary computational effort.

[0061] Generating a shard transmission score based on the network delay sequence, including: equally dividing the network delays in the network delay sequence into a set number of network delay sub-sequences, and numbering each network delay sub-sequence in chronological order; the network delays in the network delay sub-sequence are continuous in time, and the number of network delays in each network delay sub-sequence is the same;

[0062] Performing a Fourier transform on the network delay sub-sequence to obtain a spectrogram, obtaining the sum of the energies corresponding to the frequencies lower than the set frequency threshold in the spectrogram, calculating the ratio of the sum of the energies to the total energy of the spectrogram, and performing a weighted sum of the ratios corresponding to each delay sub-sequence to obtain the shard transmission score; it should be noted that in this embodiment, the closer to the current actual delay sub-sequence, the greater the set weight;

[0063] The larger the shard transmission score indicates that the frequency distribution is more in the low-frequency part, which further indicates that the recent network delay changes are relatively stable; when the fluctuation degree of the network delay is small, it indicates that the current network is relatively stable and data transmission can be carried out; its corresponding shard transmission score is slightly larger; when the fluctuation degree of the network delay is large, it indicates that the current network is unstable and the probability of mistransmission during data transmission is relatively large; its corresponding shard transmission score is smaller; the network delay in this embodiment is the difference between the actual transmission time and the theoretical transmission time.

[0064] In this embodiment, through the judgment of the recent network state, it is determined whether the current network state is suitable for transmission. If it is suitable, transmission is carried out; if not, re-sharding is performed according to the highest recent delay to ensure the transmission efficiency.

[0065] The frequency threshold is set based on the shard size, including: obtaining the set maximum shard and its corresponding frequency threshold; calculating the ratio of the shard size to the maximum shard and marking it as the down-regulation coefficient; calculating the difference between the frequency threshold and the product of the down-regulation coefficient and the frequency threshold, and recording it as the current frequency threshold.

[0066] In this embodiment, the larger the transmission shard is, the higher the probability of garbled transmission during the transmission process, and the higher the requirement for the stability of network latency; correspondingly, the set frequency threshold is smaller.

[0067] Transmitting the file to be transmitted based on the shard transmission score, including: when the shard transmission score is 1, transmitting according to the transmission shards; when the shard transmission score is greater than the set score threshold and less than 1; obtaining the maximum value in the delay sequence, and regenerating the shard size corresponding to the file type according to the maximum value, network bandwidth, and file type; when the file size is greater than the shard; then sharding the file to be transmitted based on the shard size; and transmitting the shards.

[0068] Please refer to Figure 2 , the second aspect of the present application provides a high-concurrency file queue transmission system, including: a data acquisition module, a data analysis module, a data transmission module, and a database;

[0069] The data acquisition module: is used to collect network status data, as well as the file type and file size of the file to be transmitted;

[0070] The data analysis module: performs transmission sharding on the file to be transmitted based on the file type, file size, and network status data, and estimates the transmission effect of the transmission shards based on the network status data to obtain a shard transmission score;

[0071] The data transmission module: transmits the file to be transmitted based on the shard transmission score;

[0072] The database: is used to store the data generated by this system and the required data.

[0073] The third aspect of the present application provides a computer-readable storage medium, where the computer-readable storage medium includes a stored computer program, and wherein, when the computer program runs, it controls the device where the computer-readable storage medium is located to execute any one of the above-mentioned high-concurrency file queue transmission methods.

[0074] Some of the data in the above formula is the numerical value obtained by removing the dimension, and the formula is obtained by software simulation of a large amount of collected data to obtain a formula closest to the actual situation; the preset parameters and preset thresholds in the formula are set by those skilled in the art according to the actual situation or obtained by simulation of a large amount of data.

[0075] The working principle of the present application:

[0076] This application obtains the file type and file size of the file to be transmitted, as well as the network status data of the current transmission network; performs transmission fragmentation on the file to be transmitted based on the file type, file size, and current network status data, estimates the transmission effect of the transmission fragmentation based on the network status data to obtain a fragmentation transmission score; performs the transmission of the file to be transmitted based on the fragmentation transmission score; and performs targeted fragmentation on the file according to the file type of the file to be transmitted and the current network status, so as to improve the transmission quality and transmission speed during file transmission, thereby ensuring the transmission efficiency of network transmission.

[0077] The above embodiments are only used to illustrate the technical method of this application and not to limit it. Although this application has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical method of this application can be modified or equivalently replaced without departing from the spirit and scope of the technical method of this application.

Claims

1. A high-concurrency file queue transmission method, characterized in that Including: Obtain the file type and file size of the file to be transmitted, as well as the network status data of the current transmission network; Perform transmission fragmentation on the file to be transmitted based on the file type, file size, and network status data, estimate the transmission effect of the transmission fragmentation based on the network status data to obtain a fragmentation transmission score; and perform transmission on the file to be transmitted based on the fragmentation transmission score.

2. The high-concurrency file queue transmission method according to claim 1, characterized in that The performing transmission fragmentation on the file to be transmitted based on the file type, file size, and network status data includes: Obtain the file type of the file to be transmitted, extract the network latency and network bandwidth in the network status data, and generate a fragmentation size corresponding to the file type based on the network latency, network bandwidth, and file type; When the file size is greater than the fragmentation; then fragment the file to be transmitted based on the fragmentation size to obtain a number of transmission fragments.

3. The high-concurrency file queue transmission method according to claim 2, wherein The generating a fragmentation size corresponding to the file type based on the network latency, network bandwidth, and file type includes: Obtain the standard network latency and standard network bandwidth corresponding to the file type, as well as the standard fragmentation size corresponding to the standard network latency and standard network bandwidth; mark the standard network latency, standard network bandwidth, and standard fragmentation size as BWY, BWD, and BFY respectively; Obtain the current network latency and network bandwidth, and mark them as WY and WD respectively; Through the formula calculate the shard size FY corresponding to the file type under the current network latency and network bandwidth; Where ε and δ are the adjustment coefficients corresponding to the network latency and network bandwidth respectively; and 0 < ε ≤ 1 / e, 0 < δ < 1; which are used to adjust the influence of the network latency and network bandwidth on the fragmentation size; the specific values are set according to expert experience.

4. A high-concurrency file queue transmission method according to claim 2, characterized in that The then fragmenting the file to be transmitted based on the fragmentation size to obtain a number of transmission fragments includes: Obtain the file size of the transmission file, and fragment the file to be transmitted according to the fragmentation size corresponding to the fragmentation size to obtain a number of transmission fragments.

5. A high-concurrency file queue transmission method according to claim 1, characterized in that, The estimating the transmission effect of the transmission fragments based on the network status data to obtain a fragmentation transmission score includes: Obtain the network latency in the network status data corresponding to a number of set time periods, and integrate the network latency into a network latency sequence in the order of collection time; Judge whether the maximum value in the network latency sequence is less than the set network latency threshold; if so, obtain the variance of the network latency sequence, and when the variance is less than the set variance threshold, set the fragmentation transmission score to 1; otherwise, generate a fragmentation transmission score based on the network latency sequence.

6. A high-concurrency file queue transmission method according to claim 5, characterized in that, The generating a fragmentation transmission score based on the network latency sequence includes: Equally divide the network latency in the network latency sequence into a set number of network latency sub-sequences, and number each network latency sub-sequence in the order of time; the network latency in the network latency sub-sequence is continuous in time, and the number of network latency in each network latency sub-sequence is the same; Perform Fourier transform on the network latency sub-sequence to obtain a spectrogram, obtain the sum of the energies corresponding to the frequencies lower than the set frequency threshold in the spectrogram, calculate the ratio of the sum of the energies to the total energy of the spectrogram, and perform weighted summation on the ratios corresponding to each delay sub-sequence to obtain the fragmentation transmission score.

7. A high-concurrency file queue transmission method according to claim 6, characterized in that The frequency threshold is set based on the shard size, and includes: obtaining the set maximum shard and its corresponding frequency threshold; calculating the ratio of the shard size to the maximum shard and marking it as the down - adjustment coefficient; calculating the difference between the frequency threshold and the product of the down - adjustment coefficient and the frequency threshold, and recording it as the current frequency threshold.

8. A high-concurrency file queue transmission method according to claim 1, characterized in that The transmission of the file to be transmitted based on the shard transmission score includes: When the shard transmission score is 1, transmit according to the transmission shards. When the shard transmission score is greater than the set score threshold and less than 1; obtain the maximum value in the delay sequence, regenerate the shard size corresponding to the file type according to the maximum value, network bandwidth and file type; when the file size is greater than the shard; then shard the file to be transmitted based on the shard size; and transmit the shards.

9. A high-concurrency file queue transmission system runs based on the high-concurrency file queue transmission method described in any one of claims 1 to 8; characterized in that, It includes: A data acquisition module, a data analysis module, a data transmission module and a database; The data acquisition module: is used to acquire network status data, as well as the file type and file size of the file to be transmitted. The data analysis module: shards the file to be transmitted based on the file type, file size and network status data, and estimates the transmission effect of the transmission shards based on the network status data to obtain the shard transmission score. The data transmission module: transmits the file to be transmitted based on the shard transmission score.

10. A computer storage medium, characterized in that, The computer - readable storage medium includes a stored computer program, wherein when the computer program runs, it controls the device where the computer - readable storage medium is located to execute a high - concurrency file queue transmission method according to any one of claims 1 to 8.

Images