Key file data transmission concurrency increasing method and device and storage medium

Abstract

The invention discloses a key file data transmission concurrency increasing method and device and a storage medium, and the method comprises the steps: carrying out the fragmentation processing of a key file, sorting the obtained fragmented files, and putting the sorted fragmented files into a cache queue; randomly obtaining the number of fragment files from the cache queue, sequentially putting the fragment files into slots of a hash table, and then transmitting the fragment files out; and then calculating the average transmission rate of each transmission thread and the transmission time required by the fragment files to be sent in each slot position, and then obtaining the number of the fragment files from the residual fragment files in the cache queue and putting the fragment files into the corresponding slot positions until all the fragment files in the cache queue are transmitted. In the whole sending process, the size of the fragment file of the key file to be transmitted of each thread can be dynamically adjusted according to different transmission speeds of the threads, so that the situation that the waiting state of the corresponding client is too long is avoided, and the concurrency is improved on the premise that the overall throughput is basically kept unchanged.

Description

Technical Field

[0001] The present invention relates to the field of communication technologies, and in particular to a method, device and storage medium for increasing the concurrency of key file data transmission. Background Art

[0002] With the rapid development of technology, it has become the norm to use encrypted communication to transmit data. In the process of encrypted communication, the sender and the receiver will use their own key files to encrypt and decrypt the transmitted data, so as to achieve the effect of secure transmission. However, in the current key file transmission process, for the download key file provided by the key center, a full-volume exclusive mode is adopted, that is, the thread will not send another key file that is ready to be downloaded until the current key file is sent. In other words, the execution logic will not exit before the current key file is sent; when the sending is completed (regardless of success or failure), the execution logic selects the next key file to be sent; and so on, until the key file cache queue is empty.

[0003] However, because each cache queue uses a serial FIFO method to access key files, each key file must be processed before the next one can proceed; even for a single cache queue with multiple threads, a file message cycle can only respond to the same number of clients as the number of threads. The clients corresponding to the subsequent queued key files are in a waiting state. The sending of the key file in the thread has not been completed, and the clients corresponding to the queued key files are always in a waiting state. Once the time is too long, the communication link will timeout, and the quality of each communication link is different. Once the waiting state is too long, it will be stuck.

[0004] Even if the key files in the cache queue are allocated in advance according to the number of threads, the transmission speed of each thread is different, resulting in different speeds of key file transmission. Some threads still complete the transmission while the key files in some threads are still waiting for transmission. The corresponding clients are stuck due to the long waiting state. Summary of the invention

[0005] Purpose of the invention: The purpose of the present invention is to provide a method, device and storage medium for improving the concurrency of key file data transmission, which solves the problem that the existing key file download adopts a full-volume exclusive mode, resulting in the client corresponding to the queued key file being in a waiting state all the time, which is prone to communication link timeout and stuck in the waiting state for too long. The present invention first slices the key file, and then responds to the download of all key files during the execution of the sending process. Moreover, during the entire sending process, the file size of the key file slice to be transmitted by each thread can be dynamically adjusted according to the different transmission speeds of the threads, thereby ensuring that the corresponding client does not have a situation where the waiting state is too long. Under the premise that the overall throughput remains basically unchanged, the concurrency is improved and the bandwidth utilization remains unchanged.

[0006] Technical solution: The present invention provides a method for increasing the concurrency of key file data transmission, comprising the following steps:

[0007] (1) Slicing the first key file to the Nth key file to obtain slice files of each key file, and then sorting the slice files into a cache queue;

[0008] (2) creating a hash table, the hash table including the first to Mth slots, each slot including a plurality of nodes for storing the shard files;

[0009] (3) Randomly obtain a number of shard files from the cache queue and place them in the first to Mth slots in sequence. Each slot transmits the shard files in the node through its own transmission thread;

[0010] (4) Calculate the average transmission rate of each transmission thread within the preset time at every preset time, calculate the transmission time required for the slice files to be sent in each slot according to the average transmission rate of the transmission thread, obtain the slot corresponding to the shortest transmission time and record it as the newly added slice file receiving slot, and obtain the number of slice files sent by it within the preset time, and then obtain the same number of slice files as the number of slice files sent from the remaining slice files in the cache queue in order and put them into the newly added slice file receiving slot;

[0011] (5) Repeat step (4) multiple times until all the segmented files in the cache queue are transferred.

[0012] Furthermore, the step of sorting the shard files and placing them in the cache queue refers to:

[0013] First, L shard files are taken from the shard files of the first key file to the Nth key file respectively, and then the N×L shard files are sorted in the order of the key files and put into the cache queue; then the remaining shard files in the shard files of the first key file to the Nth key file are randomly arranged in a disorderly order and put into the cache queue.

[0014] Furthermore, the ordering of the shard files and placing them into the cache queue refers to: arranging all the shard files from the first key file to the Nth key file in random order and placing them into the cache queue.

[0015] Furthermore, the step of sorting the shard files and placing them in the cache queue refers to:

[0016] First, the first shard files are respectively taken from the shard files of the first key file to the Nth key file, and then they are sorted in sequence according to the order of the key files and put into the cache queue; the second shard files are respectively taken from the shard files of the first key file to the Nth key file, and then they are sorted in sequence according to the order of the key files and put into the cache queue; and so on, until the Lth shard file is taken from the shard files of the first key file to the Nth key file, and then they are sorted in sequence according to the order of the key files and put into the cache queue; finally, the remaining shard files in the shard files of the first key file to the Nth key file are randomly arranged in a disorderly order and put into the cache queue.

[0017] Furthermore, the specific process of calculating the average transmission rate of each transmission thread is as follows:

[0018] First, the size of the segmented files sent within a preset time is calculated, and the average transmission rate of the transmission thread within the preset time is obtained by dividing the size of the segmented files sent by the preset time.

[0019] Furthermore, the preset time is 1 to 5 seconds.

[0020] Furthermore, the specific process of the transmission time required for the fragmented file to be sent is:

[0021] First, the size of the fragmented file to be sent in each slot is calculated, and the transmission time required for the fragmented file to be sent is obtained by dividing the size of the fragmented file to be sent by the average transmission rate of the transmission thread.

[0022] The present invention also includes a computer device, which includes at least a processor and a memory, and the processor is used to implement the method for improving the concurrency of key file data transmission as described in any of the above claims when executing a computer program stored in the memory.

[0023] The present invention also includes a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method for improving the concurrency of key file data transmission as described in any of the above claims.

[0024] Beneficial effects of the present invention:

[0025] The present invention slices the key file, sorts the slice files and puts them into a cache queue; then randomly obtains the number of slice files from the cache queue and puts them into the slot node of the hash table, and finally transmits the slice files in the node through respective transmission threads; in the subsequent sending process, according to the average transmission rate of the transmission thread, the transmission time required for the slice files to be sent in each slot is calculated, and then the number of slice files is obtained from the remaining slice files in the cache queue and put into the corresponding slot until all the slice files in the cache queue are transmitted. The present invention can first respond to the download of all key files during the execution of the sending process, and in the whole sending process, can dynamically adjust the size of the key file slice files to be transmitted of each thread according to the different transmission speeds of the threads, thereby ensuring that the corresponding client does not have a situation where the waiting state is too long, and the concurrency is improved under the premise that the overall throughput remains basically unchanged. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a flow chart of a method for increasing the concurrency of key file data transmission according to the present invention;

[0027] Figure 2 A schematic diagram of sorting and placing slice files into a cache queue according to the present invention;

[0028] Figure 3 It is a schematic diagram of the method for improving the concurrency of key file data transmission of the present invention. DETAILED DESCRIPTION

[0029] The present invention will be further described below in conjunction with the accompanying drawings and embodiments:

[0030] The existing key file download adopts a full-volume exclusive mode, that is, the thread will not send another key file that is ready to be downloaded until the current key file is sent, and so on, until the key file cache queue is empty. For a single cache queue multi-threaded concurrency, only the same number of clients as the number of threads can be responded to in a file message cycle. The key file sending in the thread has not been completed, and the client corresponding to the subsequent queued key file is always in a waiting state. Once the time is too long, the communication link will time out, and the quality of each communication link is different. Once the waiting state is too long, it will be stuck. In view of this, the present application proposes a method for improving the concurrency of key file data transmission.

[0031] like Figure 1 As shown, the method for increasing the concurrency of key file data transmission of the present invention comprises the following steps:

[0032] (1) The first key file to the Nth key file are sliced ​​to obtain slice files of each key file, and then the slice files are sorted and put into a cache queue; in this embodiment, the first key file to the third key file are taken as an example, and 20 slice files of the first key file, 30 slice files of the second key file, and 20 slice files of the third key file are obtained through the slice processing;

[0033] Among them, sorting the slice files and putting them into the cache queue means: first, take the first slice file from the slice files of the first key file to the Nth key file respectively, and then sort them in sequence according to the order of the key files and put them into the cache queue; take the second slice file from the slice files of the first key file to the Nth key file respectively, and then sort them in sequence according to the order of the key files and put them into the cache queue; and so on, until taking the Lth slice file from the slice files of the first key file to the Nth key file respectively, and then sort them in sequence according to the order of the key files and put them into the cache queue, and the value of L can be the common divisor of the number of slice files from the first key file to the Nth key file; finally, randomly arrange the remaining slice files in the slice files of the first key file to the Nth key file and put them into the cache queue. Figure 2 As shown, for example, the first slice file is taken from the slice files of the first key file to the third key file respectively, and then they are sorted in sequence according to the order of the key files and put into the cache queue, and the order is the first slice file in the first key file, the first slice file in the second key file and the first slice file in the third key file; the second slice file is taken from the slice files of the first key file to the third key file respectively, and then they are sorted in sequence according to the order of the key files and put into the cache queue, and the order is the second slice file in the first key file, the second slice file in the second key file and the second slice file in the third key file; and so on, the value of L can be 10, until the tenth slice file is taken from the slice files of the first key file to the third key file respectively, and then they are sorted in sequence according to the order of the key files and put into the cache queue, and the order is the tenth slice file in the first key file, the tenth slice file in the second key file and the tenth slice file in the third key file; finally, the remaining slice files in the slice files of the first key file to the third key file are randomly arranged in disorder and put into the cache queue. In this way, the download of all key files can be responded to at the beginning of the sending process, and after the key files are fragmented, the fragmented files of the key files are transmitted quickly, so that the download of all key files can be responded to, reducing the waiting time of the corresponding client;

[0034] Of course, there are many ways to sort the slice files and put them into the cache queue, for example: first, take L slice files from the slice files of the first key file to the Nth key file, and then sort the N×L slice files in the order of the key files and put them into the cache queue; then randomly arrange the remaining slice files in the slice files of the first key file to the Nth key file and put them into the cache queue. For example, first take ten slice files from the slice files of the first key file to the third key file, and then sort the thirty slice files in the order of the key files and put them into the cache queue, the order is the first to the tenth slice files in the first key file, the first to the tenth slice files in the second key file, and the first to the tenth slice files in the third key file; finally, randomly arrange the remaining slice files in the slice files of the first key file to the third key file and put them into the cache queue. It can also be: directly randomly arrange all the slice files of the first key file to the Nth key file and put them into the cache queue. For example, all seventy shard files from the first key file to the third key file are directly randomly arranged in a shuffled order and put into the cache queue. Other ways of sorting the shard files and putting them into the cache queue are not listed here one by one. The above three methods can respond to the download of all key files in the process of just starting to execute the sending, thereby reducing the waiting time of the corresponding client.

[0035] (2) Creating a hash table, the hash table includes the first to Mth slots, and this embodiment uses the first to third slots as an example, and each slot includes multiple nodes for storing shard files;

[0036] (3) Figure 3 As shown, the number of shard files is randomly obtained from the cache queue and placed in the first to Mth slots in sequence, and each slot transmits the shard files in the node through its own transmission thread;

[0037] (4) Calculate the average transmission rate of each transmission thread within the preset time at every preset time, and the preset time can be 1 to 5 seconds; the specific process of calculating the average transmission rate is as follows: first, calculate the size of the segmented file sent within the preset time, and divide the size of the segmented file sent by the preset time to obtain the average transmission rate of the transmission thread within the preset time.

[0038] Then, the transmission time required for the slice files to be sent in each slot is calculated based on the average transmission rate of the transmission thread. The specific process is as follows: First, the size of the slice files to be sent in each slot is calculated, and the transmission time required for the slice files to be sent is obtained by dividing the size of the slice files to be sent by the average transmission rate of the transmission thread. In this way, the transmission time required for each slot at the current thread speed can be obtained, which is prepared for the subsequent distribution of slice files in the cache queue;

[0039] The slot corresponding to the shortest transmission time is obtained and recorded as the newly added shard file receiving slot, and the number of shard files sent within the preset time is obtained, and then the number of shard files that is the same as the number of sent shard files is obtained from the remaining shard files in the cache queue in order and put into the newly added shard file receiving slot;

[0040] (5) Finally, step (4) is repeated multiple times, that is, the subsequent distribution of the slice files in the cache queue is performed according to step (4) until all the slice files in the cache queue are transmitted. It can be seen that in step (3), each slot obtains a certain number of slice files and starts to transmit, and step (4) calculates the average transmission rate of the transmission thread and the transmission time required for the slice files to be sent in the slot within a preset time. The subsequent slice files are supplemented based on the transmission time required for the slice files to be sent in the slot, that is, the size of the slice files of the key file to be transmitted of each thread is dynamically adjusted, thereby ensuring that the corresponding client does not have a waiting state that is too long.

[0041] In summary, the present invention slices the key file, sorts the slice files and puts them into the cache queue; then randomly obtains the number of slice files from the cache queue and puts them into the slot node of the hash table, and finally transmits the slice files in the node through the respective transmission threads; in the subsequent sending process, according to the average transmission rate of the transmission thread, the transmission time required for the slice files to be sent in each slot is calculated, and then the number of slice files is obtained from the remaining slice files in the cache queue and put into the corresponding slot until all the slice files in the cache queue are transmitted. It can respond to the download of all key files during the execution of the sending process, and in the whole sending process, it can dynamically adjust the size of the slice files of the key files to be transmitted of each thread according to the different transmission speeds of the threads, so as to ensure that the corresponding client does not have a situation where the waiting state is too long, and the concurrency is improved under the premise that the overall throughput remains basically unchanged.

[0042] An embodiment of the present invention also provides a computer device, which includes at least a processor and a memory, the memory stores a computer program, and the processor is used to implement the steps of the method for increasing the concurrency of key file data transmission in an embodiment of the present invention when executing the computer program stored in the memory.

[0043] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for increasing the concurrency of key file data transmission in an embodiment of the present invention are implemented.

Claims

1. A method for increasing the concurrency of key file data transmission, characterized in that: The following steps are involved: (1) Slicing the first key file to the Nth key file to obtain slice files of each key file, and then sorting the slice files into a cache queue; (2) creating a hash table, the hash table including the first to Mth slots, each slot including a plurality of nodes for storing the shard files; (3) Randomly obtain a number of shard files from the cache queue and place them in the first to Mth slots in sequence. Each slot transmits the shard files in the node through its own transmission thread; (4) Calculate the average transmission rate of each transmission thread within the preset time at every preset time, calculate the transmission time required for the slice files to be sent in each slot according to the average transmission rate of the transmission thread, obtain the slot corresponding to the shortest transmission time and record it as the newly added slice file receiving slot, and obtain the number of slice files sent by it within the preset time, and then obtain the same number of slice files as the number of slice files sent from the remaining slice files in the cache queue in order and put them into the newly added slice file receiving slot; (5) Repeat step (4) multiple times until all the segmented files in the cache queue are transferred.

2. A method for increasing the concurrency of key file data transmission according to claim 1, characterized in that: The said sorting the fragment files and putting them into the cache queue means: First, L fragment files are taken from the fragment files of the first key file to the Nth key file respectively, and then the N×L fragment files are sorted in the order of the key files and put into the cache queue; Then, the remaining fragment files in the fragment files from the first key file to the Nth key file are randomly arranged in a disorderly order and put into a cache queue.

3. A method for increasing the concurrency of key file data transmission according to claim 1, characterized in that: The step of sorting the shard files and placing them in the cache queue refers to: arranging all shard files from the first key file to the Nth key file in random order and placing them in the cache queue.

4. A method for increasing the concurrency of key file data transmission according to claim 1, characterized in that: The said sorting the fragment files and putting them into the cache queue means: First, the first shard files are respectively taken from the shard files of the first key file to the Nth key file, and then they are sorted in sequence according to the order of the key files and put into the cache queue; the second shard files are respectively taken from the shard files of the first key file to the Nth key file, and then they are sorted in sequence according to the order of the key files and put into the cache queue; and so on, until the Lth shard file is taken from the shard files of the first key file to the Nth key file, and then they are sorted in sequence according to the order of the key files and put into the cache queue; finally, the remaining shard files in the shard files of the first key file to the Nth key file are randomly arranged in a disorderly order and put into the cache queue.

5. A method for increasing the concurrency of key file data transmission according to claim 1, characterized in that: The specific process of calculating the average transmission rate of each transmission thread is as follows: First, the size of the segmented files sent within a preset time is calculated, and the average transmission rate of the transmission thread within the preset time is obtained by dividing the size of the segmented files sent by the preset time.

6. A method for increasing the concurrency of key file data transmission according to claim 5, characterized in that: The preset time is 1 to 5 seconds.

7. A method for increasing the concurrency of key file data transmission according to claim 1, characterized in that: The specific process of the transmission time required for the fragmented file to be sent is: First, the size of the fragmented file to be sent in each slot is calculated, and the transmission time required for the fragmented file to be sent is obtained by dividing the size of the fragmented file to be sent by the average transmission rate of the transmission thread.

8. A computer device, characterized in that: The computer device comprises at least a processor and a memory, and the processor is used to implement the method for improving the concurrency of key file data transmission as described in any one of claims 1-7 when executing a computer program stored in the memory.

9. A computer-readable storage medium, characterized in that: It stores a computer program, which, when executed by a processor, implements the method for improving the concurrency of key file data transmission as described in any one of claims 1-7.

Images