File transmission method and device based on dynamic segmentation, medium and program product

Abstract

The embodiment of the invention discloses a file transmission method and device based on dynamic segmentation, a medium and a program product. The method comprises the following steps: when a memory space value occupied by a to-be-transmitted file is not less than a first preset value, segmenting the to-be-transmitted file into N file segments, and downloading the N file segments in parallel by adopting N communication sessions; when downloading of any communication session is completed, first memory occupancy of non-downloaded fragments of other communication sessions is obtained; judging whether the maximum first memory occupancy is greater than a second preset value; if yes, two new file fragments are obtained based on the non-downloaded fragment occupied by the maximum first memory, and then the new file fragments are downloaded through the original communication session and the idle communication session in parallel; and when the total data volume of the downloaded fragments is equal to the memory space value, combining all the downloaded fragments into a to-be-transmitted file and transmitting the to-be-transmitted file to a destination end. According to the method, the transmission efficiency of the large file can be effectively improved in a dynamic segmentation and communication session parallel processing mode, and the large file can be quickly transmitted.

Description

Technical Field

[0001] The present disclosure relates to the technical field of data transmission, and in particular to a file transmission method, device, medium, and program product based on dynamic segmentation. Background Art

[0002] With the increase in network security requirements, a physical isolation device is needed between different networks and different security domains to ensure the security of data transmission. The isolation switching system came into being, which allows data to be transmitted securely in two directions while maintaining physical isolation between networks. The isolation switching system includes a front-end machine, a physical isolation component, and a back-end machine. The front-end machine provides external services and is responsible for interacting with the file server at the source end to obtain the incremental change information to be synchronized; the physical isolation component ensures the physical isolation of data during transmission, ensures security, and is responsible for forwarding the incremental change information to the back-end machine; the back-end machine provides external services and is responsible for interacting with the file server at the destination end, sending the incremental change information to the file server at the destination end for synchronization.

[0003] Existing file transfer methods use a single connection for transmission when synchronizing large files and small files, which results in low efficiency when transmitting large files. In addition, when synchronizing large files, other communication sessions may be idle due to long-term occupancy, making it impossible to achieve efficient file transmission and full utilization of device resources. Summary of the invention

[0004] In view of this, the embodiments of the present disclosure provide a file transfer method, device, medium, and program product based on dynamic segmentation, which can effectively improve the efficiency of large file transmission through dynamic segmentation and parallel processing of communication sessions, and help to quickly complete the transmission of large files.

[0005] In a first aspect, an embodiment of the present disclosure provides a file transmission method based on dynamic segmentation, which adopts the following technical solution:

[0006] Get the file to be transferred;

[0007] Determine whether the memory space occupied by the file to be transferred is less than a first preset value;

[0008] If it is not less than the first preset value, dividing the file to be transmitted into N file segments;

[0009] Using N communication sessions to download the N file segments in parallel;

[0010] When the file segments transmitted by any one of the communication sessions are downloaded completely, obtaining the first memory occupancy of the segments that have not been downloaded by the other N-1 communication sessions;

[0011] Determine whether the maximum first memory usage is greater than a second preset value;

[0012] If it is greater than a second preset value, two new file segments are obtained based on the undownloaded segment with the largest first memory occupancy, one of the new file segments continues to be downloaded by the original communication session, and the other new file segment is downloaded by the idle communication session;

[0013] The total data volume of the downloaded segments is recorded, and when the total data volume is equal to the memory space value, all the downloaded segments are combined into a file to be transmitted and transmitted to the destination end.

[0014] Optionally, dividing the file to be transmitted into N file segments includes:

[0015] Get the maximum number of connections in the isolated switching system;

[0016] According to the maximum number of connections and the memory space value, a maximum number of divisions is obtained;

[0017] Determine whether the maximum number of connections is less than the maximum number of divisions;

[0018] If yes, then the value of N is the maximum number of connections;

[0019] If not, the value of N is the maximum number of divisions;

[0020] The file to be transmitted is divided according to the value of N.

[0021] Optionally, the maximum number of divisions is calculated as follows:

[0022]

[0023] Wherein, N' is the maximum number of segments; min(·) is the minimum value function; max(·) is the maximum value function; M is the maximum number of connections; P is a preset constant; S is the memory space occupied by the file to be transferred; T is the preset initial segmentation threshold; is the ceiling function.

[0024] Optionally, the calculation formula of the second preset value is:

[0025] L = T / 2M;

[0026] Wherein, L is the second preset value, T is the preset initial segmentation threshold; and M is the maximum number of connections of the isolation switching system.

[0027] Optionally, it also includes:

[0028] Get the current number of communication sessions and the maximum number of connections in the isolated switching system;

[0029] When the current number of communication sessions is less than the maximum number of connections, obtaining the current overall transmission speed of the file to be transmitted and the maximum file transmission speed of the isolation switching system;

[0030] Based on the maximum file transfer speed, obtaining a speed threshold;

[0031] When the current overall transmission speed is less than the speed threshold, obtaining a second memory occupancy of all undownloaded segments of the communication session;

[0032] When the maximum second memory occupancy is greater than a second preset value, adding a new communication session;

[0033] Two new file segments are obtained based on the undownloaded segment with the second largest memory occupancy, one of the new file segments continues to be downloaded by the original communication session, and the other new file segment is downloaded by the new communication session.

[0034] Optionally, the speed threshold is calculated as follows:

[0035] YY+242357P

[0036] SY = SM * BF;

[0037] Wherein, SY is the speed threshold; SM is the maximum file transfer speed; and BF is a preset percentage, such as 80%.

[0038] Optionally, combining all the downloaded segments into a file to be transmitted and transmitting the file to the destination end includes:

[0039] Each time a file segment is obtained, the identifier and the start and end positions of the file segment are recorded;

[0040] Based on the identifier or the start and end positions, all downloaded segments are spliced ​​to generate a file to be transmitted;

[0041] The file to be transmitted is transmitted to the destination end.

[0042] Optionally, the file to be transmitted is a file to be synchronized; and the maximum number of connections of the isolation switching system is the maximum number of synchronization connections.

[0043] In a second aspect, the embodiment of the present disclosure further provides a file transmission system based on dynamic segmentation, which adopts the following technical solution:

[0044] A file acquisition module, used to acquire files to be transferred;

[0045] A first judging module, configured to judge whether the memory space occupied by the file to be transferred is less than a first preset value; if not, executing a segmentation module;

[0046] A segmentation module, used for segmenting the file to be transmitted into N file segments;

[0047] A parallel downloading module, configured to download the N file segments in parallel using N communication sessions;

[0048] A memory acquisition module, configured to acquire first memory occupancy of undownloaded segments of other N-1 communication sessions when the download of the file segments transmitted by any one of the communication sessions is completed;

[0049] The second judgment module is used to judge whether the maximum first memory usage is greater than the second preset value; YY+242357P

[0050] If yes, the fragment acquisition module is executed;

[0051] A segment acquisition module, used for acquiring two new file segments based on the undownloaded segment with the largest first memory occupancy, downloading one of the new file segments through the original communication session, and downloading the other new file segment through the idle communication session;

[0052] The segment combination module is used to record the total data volume of the downloaded segments, and when the total data volume is equal to the memory space value, combine all the downloaded segments into a file to be transmitted and transmit it to the destination.

[0053] In a third aspect, the embodiments of the present disclosure further provide a computer device, which adopts the following technical solution:

[0054] The computer device comprises:

[0055] at least one processor; and,

[0056] a memory communicatively connected to the at least one processor; wherein,

[0057] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any of the above-mentioned file transmission methods based on dynamic segmentation.

[0058] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to execute any of the above-mentioned file transfer methods based on dynamic segmentation.

[0059] In a fifth aspect, an embodiment of the present disclosure further provides a computer program product, including a computer program / instruction, which implements the steps of any of the above methods when executed by a processor.

[0060] The file transmission method based on dynamic segmentation provided by the embodiment of the present disclosure monitors the file changes at the source end, and immediately obtains the file to be transmitted when an update is detected, thereby reducing the transmission delay and ensuring the timeliness of the data. For large files, i.e., files to be transmitted whose memory space value is greater than or equal to the first preset value, the direct download speed is slow. By dividing the large file into multiple file fragments and using multiple communication sessions to download these fragments in parallel, the download speed can be significantly improved, ensuring that the system will not exhaust the memory and network due to a single large file, so as not to affect the execution of other services by the system. When a communication session is downloaded, the memory occupancy of the undownloaded fragments of other sessions is automatically checked. If the remaining undownloaded part of a communication session is too large, the remaining undownloaded part will be further divided to form two smaller new file fragments, one of which is continued to be downloaded by the original communication session, and the other is allocated to the idle communication session for download. This dynamic adjustment realizes the reuse of idle communication sessions, ensures that all communication sessions can maintain high utilization, and avoids the situation where some sessions are overloaded while other sessions are idle. When the total data volume of the downloaded segments reaches the memory space value of the original file to be transferred, it means that the download has been completed. All downloaded segments are combined into a complete file, i.e., the file to be transferred, and then transmitted to the destination to reduce the overall delay. In summary, this method improves the efficiency of large file transfer by logically segmenting large files and then using the breakpoint resume function for concurrent transmission. Especially in the field of file synchronization, this method further dynamically segments files for transmission by multiplexing idle communication sessions. During the transmission process, all communication sessions are kept busy as much as possible to make full use of device resources and solve the problem of slow synchronization of large files.

[0061] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the following preferred embodiments are specifically cited and described in detail with reference to the accompanying drawings.

[0062] YY+242357P BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0064] Figure 1 A schematic diagram of a process of a file transmission method based on dynamic segmentation provided in an embodiment of the present disclosure;

[0065] Figure 2A flowchart of a method for segmenting files to be transmitted provided in an embodiment of the present disclosure;

[0066] Figure 3 A flowchart of a method for adding a new communication session provided in an embodiment of the present disclosure;

[0067] Figure 4 A principle block diagram of a file transmission system based on dynamic segmentation provided by an embodiment of the present disclosure;

[0068] Figure 5 A schematic diagram of the structure of a computer device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0069] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0070] It should be clear that the following embodiments of the present disclosure are described by specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of them. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present disclosure, the ordinary technicians in this field can obtain the invention without creative work.

[0071] All other embodiments obtained are within the scope of protection of this disclosure.

[0072] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.

[0073] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The drawings only show components related to the present disclosure rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0074] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.

[0075] Reference Figure 1 The present disclosure provides a file transmission method based on dynamic segmentation, comprising the following steps:

[0076] S1: Get the file to be transferred;

[0077] S2: Determine whether the memory space occupied by the file to be transferred is less than a first preset value; if so, execute S3; if not, execute S4;

[0078] S3: Use a communication session to download the file to be transferred, and send the downloaded file to be transferred to YY+242357P

[0079] Transmit to the destination;

[0080] S4: dividing the file to be transmitted into N file segments;

[0081] S5: Using N communication sessions to download N file segments in parallel;

[0082] S6: When the file segments transmitted by any one of the communication sessions are downloaded completely, obtaining the first memory occupancy of the segments that have not been downloaded by the other N-1 communication sessions;

[0083] S7: Determine whether the maximum first memory usage is greater than the second preset value; if so, execute S8; if not, execute S9;

[0084] S8: acquiring two new file segments based on the undownloaded segment with the largest first memory usage, downloading one of the new file segments through the original communication session, and downloading the other new file segment through the idle communication session; returning to S6;

[0085] S9: close the idle communication session;

[0086] S10: Record the total data volume of the downloaded segments. When the total data volume is equal to the memory space value, combine all the downloaded segments into a file to be transmitted and transmit it to the destination.

[0087] The file transmission method based on dynamic segmentation provided by the present disclosure monitors the file changes at the source end, and immediately obtains the file to be transmitted when an update is detected, thereby reducing the transmission delay and ensuring the timeliness of the data. For large files, i.e., files to be transmitted whose memory space value is greater than or equal to the first preset value, the direct download speed is slow. By dividing the large file into multiple file fragments and using multiple communication sessions to download these fragments in parallel, the download speed can be significantly improved, ensuring that the system will not exhaust the memory and network due to a single large file, so as not to affect the execution of other services by the system. When a communication session is downloaded, the memory occupancy of the undownloaded fragments of other sessions is automatically checked. If the remaining undownloaded part of a communication session is too large, the remaining undownloaded part will be further divided to form two smaller new file fragments, one of which is continued to be downloaded by the original communication session, and the other is allocated to the idle communication session for download. This dynamic adjustment realizes the reuse of idle communication sessions, ensures that all communication sessions can maintain high utilization, and avoids the situation where some sessions are overloaded while other sessions are idle. When the total data volume of the downloaded segments reaches the memory space value of the original file to be transmitted, it means that the download has been completed. All downloaded segments are combined into a complete file, namely the file to be transmitted, and then transmitted to the destination to reduce the overall delay.

[0088] In summary, this method improves the efficiency of large file transfer by logically segmenting large files and then using the breakpoint resume function for concurrent transmission. Especially in the field of file synchronization, this method further dynamically segments files for transmission by reusing idle communication sessions. During the transmission process, all communication sessions are kept busy as much as possible, making full use of device resources and solving the problem of slow synchronization of large files.

[0089] In S1, an isolated exchange system is used to periodically scan the file server at the source end, and the file information on the file server, such as the file name, modification time, file size, etc., is recorded. By comparing the results of two adjacent scans, it is determined whether there is a file change at the source end. When it is determined that a file change has occurred, all the changed parts are collectively referred to as files to be transmitted. Note that here, it is not distinguished how many files are involved, but all incremental changes are regarded as a whole for transmission processing.

[0090] According to the business process or by parsing the destination information provided by the source end, the specific location where the file to be transferred needs to be transferred is obtained. This location is called the target storage path.

[0091] In S2-S5, when the file to be transmitted is too small, segmented transmission will reduce the transmission efficiency. Therefore, the present application sets a first preset value. Only when the memory space occupied by the file to be transmitted is greater than or equal to the first preset value, the file to be transmitted is dynamically segmented and transmitted; if the memory space occupied by the file to be transmitted is less than the first preset value, the file to be transmitted is not segmented, a communication session is created, and the communication session is used to download the file to be transmitted from the source end, an empty temporary folder is created in advance, and a temporary file is created in the temporary folder. The downloaded file to be transmitted is stored in the temporary file. There is no need to assemble the files, and the temporary file is directly moved from the temporary folder to the target storage path.

[0092] For files to be transmitted whose memory space value is greater than or equal to the first preset value, they are divided into N equal parts to obtain N file fragments, and a unique identifier is provided for each file fragment to distinguish these file fragments. The number of parts N here affects the overall download efficiency and the subsequent file assembly efficiency. If the value of N is too small, each file fragment obtained by segmentation is still too large and cannot be effectively managed and transmitted. If the value of N is too large, each file fragment obtained by segmentation is too small, and the number of file fragments is too large, so more communication sessions are required to process these file fragments in parallel. Using too many communication sessions to transmit multiple file fragments with very small memory space values ​​will not only cause a waste of system resources, but also reversely reduce the overall download efficiency and subsequent file assembly efficiency. Therefore, the file transmission method based on dynamic segmentation designs a specific scheme, which comprehensively considers the size of the file to be transmitted and the maximum number of connections allowed by the current isolation switching system (i.e., the maximum number of communication sessions), obtains a reasonable value of N, and then uses fewer communication sessions in the initial stage of transmission to achieve optimal work efficiency. Reference Figure 2 The flowchart of the method for splitting the files to be transmitted is shown. The specific scheme for obtaining a reasonable value of N includes the following steps:

[0093] S41: Obtain the maximum number of connections of the isolated switching system;

[0094] S42: Obtain the maximum number of splits according to the maximum number of connections and the memory space value;

[0095] S43: Determine whether the maximum number of connections is less than the maximum number of partitions; if so, execute S44; if not, execute S45;

[0096] S44: The value of N is the maximum number of connections;

[0097] S45: The value of N is the maximum number of divisions;

[0098] S46: Split the file to be transmitted according to the value of N.

[0099] Through the above method, while ensuring that the maximum number of connections is not exceeded, the files to be transmitted are divided as much as possible to utilize more communication sessions for parallel downloading, thereby greatly shortening the download time.

[0100] The calculation formula for the maximum number of divisions is as follows:

[0101]

[0102] In formula 1, N' is the maximum number of divisions; min(·) is a function for finding the minimum value; max(·) is a function for finding the maximum value; M is the maximum number of connections; P is a preset constant. The value of P can be adjusted according to the user's needs for different business scenarios. For example, P can be set to 2, so that the value of the initial number of divisions N is usually smaller. For another example, P can be set to 4, so that the value of the initial number of divisions N is usually larger; S is the memory space occupied by the file to be transferred; T is the preset initial segmentation threshold; is the ceiling function.

[0103] Optionally, the first preset value is greater than or equal to the initial segmentation threshold, and such a setting can avoid segmenting small files. For example, the initial segmentation threshold is 20MB, the first preset value is 0.001MB, and the memory space value of the file to be transmitted is only 0.0011MB. Since the memory space value of the file to be transmitted is greater than the first preset value, the too small file to be transmitted will be segmented and transmitted, which may affect the transmission efficiency. Therefore, by setting the first preset value to be greater than or equal to the initial segmentation threshold, this problem can be effectively solved.

[0104] After determining the value of N, N communication sessions are created, each of which is responsible for downloading a file segment from the source. This parallel processing of multiple communication sessions can greatly improve the overall download efficiency and reduce the download time.

[0105] In S6, when any one of the N communication sessions completes the download task of the current file segment, it indicates that the communication session is in an idle state. At this time, other communication sessions are detected, and the part of the content that has been downloaded is called the downloaded segment, and the part of the content that has not been downloaded is called the undownloaded segment. The memory occupancy of the undownloaded segment is obtained. In order to facilitate subsequent distinction, the memory occupancy obtained at this time is called the first memory occupancy.

[0106] In S7-S9, the first memory occupancy of the undownloaded segments of other communication sessions is sorted, and the largest first memory occupancy is selected. If the largest first memory occupancy is less than or equal to the second preset value, it indicates that the other communication sessions have almost completed the download of their respective file segments, which means that the download of the entire file to be transmitted is also nearly completed. In this case, it is not meaningful to continue to split the file. At this time, the idle communication session is closed, and the other communication sessions are waited for to complete the download. Once the file segments are downloaded, the corresponding communication session should also be closed.

[0107] If the maximum first memory occupancy is greater than the second preset value, the undownloaded segment of the maximum first memory occupancy is equally divided into two parts, the first half is combined with the original file segment into a new file segment and continues to be allocated to the original communication session for downloading, and the second half is used as another new file segment and allocated to the idle communication session for downloading. This method of continuing to divide the undownloaded segment realizes the function of breakpoint resume, allowing the communication session that has completed downloading to help download the slower communication session, realizing the reuse of the communication session, keeping all communication sessions busy as much as possible, so as to make more full use of network bandwidth and server resources, avoid resource waste, and improve the overall download efficiency.

[0108] Furthermore, if a communication session encounters a problem causing transmission interruption, other communication sessions can still continue to download other segments, reducing the delay or failure of the entire download task caused by a single session failure and improving fault tolerance.

[0109] The purpose of setting the second preset value is to prevent unlimited segmentation of the undownloaded segments and avoid extra overhead caused by excessive segmentation and frequent switching of communication sessions. When determining the second preset value, the maximum number of connections is taken into account to ensure that system resources are reasonably allocated and utilized, avoid resource waste and system instability caused by too many communication sessions, and ensure that such file transfer work does not affect other business tasks of the system. The calculation formula of the second preset value is as follows:

[0110] L = T / 2M; (Formula 2)

[0111] In Formula 2, L is a second preset value.

[0112] Furthermore, the current number of communication sessions is obtained in real time. If the current number of communication sessions is equal to the maximum number of connections, there is no need to create a new communication session; if the current number of communication sessions is less than the maximum number of connections, whether to add a new communication session will be considered based on the current download details. Figure 3 The flowchart of the method for adding a new communication session is shown, and the conditions for adding a new communication session and the method for using a new communication session include the following steps:

[0113] S11: obtaining the current overall transmission speed of the file to be transmitted and the maximum file transmission speed of the isolated switching system;

[0114] S12: obtaining a speed threshold based on the maximum file transfer speed;

[0115] S13: Determine whether the current overall transmission speed is less than the speed threshold; if so, execute S14; if not, execute S15;

[0116] S14: Obtain the second memory usage of all undownloaded segments of the communication session; execute S16;

[0117] S15: Determine that there is no need to add a new communication session temporarily, return to S11, and update the file to be transmitted YY+242357P

[0118] The current overall transfer speed of the file;

[0119] S16: Determine whether the maximum second memory usage is greater than the second preset value; if so, execute S17; if not, execute S19;

[0120] S17: acquiring two new file segments based on the undownloaded segment with the second largest memory usage;

[0121] S18: continue to download one of the new file segments through the original communication session, and download the other new file segment through the new communication session;

[0122] S19: It is determined that there is no need to add a new communication session.

[0123] In S11, a file segment information record table is created, which is used to record the identifier, start and end positions (i.e., the starting position and the end position) of each current file segment, and the data volume of the downloaded segment (i.e., the data volume of the downloaded portion of the file segment). On this basis, the current single-unit transmission speed, the current single-unit transmission progress, and the current overall transmission speed and the current overall transmission progress of the file to be transmitted are calculated. Among them, the current single-unit transmission speed refers to the transmission speed of the file segment itself; the current single-unit transmission progress refers to the download progress of the file segment itself; the current overall transmission speed refers to the transmission speed of the file to be transmitted, which is equal to the sum of the current single-unit transmission speeds of all current file segments; the current overall transmission progress refers to the download progress of the file to be transmitted, which is equal to the ratio between the total data volume of all currently downloaded segments and the memory space value occupied by the file to be transmitted.

[0124] In addition, when a communication session encounters a problem causing transmission interruption, the file segment information record table can be used to quickly retransmit the file segments.

[0125] In S12, the speed threshold is calculated as follows:

[0126] SY=SM*BF; (Formula 3)

[0127] In formula 3, SY is the speed threshold; SM is the maximum file transfer speed; BF is the estimated YY+242357P

[0128] Set a percentage, such as 80%.

[0129] In S13-S19, when the current overall transmission speed is greater than or equal to the speed threshold, it means that the current network connection has been fully utilized, the network bandwidth occupancy rate is high, and the expected download efficiency can be achieved or exceeded. This further indicates that the currently configured number of communication sessions is reasonable under the current conditions, and there is no need to add new communication sessions for the time being. Continue to detect the current single transmission speed of each file fragment to update the current overall transmission speed.

[0130] When the current overall transmission speed is less than the speed threshold, it means that the current overall transmission speed can be further improved. At this time, the memory occupancy of all the undownloaded segments of the current communication session is obtained. In order to distinguish it from the above, the memory occupancy obtained at this time is called the second memory occupancy. All the currently obtained second memory occupancy are sorted, and the largest second memory occupancy is selected.

[0131] When the maximum second memory occupancy is greater than the second preset value, it indicates that more file segments are currently supported, and the undownloaded segments with the maximum second memory occupancy are equally divided into two parts, the first half is combined with the original file segment into a new file segment and continues to be allocated to the original communication session for download, and the second half is used as another new file segment and allocated to the new communication session for download, so as to further improve the transmission efficiency. When the maximum first memory occupancy is less than or equal to the second preset value, it indicates that each current communication session has completed the download of its own file segment or has almost completed the download of its own file segment, which means that the download of the entire file to be transmitted is nearly completed. In this case, there is no need to add a new communication session.

[0132] This method of dynamically adding new communication sessions can further fully utilize system resources and maximize transmission efficiency.

[0133] In S10, each time a file segment is added and downloaded, a corresponding temporary file is created in the temporary folder, and the downloaded segment is stored in the temporary file.

[0134] The total data volume of all currently downloaded segments is calculated through the data recorded in the file segment information record table. When the total data volume reaches the memory space value occupied by the file to be transferred, that is, when the current overall transfer progress of the file to be transferred is 100%, it means that the file to be transferred has been downloaded. At this time, the start and end positions of the file segment are the start and end positions of the downloaded segment. According to the identifier or start and end position recorded in the file segment information, all temporary files in the temporary folder are spliced ​​to generate the file to be transferred, and the file to be transferred is transferred to the target storage path to complete the file transfer from the source end to the destination end.

[0135] The present disclosure also provides a specific example:

[0136] The method is applied to the field of file synchronization. At this time, the file to be transmitted is the file to be synchronized, and the maximum number of connections of the isolation exchange system is the maximum number of synchronization connections. The maximum number of divisions N' is calculated to be 4. When the maximum number of synchronization connections is greater than or equal to 4, the file to be synchronized is initially divided into 4 file segments; when the maximum number of synchronization connections is less than 4, for example, the maximum number of synchronization connections is 3, the file to be synchronized is initially divided into 3 file segments.

[0137] The initial size of the file to be synchronized, i.e., the memory space occupied, is 40MB. Taking the initial segmentation into 4 file segments as an example, each file segment is assigned an identifier, namely A, B, C, and D. File segment A refers to the [0MB, 10MB] part of the file to be synchronized, file segment B refers to the (10MB, 20MB] part of the file to be synchronized, file segment C refers to the (20MB, 30MB] part of the file to be synchronized, and file segment D refers to the (30MB, 40MB] part of the file to be synchronized. Four communication sessions are initially created, and each file segment uses one communication session to transmit data.

[0138] Taking the FTP server as an example, file fragment A sends a RETR download command through control communication session 1 to download the corresponding data from the starting position of the file to be synchronized. File fragments B, C, and D respectively send REST retransmission commands through control communication sessions 2, 3, and 4 to inform the FTP server to resume transmission from the corresponding position of the file to be synchronized, and then download the corresponding data through the RETR download command to realize the function of breakpoint resumption.

[0139] The client monitors the download status of each file segment at the receiving end. When file segment A is downloaded but file segments B, C, and D are still being downloaded, the data connection of communication session 1 is disconnected and the control connection of communication session 1 is reused to help other segments with slower download speeds to transmit. Through detection, it is determined that the first memory occupancy of the undownloaded segment of file segment D is the largest, and the undownloaded segment of file segment D is split from the middle position, and communication session 1 is used to start downloading from the middle position of the undownloaded segment of file segment D. For example, if the remaining (35MB, 40MB] of file segment D has not been downloaded, the undownloaded segment is segmented, and the first half of the segment is classified as file segment D, which is continued to be downloaded by communication session 4. The second half of the segment is assigned the identifier D.1 and is downloaded by communication session 1. The data range of file segment D is changed to (30MB, 37.5MB], and the data range of file segment D.1 is (37.5MB, 40MB]. Similarly, when file segment D is segmented again, a new identifier D.2 is added, and when file segment D.1 is segmented, a new identifier D.1.1 is added. Assuming that the segmentation is performed only in this way and after all file segments are downloaded, they can be spliced ​​according to the identifiers of the file segments, and the splicing order is D, D.2, D.1, D.1.1. They can also be spliced ​​according to the start and end positions of the file segments to form the file to be transmitted, and then transmitted to the destination.

[0140] Reference Figure 4 The present disclosure provides a file transmission system based on dynamic segmentation, comprising:

[0141] The file acquisition module 101 is used to acquire the file to be transmitted;

[0142] The first judgment module 102 is used to judge whether the memory space occupied by the file to be transferred is less than YY+242357P

[0143] at a first preset value; if not, executing a segment segmentation module;

[0144] The segmentation module 103 is used to segment the file to be transmitted into N file segments;

[0145] A parallel downloading module 104, configured to download N file segments in parallel using N communication sessions;

[0146] The memory acquisition module 105 is used to acquire the first memory occupancy of the undownloaded segments of the other N-1 communication sessions when the file segments transmitted by any one of the communication sessions are downloaded completely;

[0147] The second judgment module 106 is used to judge whether the maximum first memory occupancy is greater than a second preset value; if so, execute the fragment acquisition module;

[0148] The segment acquisition module 107 is used to acquire two new file segments based on the undownloaded segment with the largest first memory occupancy, and continue to download one of the new file segments through the original communication session, and download the other new file segment through the idle communication session;

[0149] The segment combining module 108 is used to record the total data volume of the downloaded segments. When the total data volume is equal to the memory space value, all the downloaded segments are combined into a file to be transmitted and transmitted to the destination.

[0150] The various variations and specific examples of the dynamic segmentation-based file transfer method provided above are also applicable to the dynamic segmentation-based file transfer system provided in the present disclosure. Through the above detailed description of the dynamic segmentation-based file transfer method, those skilled in the art can clearly know the implementation method of the dynamic segmentation-based file transfer system. For the sake of brevity of the specification, it will not be described in detail here.

[0151] The computer device according to the embodiment of the present disclosure includes a memory and a processor. The memory is used to store non-temporary computer-readable instructions. Specifically, the memory may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may, for example, include a read-only memory (ROM), a hard disk, a flash memory, etc.

[0152] The processor may be a central processing unit (CPU) or other forms of processing units with data processing capabilities and / or instruction execution capabilities, and may control other components in the computer device to perform desired functions. In one embodiment of the present disclosure, the processor is used to run the computer-readable instructions stored in the memory, so that the computer device performs all or part of the steps of the file transmission method based on dynamic segmentation in the aforementioned embodiments of the present disclosure.

[0153] Those skilled in the art should be able to understand that in order to solve the technical problem of how to obtain a good user experience, the present embodiment may also include well-known structures such as a communication bus and an interface, and these well-known structures should also be included in the protection scope of the present disclosure.

[0154] like Figure 5 A schematic diagram of the structure of a computer device provided in an embodiment of the present disclosure is shown, which is a schematic diagram of the structure of a computer device suitable for implementing the embodiment of the present disclosure. Figure 5 The computer device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0155] like Figure 5 As shown, the computer device may include a processor (such as a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) or a program loaded from a storage device to a random access memory (RAM). In the RAM, various programs and data required for the operation of the computer device are also stored. The processor, ROM, and RAM are connected to each other through a bus. Input / output (I / O) YY+242357P

[0156] The interface is also connected to the bus.

[0157] Typically, the following devices can be connected to the I / O interface: input devices such as sensors or visual information acquisition devices; output devices such as display screens; storage devices such as tapes, hard disks, etc.; and communication devices. The communication device can allow the computer device to communicate with other devices (such as edge computing devices) wirelessly or by wire to exchange data. Figure 5 A computer device having various devices is shown, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0158] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processor, all or part of the steps of the file transfer method based on dynamic segmentation of an embodiment of the present disclosure are executed.

[0159] For detailed description of this embodiment, reference may be made to the corresponding descriptions in the aforementioned embodiments, which will not be repeated here.

[0160] The computer-readable storage medium according to the embodiment of the present disclosure stores non-transitory computer-readable instructions, and when the non-transitory computer-readable instructions are executed by a processor, all or part of the steps of the file transmission method based on dynamic segmentation in the above-mentioned embodiments of the present disclosure are executed.

[0161] YY+242357P

[0162] The above-mentioned computer-readable storage media include, but are not limited to: optical storage media (e.g., CD-ROM and DVD), magneto-optical storage media (e.g., MO), magnetic storage media (e.g., magnetic tape or mobile hard disk), media with built-in rewritable non-volatile memory (e.g., memory card) and media with built-in ROM (e.g., ROM box).

[0163] For detailed description of this embodiment, reference may be made to the corresponding descriptions in the aforementioned embodiments, which will not be repeated here.

[0164] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.

[0165] In the present disclosure, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The block diagrams of the devices, apparatuses, equipment, and systems involved in the present disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, meaning "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them unless the context clearly indicates otherwise. The word "such as" used here refers to the phrase "such as but not limited to", and can be YY+242357P

[0166] Used interchangeably with.

[0167] Additionally, as used herein, "or" used in a list of items beginning with "at least one" indicates a separate list, so that, for example, a list of "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not mean that the example described is preferred or better than other examples.

[0168] It should also be noted that in the system and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0169] Various changes, substitutions, and modifications of the techniques described herein may be made without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of the present disclosure is not limited to the specific aspects of the processes, machines, manufactures, compositions of events, means, methods, and actions described above. Currently existing or later to be developed processes, machines, manufactures, compositions of events, means, methods, or actions that perform substantially the same functions or achieve substantially the same results as the corresponding aspects described herein may be utilized. Thus, the appended claims include such processes, machines, manufactures, compositions of events, means, methods, or actions within their scope.

[0170] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0171] The above description has been presented for the purpose of illustration and description.

[0172] The figures limit the embodiments of the present disclosure to the forms disclosed herein.While various example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions, and sub-combinations thereof.

Claims

1. A file transmission method based on dynamic segmentation, characterized in that: include: Get the file to be transferred; Determine whether the memory space occupied by the file to be transferred is less than a first preset value; If it is not less than the first preset value, dividing the file to be transmitted into N file segments; Using N communication sessions to download the N file segments in parallel; When the file segments transmitted by any one of the communication sessions are downloaded completely, obtaining the first memory occupancy of the segments that have not been downloaded by the other N-1 communication sessions; Determine whether the maximum first memory usage is greater than a second preset value; If it is greater than a second preset value, two new file segments are obtained based on the undownloaded segment with the largest first memory occupancy, one of the new file segments continues to be downloaded by the original communication session, and the other new file segment is downloaded by the idle communication session; The total data volume of the downloaded segments is recorded, and when the total data volume is equal to the memory space value, all the downloaded segments are combined into a file to be transmitted and transmitted to the destination end.

2. The file transmission method based on dynamic segmentation according to claim 1, characterized in that: The step of dividing the file to be transmitted into N file segments includes: Get the maximum number of connections in the isolated switching system; According to the maximum number of connections and the memory space value, a maximum number of divisions is obtained; Determine whether the maximum number of connections is less than the maximum number of divisions; If yes, then the value of N is the maximum number of connections; If not, the value of N is the maximum number of divisions; The file to be transmitted is divided according to the value of N.

3. The file transmission method based on dynamic segmentation according to claim 2, characterized in that: The calculation formula of the maximum number of divisions is: Wherein, N' is the maximum number of segments; min(·) is the minimum value function; max(·) is the maximum value function; M is the maximum number of connections; P is a preset constant; S is the memory space occupied by the file to be transferred; T is the preset initial segmentation threshold; is the ceiling function.

4. The file transmission method based on dynamic segmentation according to claim 1, characterized in that: The calculation formula of the second preset value is: L = T / 2M; Wherein, L is the second preset value, T is the preset initial segmentation threshold; and M is the maximum number of connections of the isolation switching system.

5. The file transmission method based on dynamic segmentation according to claim 1, characterized in that: Also includes: Get the current number of communication sessions and the maximum number of connections in the isolated switching system; When the current number of communication sessions is less than the maximum number of connections, obtaining the current overall transmission speed of the file to be transmitted and the maximum file transmission speed of the isolation switching system; Based on the maximum file transfer speed, obtaining a speed threshold; When the current overall transmission speed is less than the speed threshold, obtaining a second memory occupancy of all undownloaded segments of the communication session; When the maximum second memory occupancy is greater than a second preset value, adding a new communication session; Based on the undownloaded segment with the largest second memory usage, two new file segments are obtained, one of the new file segments is continued to be downloaded by the original communication session, and the other new file segment YY+242357P is downloaded. Segments are downloaded by new communication sessions.

6. The file transmission method based on dynamic segmentation according to claim 5, characterized in that: The calculation formula of the speed threshold is: SY = SM * BF; Wherein, SY is the speed threshold; SM is the maximum file transfer speed; and BF is a preset percentage, such as 80%.

7. The file transmission method based on dynamic segmentation according to any one of claims 1, 2 or 5, characterized in that: The file to be transmitted is a file to be synchronized; the maximum number of connections of the isolation exchange system is the maximum number of synchronization connections.

8. A computer device, characterized in that: The computer device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the file transmission method based on dynamic segmentation as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the file transmission method based on dynamic segmentation as described in any one of claims 1-7.

10. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images