A file transmission method based on dynamic fragmentation and electronic equipment

Abstract

The application provides a file transmission method based on dynamic fragmentation, comprising: performing sampling processing on a target file to obtain file sampling feature information; determining a predicted multiplexing parameter corresponding to the target file according to the file sampling feature information and pre-stored historical file data; adjusting the weights of a calculation cost parameter and a communication interaction parameter according to the predicted multiplexing parameter; determining a target fragmentation parameter corresponding to the target file based on the adjusted weights of the calculation cost parameter and the communication interaction parameter; fragmenting the target file according to the target fragmentation parameter and performing transmission processing. The application obtains a predicted multiplexing parameter by using sampling features before transmission on fragments, and adjusts the weights of a calculation cost parameter and a communication interaction parameter to determine the size of the fragments, so that data multiplexing, hash calculation cost and front-end and back-end interaction cost are considered in large file fragmentation transmission, and the efficient and stable transmission of large files is effectively ensured.

Description

Technical Field

[0001] This application belongs to the field of data processing technology, and in particular relates to a file transfer method and electronic device based on dynamic fragmentation. Background Technology

[0002] In scenarios such as cloud storage, enterprise cloud drives, data backup, object storage, and file synchronization, large files are typically uploaded in chunks. However, there is an inherent performance conflict between the time-consuming hash calculations of chunks, the frequency of front-end and back-end interactions, and file block reuse. In related technologies, the chunk size is usually a fixed value, or adjusted mainly based on factors such as network bandwidth, client performance, and server load. The finer the chunk granularity, the better it is for discovering reusable data, but it also increases the number of hash calculations, index queries, and interactions between the client and server, ultimately leading to a longer total upload time. Conversely, the coarser the chunk granularity can reduce computational and interaction overhead, but it may reduce the granularity of duplicate data identification, resulting in lower data block reuse rates, increased write storage, high server load, and a longer total upload time. Summary of the Invention

[0003] In view of this, this application aims to propose a file transfer method and electronic device based on dynamic fragmentation, so as to at least solve the problem that data fragmentation cannot simultaneously take into account data reuse, hash calculation cost and front-end and back-end interaction cost in the large file fragmentation transfer.

[0004] This application provides a file transfer method based on dynamic fragmentation, comprising: acquiring a target file; the target file being a file to be transferred; sampling the target file to obtain file sampling feature information; determining the predicted reuse parameters corresponding to the target file based on the file sampling feature information and pre-stored historical file data, wherein the predicted reuse parameters characterize the degree to which data in the target file can be reused by existing stored data; acquiring computational cost parameters and communication interaction parameters during the transmission process; adjusting the weights corresponding to the computational cost parameters and communication interaction parameters based on the predicted reuse parameters; determining the target fragmentation parameters corresponding to the target file based on the adjusted weights of the computational cost parameters and communication interaction parameters; fragmenting the target file according to the target fragmentation parameters and performing transmission processing.

[0005] Furthermore, the sampling process includes: obtaining at least one sampling region from the header region, tail region, middle region, and random region of the target file, and performing hash processing on the data in the sampling region to obtain sampling hash information.

[0006] The file sampling feature information includes at least one of the following: sampling hash information corresponding to the sampling area data, sampling area location, sampling area length, target file size information, and target file type information.

[0007] Furthermore, the historical file data includes at least one of the following: historical file sampling feature information, historical file actual reuse results, historical file fragmentation parameters, historical prediction reuse parameters, and historical prediction errors.

[0008] Furthermore, determining the predicted reuse parameters corresponding to the target file includes: matching the current file sampling feature information with the historical file sampling feature information to obtain candidate historical file records; and determining the predicted reuse parameters based on the feature similarity of the candidate historical file records and the corresponding historical real reuse results.

[0009] Furthermore, the computational cost parameter is used to characterize the client's data processing capability; the computational cost parameter is determined based on the amount of data processed per unit time when the client performs a hash processing task, and is adjusted based on at least one of the client's processor performance, current processing load, memory status, and number of background tasks.

[0010] Furthermore, communication interaction parameters are used to characterize the client-server interaction capacity during the upload process; the communication interaction parameters are determined based on at least one of network latency, available bandwidth, server request load, request queue length, interface rate limiting status, storage index query pressure, and resource quota.

[0011] Furthermore, the weights corresponding to the computational cost parameters and communication interaction parameters are adjusted according to the predicted reuse parameters; this includes: increasing or decreasing the weights corresponding to the computational cost parameters according to the predicted reuse parameters, and adjusting the weights corresponding to the communication interaction parameters accordingly, so that the relative influence of the computational cost parameters and communication interaction parameters changes with the degree of predicted reuse.

[0012] Furthermore, determining the target fragmentation parameters corresponding to the target file based on the adjusted computational cost parameter weights and communication interaction parameter weights includes: obtaining a comprehensive evaluation result based on the computational cost parameters, communication interaction parameters, and adjusted weights; determining the target fragmentation parameters based on the comprehensive evaluation result and the preset mapping relationship; the target fragmentation parameters include at least one of fragment size, fragment quantity, concurrent transmission quantity, batch query quantity, and prefetch quantity.

[0013] Furthermore, it also includes calculating the fragment hash of the fragment data during fragmented transmission, sending the fragment hash to the server for querying existing data; for fragments that the server determines already exist, establishing a reuse relationship without repeatedly uploading the corresponding fragment data; and for fragments that the server determines do not exist, uploading the corresponding fragment data.

[0014] This application also provides a file transfer system based on dynamic fragmentation, comprising: a file acquisition module for acquiring a target file, wherein the target file is a file to be transferred; a sampling module for sampling the target file to obtain file sampling feature information; a multiplexing parameter prediction module for determining predicted multiplexing parameters based on the file sampling feature information and historical file data; a computational cost evaluation module for obtaining client computational cost parameters; an interaction capability evaluation module for determining communication interaction parameters; a weight adjustment module for adjusting the weights corresponding to the computational cost parameters and communication interaction parameters based on the predicted multiplexing parameters; a fragmentation decision module for determining target fragmentation parameters; a transmission execution module for performing fragmentation verification and transmission based on the target fragmentation parameters; and a feedback update module for updating historical file data based on actual multiplexing results.

[0015] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above-described file transfer methods based on dynamic fragmentation.

[0016] This application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of any of the above-described file transfer methods based on dynamic fragmentation.

[0017] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described file transfer methods based on dynamic fragmentation.

[0018] This application first samples the target file to be transmitted to obtain file sampling feature information. Based on the file sampling feature information and pre-stored historical file data, it determines the predicted reuse parameters corresponding to the target file to be transmitted. This allows the potential reuse degree to be predicted using sampling features before formal fragmentation and uploading, enabling the file's own data reuse characteristics to participate in the fragmentation decision in advance. Then, it obtains the computational cost parameters and communication interaction parameters during the transmission process, adjusts the weights of the computational cost parameters and communication interaction parameters according to the predicted reuse parameters, and determines the target fragmentation parameters corresponding to the target file based on the adjusted weights of the computational cost parameters and communication interaction parameters. This comprehensively considers the predicted reuse parameters with the client's computational cost and the system's interaction capacity, thus simultaneously taking into account data reuse, hash calculation cost, and front-end and back-end interaction costs in large file fragmentation transmission. Finally, it fragments the target file to be transmitted according to the target fragmentation parameters and performs transmission processing, thereby solving the problem of efficient and stable transmission of large files in related technologies. Attached Figure Description

[0019] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a schematic diagram of a file transfer system based on dynamic fragmentation according to an embodiment of this application; Figure 2 This is a flowchart illustrating a file transfer method based on dynamic fragmentation, according to an embodiment of this application. Figure 3 This is a schematic diagram of an electronic device according to an embodiment of this application.

[0020] Explanation of reference numerals in the attached figures: 102 - Processor; 104 - Memory; 106 - Transmission device; 108 - Input / output device. Detailed Implementation

[0021] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0022] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0023] It should be understood that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0024] The file transfer method based on dynamic fragmentation provided in this application can be applied to computer systems that require large-scale file uploads, file synchronization, data backup, and deduplication. Specific application scenarios include, but are not limited to: (1) Enterprise data backup scenario: In an enterprise data backup system, client devices typically need to regularly upload a large number of business data files to the cloud storage system. For example: database backup files, virtual machine image files, log archive files, project files, and multi-version document files.

[0025] (2) Cloud storage and file synchronization scenarios: In personal cloud storage, enterprise cloud storage and collaborative office systems, different users may upload the same software installation packages, the same template files, the same image resources, the same video materials and the same office documents; the server side usually maintains a large number of historical file fragment indexes. When a new file is uploaded: the traditional solution needs to complete the file fragmentation and then query each fragment; while this solution can predict in advance whether the current file has a high possibility of data reuse.

[0026] (3) Software image and version release scenario: In software distribution, container image management and virtualization environment: There are usually only a few content changes between different version images.

[0027] like Figure 1 As shown in the figure, a file transfer system based on dynamic fragmentation according to an embodiment of this application includes a client and a server. The client includes a file acquisition module, a sampling module, a computational cost evaluation module, an upload execution module, and a result receiving module; the server includes a sampling feature receiving module, a historical file data management module, a reuse parameter prediction module, an interactive capability evaluation module, a weight adjustment module, a fragmentation decision module, a transmission execution module, and a feedback update module.

[0028] The client and server can communicate via HTTP, HTTPS, RPC, WebSocket, or other data communication protocols. Each module can be implemented by software programs, hardware circuits, firmware, or a combination thereof.

[0029] The hardware architecture corresponding to this invention will be described below using an "enterprise data backup system" as an example. This system includes: (1) The client acquisition device can be: an enterprise server, a database server, a file server, a workstation, or an edge computing node. The client acquisition device includes: a processor module, a storage module, a network communication module, and a memory module. The processor module can include: a CPU, GPU, NPU, or FPGA acceleration module, used to perform file reading, sampling processing, hash calculation, and upload control. The storage module can include solid-state drives (SSDs), hard disk drives (HDDs), RAID storage arrays, or network attached storage (NAS), used to store files to be uploaded, temporary sampling data, fragmented cache data, and upload status information. The network communication module can include Ethernet interfaces, fiber optic communication interfaces, WiFi communication modules, or 5G communication modules; used for data interaction between the client and the server. For example, connecting to the server via a 1Gbps, 10Gbps, or higher speed network. The memory module can include DRAM, DDR memory, or a cache, used to cache file sampling data, cache fragmented calculation results, and save upload task status.

[0030] (2) Server-side computing nodes can be deployed in: enterprise private clouds, public cloud platforms, and data centers. The server includes: a business processing server and a predictive computing server. The business processing server is used to receive client upload requests, perform reuse prediction, and generate sharding strategies. The predictive computing server is used to perform file feature matching, historical data retrieval, and reuse rate prediction.

[0031] (3) Under the above hardware architecture, the file transfer method based on dynamic fragmentation in this application includes: Step S101: The client processor reads the file to be uploaded; Step S102: The client uses memory to cache the sampled region data; Step S103: The processor performs a hash calculation to obtain the file sampling fingerprint; Step S104: The client sends file sampling characteristics, file size, file attributes, and hash speed information to the server through the network communication module.

[0032] Step S105: Server-side business processing: The server receives data, and the prediction calculation server performs historical index query, feature matching, and parameter reuse prediction.

[0033] Step S106: The server determines the target sharding parameters based on the predicted reuse parameters, the client's calculated cost parameters, and the communication interaction parameters.

[0034] Step S107: The client performs file chunk upload based on the returned result.

[0035] This embodiment provides a file transfer method based on dynamic fragmentation, applied to the aforementioned hardware system architecture. Figure 2This is a flowchart of a file transfer method based on dynamic fragmentation according to an embodiment of this application, including the following steps: Step S201: Obtain the target file, wherein the target file is the file to be transferred; Step S202: Perform sampling processing on the target file to obtain file sampling feature information; Step S203: Based on the file sampling feature information and the pre-stored historical file data, determine the prediction reuse parameters corresponding to the target file. The prediction reuse parameters are used to characterize the degree to which the data in the target file can be reused by the existing stored data. Step S204: Obtain the computational cost parameters and communication interaction parameters during the transmission process; Step S205: Adjust the weights corresponding to the calculation cost parameters and communication interaction parameters according to the predicted reuse parameters; Step S206: Determine the target fragment parameters corresponding to the target file based on the adjusted calculation cost parameter weights and communication interaction parameter weights; Step S207: The target file is fragmented according to the target fragmentation parameters, and the transmission process is performed.

[0036] Through the above steps, the potential reuse level is predicted using sampling feature information before the actual fragmented upload. This allows the data duplication characteristics of the file itself to participate in the fragmentation decision in advance. The predicted reuse value is considered in conjunction with the client's computing power and the system's interaction capacity, dynamically balancing fine-grained reuse benefits with the costs of hash calculation, index lookup, and network interaction. This scheme enables dynamic adjustment of fragment size based on the current target file information and the system's computing and interaction capabilities in large file transfer scenarios, achieving efficient transmission of large files.

[0037] In some embodiments, the sampling process in step S202 includes: obtaining at least one sampling region from the header region, tail region, middle region, and random region of the target file, and performing hash processing on the data in the sampling region to obtain sampling hash information.

[0038] In some embodiments, the client first obtains the file size (FileSize), file type (FileType), and optional file metadata of the file to be uploaded. Then, it determines the sampling region based on a sampling strategy, which can be fixed sampling, dynamic sampling, random sampling, or a combination of multiple sampling methods.

[0039] In some embodiments, data from the header and footer regions of the target file are acquired, for example, by reading 128KB of data from the header and 128KB of data from the footer as sampled data. This value is only used to illustrate one implementation method and is not intended to limit this application. For very large files, intermediate sampling regions or random sampling regions can be added according to the file size; for small files, the entire file can be used directly as the sampling region.

[0040] By sampling only the first and last fixed bytes of the target file, without reading the entire large file or calculating the full file hash, millisecond-level sampling can be achieved without blocking front-end page interaction. Furthermore, by supplementing sampling with middle regions or random positions for different file sizes and types, the comprehensiveness of the sampling information can be ensured, and the accuracy of subsequent predictions can be improved.

[0041] In some embodiments, a hash calculation is performed on each sampling region to obtain sampling hash information; wherein the hash calculation may employ one or more hash algorithms such as SM3 / SHA256 / MD5. Multiple sampling hash information pieces are combined according to the sampling region location and sampling region length to generate sampling feature information. In some embodiments, the sampling feature information may also include auxiliary features such as target file size and target file type.

[0042] This application generates sampling feature information by combining each sampling hash with the sampling region location and length, enabling the features to carry sampling spatial distribution information, reducing the probability of hash collisions, and improving feature discriminativeness. Furthermore, it introduces auxiliary features such as target file size and file type to enrich the feature dimensions, reduce misjudgments during file reuse comparison, and effectively improve the accuracy of file reuse judgment.

[0043] It should be noted that the sampled feature information is used for reusability parameter prediction before uploading, not to replace the complete data verification after the actual fragmentation. During the actual upload phase, fragment hashes can still be calculated for the target fragments to ensure the accuracy of the reusability determination.

[0044] In some embodiments, historical file data includes at least one of historical file sampling feature information, historical file actual reuse results, historical file fragmentation parameters, historical prediction reuse parameters, and historical prediction errors. The historical file data is used for matching calculations with the file sampling feature information.

[0045] In some embodiments, the server pre-maintains a historical file data index. Each historical file data record includes at least sampling feature information and the actual reuse results obtained during the upload process; it may also include file size, file type, historical fragment size, historical prediction reuse parameters, prediction confidence, and prediction error. The historical file data index can be a relational database, key-value database, inverted index, distributed cache, or object storage.

[0046] After receiving the current sampling feature information, the server retrieves candidate records with similar sampling features from the historical index. For each candidate record, feature similarity can be calculated. Feature similarity can be obtained by comprehensively considering factors such as hash matching of multiple sampling regions, similarity in file size, and consistency in file type. For candidate records, the historical true reuse parameters can be weighted according to similarity to obtain predicted reuse parameters. One example calculation method is: the predicted reuse parameter equals the sum of the products of the similarity of each candidate record and its true reuse parameter, divided by the sum of the similarities of the candidate records. In some embodiments, the prediction reuse parameter calculation can also employ classification models, regression models, probabilistic models, nearest neighbor models, or machine learning models, as long as they can output the probability of file reuse based on the file sampling features obtained before uploading, they fall under the technical concept of this application.

[0047] In this application, the calculation of the predicted reuse parameters is performed before the formal fragmentation and full fragmentation verification. This differs from the post-verification of reuse rate in related technologies. It enables the prediction of the reuse of existing data in the file to be uploaded, thereby enabling the dynamic correction of file fragment size in advance.

[0048] In some embodiments, to avoid prediction bias caused by relying on only a small number of sampled features, the server can simultaneously generate prediction confidence. Prediction confidence can comprehensively consider the following factors: the number of candidate historical records, the degree of matching of sampled features, the consistency of prediction results of different candidate records, historical prediction accuracy, and current sampling coverage.

[0049] For example, matching confidence, historical accuracy, and candidate consistency can be obtained separately, and prediction confidence can be obtained based on a preset combination function. Prediction confidence can be a numerical value between 0 and 1, or it can be represented in a hierarchical form.

[0050] In some embodiments, effective prediction reuse parameters are used in subsequent weight adjustments. These effective prediction reuse parameters can be determined jointly by the prediction reuse parameters and the prediction confidence, for example, through product, piecewise attenuation, or other calculation combinations. Therefore, when the prediction confidence is low, the impact of the prediction reuse parameters on the sharding strategy is automatically reduced.

[0051] In some embodiments, when a valid candidate record cannot be found in the historical index based on the sampled feature information, or when the reuse results between candidate records differ significantly, the prediction confidence may be lower than a first preset confidence threshold. In this case, the system can select a preset secure sharding strategy. This secure sharding strategy can be a fixed shard size, a dynamic strategy determined solely by network and client computing power, or a strategy that limits the impact of prediction reuse parameters on weights to a small range to reduce performance loss caused by prediction errors. This mechanism ensures that the prediction model does not forcibly influence the upload strategy in all cases, thereby avoiding erroneous decisions caused by new file types, abnormal files, or insufficient historical samples.

[0052] In some embodiments, when the prediction confidence is higher than the second preset confidence threshold, which is higher than the first preset confidence threshold, the system can determine that the file reuse rate is high. Therefore, it can use larger fragments or transmit directly without fragmentation and call the backend interface to generate file data.

[0053] In some embodiments, an evaluation score is calculated based on a combination of computational cost parameters and communication interaction parameters, and the server determines the target sharding parameters based on the evaluation score. The target sharding parameters can be determined using a preset mapping table. For example, the evaluation score can be divided into multiple intervals, each interval corresponding to a sharding size level, as shown in Table 1.

[0054] Table 1 Preset Fragment Size Mapping Table In some embodiments, the target fragment size can be continuously calculated based on the evaluation score, and minimum and maximum values ​​can be set for boundary constraints. For example, the target fragment size can be selected within the range of 2MB to 10MB; this range is only an example, and the actual system can set other ranges according to file type, network environment, storage system capacity, and business needs. Thus, by fixing the upper and lower limits of fragments, the resource consumption of the backend cluster has a stable upper limit, which facilitates capacity planning and cluster expansion; at the same time, it prevents oversized fragments from causing hash timeouts on weak terminals and extremely small fragments from overwhelming the gateway with hundreds of thousands of fragments.

[0055] In some embodiments, in addition to the fragment size, the system can also determine parameters such as the number of concurrent uploads, the number of prefetches, and the number of batch queries based on the same evaluation score or evaluation results related to the score, so as to coordinate the adjustment of fragment granularity and the degree of interaction concurrency.

[0056] In some embodiments, the computational cost parameter is used to characterize the client's data processing capability, is determined based on the amount of data processed per unit time when the client performs a hash processing task, and is adjusted based on at least one of the client's processor performance, current processing load, memory status, and number of background tasks.

[0057] In some embodiments, the computational cost parameter can be the hash calculation speed, with the client performing a baseline hash calculation task using sampled data. For example, if the amount of sampled data processed during the test is D, the test start time is T1, and the end time is T2, then the original hash processing capacity can be obtained based on the relationship between D and T2-T1. To reduce the impact of instantaneous load, the test can be repeated multiple times, and the average, median, or statistical value after removing outliers can be used as the baseline result. Furthermore, the baseline result can be corrected based on the client's processor capabilities, current CPU usage, memory pressure, number of background tasks, etc., to obtain the computational cost parameter.

[0058] In some embodiments, communication interaction parameters are used to characterize the client-server interaction capacity during the upload process, and to evaluate the capacity to perform interactive operations such as fragment query, verification and confirmation in the current upload environment; communication interaction parameters may include the frequency of backend security interactions.

[0059] In some embodiments, communication interaction parameters are determined based on at least one of network latency, available bandwidth, server request load, request queue length, interface rate limiting status, storage index query pressure, and resource quota. For example, the basic interaction overhead is first determined based on the expected number of shards, and then adjusted in conjunction with the network and server status.

[0060] In some embodiments, to facilitate combination with other parameters, the computational cost parameters and communication interaction parameters can be normalized. Normalization methods can include interval normalization, ranking normalization, logarithmic normalization, or other equivalent methods. This maps the differentiated computational parameters uniformly to the 0-1 standard range, enabling weighted calculation of each parameter and resolving the problem of inconsistent quantitative comparison of different devices and parameter indicators.

[0061] In some embodiments, the computational cost parameter H and the communication interaction parameter Q are quantized separately and given fixed independent weights. A comprehensive score is obtained by weighted summation, and the optimal file block size is adaptively determined by matching a preset block mapping table based on the comprehensive score.

[0062] In some embodiments, the basic weight W for calculating cost parameters is set. h and communication interaction parameter basic weight W q The sum of the two is 1, that is, W. h +W q =1, for example W h The base weight is 0.5, W q The base weight is 0.5. Then, the base weights of both are dynamically adjusted according to the prediction reuse parameter R.

[0063] In one embodiment, the adjusted weights are: Furthermore, to ensure that the sum of the adjusted weights is 1, the adjusted weights are corrected to obtain the final calculated weights for the cost parameters and communication interaction parameters: The above weight adjustment is only one possible approach. In practice, the system can use linear functions, nonlinear functions, piecewise functions, lookup table functions, or weight functions obtained through training.

[0064] In one implementation, both the computational cost parameter H and the communication interaction parameter Q are converted into normalized indicators with the same evaluation direction, and a comprehensive evaluation score (Score) is calculated. The block size is then determined based on a preset mapping table. For example, the Score can be determined by H and... And Q and The weighted combination yields: In this scenario, when R is high, the system can handle the increased computation and verification work required to improve the granularity of multiplexing identification, thus favoring a finer-grained sharding strategy. When R is low, continuing to use extremely fine sharding may result in higher hashing and interaction overhead, thus favoring a strategy that reduces the number of shards and interactions. For example, a smooth adjustment can be achieved using three multiplexing intervals: high multiplexing interval (R ≥ 0.6), increasing the weight of computational cost parameters and decreasing the weight of communication interaction parameters, with the algorithm biased towards larger blocks; low multiplexing interval (R ≤ 0.2), decreasing the weight of computational cost parameters and increasing the weight of communication interaction parameters, with the algorithm biased towards limiting the number of shards; medium multiplexing interval (0.2 < R < 0.6), where the weights of both parameters change linearly and continuously with R.

[0065] By dynamically adjusting the weights of computational cost parameters and communication interaction parameters, and consequently adjusting the fragment size, the shortcomings of fixed weights in related technologies are resolved. File reuse automatically participates in fragmentation decisions, reducing the total number of front-end hashes in high-reuse scenarios and lowering the concurrency pressure on back-end interfaces in low-reuse scenarios. Simultaneously, through linearly smoothing weight changes, fragment size does not jump, resulting in a smoother user upload experience. Hash power, interaction frequency, and fragment reuse are interconnected and balanced, ensuring efficient and stable file transmission.

[0066] In some embodiments, the target sharding parameters include at least one of shard size, number of shards, number of concurrent transmissions, number of batch queries, and number of prefetches.

[0067] In some embodiments, fragmentation decisions can be executed independently by the client, executed by the server and then issued, or executed collaboratively by the client and the server. The target fragmentation parameters can be determined once before file transfer, or recalculated during the upload process based on changes in network and server load, thereby enabling more granular dynamic fragmentation of the file.

[0068] In some embodiments, after determining the target sharding parameters, the client shards the file according to the target sharding parameters and calculates a shard hash for each shard. The client can send multiple shard hash query requests to the server in batches. The server determines the existing shards and the shards that need to be uploaded based on the shard hash query results. For existing shards, the corresponding reuse relationships are recorded; for non-existent shards, the client uploads the actual data.

[0069] After confirming that all necessary data exists or has been uploaded, the server constructs the logical file based on the fragment order, file index, or other file organization information. For fragments that already exist, there is no need to re-transmit the actual data.

[0070] In some embodiments, the file transfer method based on dynamic fragmentation of this application further includes, after the upload is completed, the server obtains the actual reuse status based on the fragmentation query results of this upload. The actual reuse rate can be determined based on the relationship between the actual amount of reused data and the total amount of file data, or it can use the number of reused fragments, the amount of reused bytes, or other indicators that can represent the actual degree of reuse. The server associates and stores the current adoption feature information, predicted reuse parameters, prediction confidence, actual reuse results, and target fragmentation parameters, and can further calculate the prediction error and update the historical index using new records. In subsequent file upload processes, the server can re-perform candidate matching and prediction based on the updated historical data, thereby forming a closed loop of pre-upload prediction—adaptive fragmentation—actual reuse feedback—historical data update—next prediction.

[0071] In some embodiments, after obtaining the basic comprehensive score based on the weighted adjustment of computational cost parameters and communication interaction parameters, this application further includes two independent business adjustment coefficients based on user factors and permission factors to perform hierarchical and step-by-step adjustment of the basic comprehensive score, so as to realize the fragmentation size decision logic to synchronously adapt to the client hardware environment, network transmission environment, platform account management strategy and data security management requirements. Compared with the fragmentation strategy of related technologies that only combines network, system and file single-dimensional factors, this application solves the technical defects of related technologies that cannot take into account the terminal computing power layering, regional link differences, account bandwidth quotas and audit management of confidential documents through two-layer factor hierarchical adjustment.

[0072] In some embodiments, the first-level independent service adjustment is the user factor F. uUser factor F u This is a static adjustment coefficient without behavioral attributes, calculated by combining three objective environmental parameters: terminal hardware type, terminal real-time hardware load, and client access region. The calculation formula is as follows: in, The user factor value is assigned to the terminal type. The user factor value is assigned to the terminal load. The user factor value is assigned to the region of access.

[0073] In some embodiments, the values ​​of each sub-factor and the corresponding technical logic are shown in Table 2 below.

[0074] Table 2 User Factor Sub-Factor Values ​​and Adaptation Logic The weighted composite score (Score) and the user factor (F) are calculated together. u Multiplication is performed to correct the first layer of environmental dimensions, resulting in the terminal adaptation score. Calculation formula: In some embodiments, the second-tier independent service regulation is the permission factor F. a Permission factor F a This is a static policy adjustment coefficient for the platform, generated solely based on the account's preset permission level, upload bandwidth quota, and file security control rules. It is unrelated to real-time user actions. The calculation formula is as follows: In some embodiments, the account tier coefficient may include: high privilege, normal privilege, and restricted privilege; the values ​​of each tier and the control adaptation logic are shown in Table 3 below: Table 3. Permission Factor Level Values ​​and Control Adaptation Logic The terminal adaptation score after user factor correction With permission factor F a Multiplication is performed to complete secondary corrections in platform control and security dimensions, and the final global comprehensive adaptation score is output. Calculation formula: This application employs a two-tiered business adjustment factor system to stratify and correct the overall score. By differentiating terminal computing power and access network through a user factor, it achieves adaptive sharding adjustment based on the strength of devices and the quality of links, significantly reducing hash calculation blockage on weak mobile terminals and the probability of timeouts in cross-border network sharding transmission. Through a permission factor, platform bandwidth quotas and confidentiality audit security rules are embedded into the sharding decision logic, enabling user-tiered rate limiting and security compliance verification without the need for additional independent business processes, thus reducing business development costs. The two factors are independent and do not interfere with each other, allowing for separate activation / deactivation and parameter configuration. When there is no need to differentiate account permissions, F can be fixed. a =1.0, only user factor adjustment is retained; when all clients are unified high-performance intranet devices, F can be fixed. u =1.0, only enabling permission security control adjustment; the final score output after the two-layer factor synergy is used as the only input value for shard size mapping, realizing multi-dimensional synergistic balance of hash power, interaction frequency, shard reuse rate, terminal hardware, network link, account permissions, security audit, etc., fundamentally solving the technical defects of existing sharding strategies that are single in dimension and poor in scenario adaptability.

[0075] Embodiments of this application also provide an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to perform the following steps: Step S301: Obtain the target file, wherein the target file is the file to be transferred; Step S302: Perform sampling processing on the target file to obtain file sampling feature information; Step S303: Based on the file sampling feature information and the pre-stored historical file data, determine the prediction reuse parameters corresponding to the target file. The prediction reuse parameters are used to characterize the degree to which the data in the target file can be reused by the existing stored data. Step S304: Obtain the computational cost parameters and communication interaction parameters during the transmission process; Step S305: Adjust the weights corresponding to the calculation cost parameters and communication interaction parameters according to the predicted reuse parameters; Step S306: Determine the target fragment parameters corresponding to the target file based on the adjusted calculation cost parameter weights and communication interaction parameter weights; Step S307: The target file is fragmented according to the target fragmentation parameters, and the transmission process is performed.

[0076] like Figure 3As shown, the electronic device may include one or more processors 102 (processor 102 may include, but is not limited to, a processing system such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data. In one exemplary embodiment, the electronic device may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 3 The structure shown is for illustrative purposes only and does not limit the structure of the electronic device described above. For example, the electronic device may also include components that are more... Figure 3 The more or fewer components shown, or having the same Figure 3 Equivalent functions or ratios shown Figure 3 The functions shown have more different configurations.

[0077] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the file upload method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thus implementing the above-described method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage systems, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to a secure text network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0078] The transmission system 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider for the computer terminal. In one example, the transmission system 106 includes a network interface controller (NIC), which can connect to other network devices via a base station to communicate with the Internet.

[0079] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above embodiments of the file transfer method based on dynamic fragmentation.

[0080] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0081] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0082] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing systems. They can be centralized on a single computing system or distributed across a network of multiple computing systems. Optionally, they can be implemented using program code executable by a computing system, thereby storing them in a storage system for execution by the computing system. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0083] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A file transfer method based on dynamic fragmentation, characterized in that: Obtain the target file; the target file is the file to be transmitted. The target file is sampled to obtain file sampling feature information; Based on the file sampling feature information and the pre-stored historical file data, the prediction reuse parameters corresponding to the target file are determined. The prediction reuse parameters are used to characterize the degree to which the data in the target file can be reused by the existing stored data. Obtain computational cost parameters and communication interaction parameters during the transmission process; The weights corresponding to the calculation cost parameter and the communication interaction parameter are adjusted according to the prediction reuse parameter; The target fragmentation parameters corresponding to the target file are determined based on the adjusted weights of the computational cost parameters and the communication interaction parameters. The target file is fragmented according to the target fragmentation parameters, and then transmission processing is performed.

2. The file transfer method based on dynamic fragmentation according to claim 1, characterized in that: The sampling process includes: obtaining at least one sampling region from the header region, tail region, middle region, and random region of the target file, and performing hash processing on the data in the sampling region to obtain sampling hash information; The file sampling feature information includes at least one of the following: sampling hash information corresponding to the sampling area data, sampling area location, sampling area length, target file size information, and target file type information.

3. The file transfer method based on dynamic fragmentation according to claim 1, characterized in that: The historical file data includes at least one of the following: historical file sampling feature information, historical file actual reuse results, historical file fragmentation parameters, historical prediction reuse parameters, and historical prediction errors.

4. The file transfer method based on dynamic fragmentation according to claim 3, characterized in that: The step of determining the predicted reuse parameters corresponding to the target file includes: matching the current file sampling feature information with the historical file sampling feature information to obtain candidate historical file records; and determining the predicted reuse parameters based on the feature similarity of the candidate historical file records and the corresponding historical real reuse results.

5. The file transfer method based on dynamic fragmentation according to claim 1, characterized in that: The computational cost parameter is used to characterize the client's data processing capability; the computational cost parameter is determined based on the amount of data processed per unit time when the client performs a hash processing task, and is adjusted based on at least one of the client's processor performance, current processing load, memory status, and number of background tasks.

6. The file transfer method based on dynamic fragmentation according to claim 1, characterized in that: The communication interaction parameters are used to characterize the client-server interaction capacity during the upload process; the communication interaction parameters are determined based on at least one of network latency, available bandwidth, server request load, request queue length, interface rate limiting status, storage index query pressure, and resource quota.

7. The file transfer method based on dynamic fragmentation according to claim 1, characterized in that: The step of adjusting the weights corresponding to the computation cost parameter and the communication interaction parameter according to the predicted reuse parameter includes: increasing or decreasing the weight corresponding to the computation cost parameter according to the predicted reuse parameter, and adjusting the weight corresponding to the communication interaction parameter accordingly, so that the relative influence of the computation cost parameter and the communication interaction parameter changes with the degree of predicted reuse.

8. The file transfer method based on dynamic fragmentation according to claim 1, characterized in that: The step of determining the target fragment parameters corresponding to the target file based on the adjusted computational cost parameter weights and communication interaction parameter weights includes: obtaining a comprehensive evaluation result based on the computational cost parameters, communication interaction parameters, and adjusted weights; determining the target fragment parameters based on the comprehensive evaluation result and a preset mapping relationship; the target fragment parameters include at least one of fragment size, fragment quantity, concurrent transmission quantity, batch query quantity, and prefetch quantity.

9. The file transfer method based on dynamic fragmentation according to claim 1, characterized in that: It also includes calculating the fragment hash of the fragment data when performing fragmented transmission, sending the fragment hash to the server for querying the existing data; and establishing a reuse relationship for fragments that the server has determined to exist, so as not to upload the corresponding fragment data repeatedly. For shards that the server determines do not exist, upload the corresponding shard data.

10. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor is configured to implement the file transfer method based on dynamic fragmentation as described in any one of claims 1-9 when executing the computer program.

Images