Data transmission method, device and equipment and readable storage medium

Abstract

The invention provides a data transmission method, device and equipment and a readable storage medium, and the method comprises the steps: obtaining the file attribute information of a to-be-transmitted file, and enabling fragmentation transmission if the file attribute information meets a preset condition; hardware resource information is obtained, configuration information of fragment transmission is configured according to the hardware resource information, and the configuration information at least comprises the fragment size; executing the fragmentation transmission according to the configuration information of the fragmentation transmission; the file to be transmitted is a file to be sent by the opposite terminal equipment and / or a file to be acquired by the opposite terminal equipment, and the fragment transmission comprises fragment receiving of the file from the opposite terminal equipment and / or fragment sending of the file to the opposite terminal equipment. Through the technical scheme of the specification, the size of the fragment is dynamically adjusted, the transmission efficiency of the large file is optimized, the memory pressure is reduced, the network retry cost is reduced, and the object storage service performance is improved.

Description

Technical Field

[0001] The present specification relates to the field of communication technology, and in particular to a data transmission method, device, equipment and readable storage medium. Background Art

[0002] Object Storage Service (OBS), as a service that can efficiently manage and store large amounts of unstructured data, has become an important part of the cloud storage field. OBS allows users to upload, download, and manage data in the form of files. These files can be of any size and format and are widely used in various application scenarios, such as backup and recovery, content distribution network (CDN), and archiving.

[0003] However, in actual applications, with the continuous increase in the size of a single storage object and the increasing complexity of the network environment, traditional data transmission methods face many challenges. When faced with extremely large files, the process of loading the entire file into memory at one time and completing the upload or download becomes inefficient and risky. For example, when uploading a large file, if all the data is pre-cached in the client memory and then sent at one time, it may cause excessive local resource usage, especially for those devices with low hardware configuration or that handle multiple tasks at the same time. This undoubtedly increases the system burden and may even cause the system to crash or significantly degrade performance. In addition, once any failure occurs during network transmission - whether it is a brief loss of connection or a more serious interruption - it means that the previously transmitted data may need to be restarted, which not only wastes precious bandwidth resources but also prolongs the total transmission time.

[0004] Similarly, during the download process, if the same overall transmission method is used, that is, requiring the client to receive the complete file before further processing, similar problems will be faced: on the one hand, it places higher demands on the client's memory; on the other hand, the server also needs extra time to prepare the entire file for download, thereby increasing the request response delay and affecting the user experience. Summary of the invention

[0005] In view of this, the present specification provides a data transmission method, device, electronic device, and readable storage medium to improve the above-mentioned problem of poor data transmission experience.

[0006] The specific technical solutions are as follows:

[0007] The present specification provides a data transmission method, which is applied to an object storage service end, and the method includes: obtaining file attribute information of a file to be transmitted, and enabling segmented transmission if the file attribute information meets a preset condition; obtaining hardware resource information, and configuring configuration information of segmented transmission according to the hardware resource information, wherein the configuration information at least includes a segment size; performing segmented transmission according to the configuration information of segmented transmission; the file to be transmitted is a file to be sent by a peer device and / or a file to be obtained by a peer device, and the segmented transmission includes receiving a file from a segment of the peer device and / or sending a file to a segment of the peer device.

[0008] As a technical solution, the file to be transmitted is a file to be sent by a peer device; the file attribute information of the file to be transmitted is obtained, and if the file attribute information meets a preset condition, segmented transmission is enabled, including: obtaining judgment information that the file size of the file to be transmitted exceeds a preset threshold, and enabling segmented transmission in response to a segmented transmission request of the peer device, wherein the judgment information is generated when the peer device monitors that the cache data associated with the file to be transmitted is greater than the preset threshold.

[0009] As a technical solution, the hardware resource information is obtained, and the configuration information of the fragment transmission is configured according to the hardware resource information. The configuration information at least includes the fragment size, including: configuring the upper limit of the fragment size of the fragment transmission according to the network status information, CPU usage, and memory usage of the current local device and the peer device, and selecting the fragment size within the determined upper limit range of the fragment size according to the file attribute information of the file to be transmitted.

[0010] As a technical solution, the file to be transmitted is a file to be sent by the first peer device and a file to be obtained by the second peer device.

[0011] As a technical solution, the method of obtaining hardware resource information and configuring configuration information for segmented transmission according to the hardware resource information, wherein the configuration information at least includes a segment size, includes: respectively obtaining local hardware resource information and hardware resource information of the first peer device, as well as local hardware resource information and hardware resource information of the second peer device associated with the second peer device, and respectively configuring configuration information for segmented transmission associated with the first peer device and the second peer device, wherein the segment size associated with the first peer device and the segment size associated with the second peer device are the same or different; the method of performing segmented transmission according to the configuration information for segmented transmission includes: receiving a file to be transmitted from the first peer device in a segmented transmission manner according to the configuration information for segmented transmission associated with the first peer device, and sending the file to be transmitted to the second peer device in a segmented transmission manner according to the configuration information for segmented transmission associated with the second peer device.

[0012] This specification also provides a data transmission device, which is applied to an object storage service end, and the device includes: a first module, which is used to obtain file attribute information of a file to be transmitted, and if the file attribute information meets a preset condition, enabling segmented transmission; a second module, which is used to obtain hardware resource information, and configure configuration information of segmented transmission according to the hardware resource information, and the configuration information at least includes the segment size; a third module, which is used to perform segmented transmission according to the configuration information of segmented transmission; the file to be transmitted is a file to be sent by a peer device and / or a file to be obtained by a peer device, and the segmented transmission includes receiving a file from a segment of the peer device and / or sending a file to a segment of the peer device.

[0013] As a technical solution, the file to be transmitted is a file to be sent by a peer device; the file attribute information of the file to be transmitted is obtained, and if the file attribute information meets a preset condition, segmented transmission is enabled, including: obtaining judgment information that the file size of the file to be transmitted exceeds a preset threshold, and enabling segmented transmission in response to a segmented transmission request of the peer device, wherein the judgment information is generated when the peer device monitors that the cache data associated with the file to be transmitted is greater than the preset threshold.

[0014] As a technical solution, the hardware resource information is obtained, and the configuration information of the fragment transmission is configured according to the hardware resource information. The configuration information at least includes the fragment size, including: configuring the upper limit of the fragment size of the fragment transmission according to the network status information, CPU usage, and memory usage of the current local device and the peer device, and selecting the fragment size within the determined upper limit range of the fragment size according to the file attribute information of the file to be transmitted.

[0015] As a technical solution, the file to be transmitted is a file to be sent by the first peer device and a file to be obtained by the second peer device.

[0016] As a technical solution, the method of obtaining hardware resource information and configuring configuration information for segmented transmission according to the hardware resource information, wherein the configuration information at least includes a segment size, includes: respectively obtaining local hardware resource information and hardware resource information of the first peer device, as well as local hardware resource information and hardware resource information of the second peer device associated with the second peer device, and respectively configuring configuration information for segmented transmission associated with the first peer device and the second peer device, wherein the segment size associated with the first peer device and the segment size associated with the second peer device are the same or different; the method of performing segmented transmission according to the configuration information for segmented transmission includes: receiving a file to be transmitted from the first peer device in a segmented transmission manner according to the configuration information for segmented transmission associated with the first peer device, and sending the file to be transmitted to the second peer device in a segmented transmission manner according to the configuration information for segmented transmission associated with the second peer device.

[0017] This specification also provides an electronic device, including a processor and a readable storage medium, wherein the readable storage medium stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the aforementioned data transmission method.

[0018] The present specification also provides a readable storage medium, wherein the readable storage medium stores machine executable instructions. When the machine executable instructions are called and executed by a processor, the machine executable instructions prompt the processor to implement the aforementioned data transmission method.

[0019] The above technical solution provided in this specification brings at least the following beneficial effects:

[0020] It intelligently determines file attributes, enables fragmented transmission for large files, and dynamically configures the optimal fragment size based on real-time hardware resources, significantly reducing memory usage and network retry costs, and improving data transmission efficiency and stability. It is suitable for large file upload and download scenarios on the object storage service side, enhancing the overall system performance and user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the implementation methods of this specification or the technical solutions in the prior art, the drawings required for use in the implementation methods of this specification or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some implementation methods recorded in this specification. For ordinary technicians in this field, other drawings can also be obtained based on these drawings of the implementation methods of this specification.

[0022] Figure 1 is a flow chart of a data transmission method in one embodiment of the present specification;

[0023] Figure 2 is a structural diagram of a data transmission device in one embodiment of the present specification;

[0024] Figure 3 It is a schematic diagram of the architecture in one implementation of this specification;

[0025] Figure 4 It is a hardware structure diagram of an electronic device in one implementation mode of this specification.

[0026] Reference numerals: first module 21 , second module 22 , third module 23 . DETAILED DESCRIPTION

[0027] The terms used in the embodiments of this specification are only for the purpose of describing specific embodiments, rather than limiting this specification. The singular forms of "a", "said" and "the" used in this specification and claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to any or all possible combinations of one or more associated listed items.

[0028] It should be understood that although the terms first, second, third, etc. may be used to describe various information in the embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this specification, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, in addition, the word "if" used may be interpreted as "at the time of" or "when" or "in response to determining".

[0029] In view of this, the present specification provides a data transmission method, device, electronic device, and readable storage medium to improve the above technical problems.

[0030] The specific technical solution is described below.

[0031] In one embodiment, the present specification provides a data transmission method, which is applied to an object storage service, and the method includes: obtaining file attribute information of a file to be transmitted, and enabling segmented transmission if the file attribute information meets preset conditions; obtaining hardware resource information, and configuring configuration information of segmented transmission according to the hardware resource information, wherein the configuration information includes at least the segment size; performing segmented transmission according to the configuration information of segmented transmission; the file to be transmitted is a file to be sent by a peer device and / or a file to be obtained by a peer device, and the segmented transmission includes receiving a file from a segment of the peer device and / or sending a file to a segment of the peer device.

[0032] Specifically, Figure 1 , including the following steps:

[0033] Step S11, obtaining file attribute information of the file to be transmitted, and if the file attribute information meets a preset condition, enabling segmented transmission.

[0034] When a user is ready to upload or download a file, the system will first automatically obtain the attribute information of the file, which includes but is not limited to metadata such as file size, type, and creation time. Next, the system compares this attribute information with a set of pre-set conditions to determine whether the fragmented transmission mechanism needs to be enabled. For example, if the file size exceeds a certain threshold (such as 50MB), or if historical transmission records indicate that this type of file is prone to errors when transmitted over the network, the system will automatically select the fragmented transmission mode. In addition, other factors, such as the content sensitivity of the file or user-specified priority settings, can be considered to further refine the criteria for enabling fragmented transmission.

[0035] Step S12, obtaining hardware resource information, and configuring configuration information for slice transmission according to the hardware resource information, wherein the configuration information at least includes the slice size.

[0036] Once it is determined to use sharded transmission, the next step is to evaluate the current available hardware resources, including but not limited to key indicators such as network bandwidth, CPU utilization, and remaining memory space. Based on this information, the system will calculate the shard size and other necessary configuration parameters that are most suitable for the current environment. For example, in a scenario with a high-speed broadband connection but a high CPU load, a relatively large shard size may be selected to balance resource consumption; in other cases, if lower network stability and sufficient free memory are detected, smaller shards may be preferred to complete the transmission faster and reduce the risk of network fluctuations.

[0037] Specifically, suppose that a large file of about 2GB, such as a video file, is being uploaded through OBS. Since this video is very large, uploading it directly may cause a large amount of memory to be occupied for a long time, and once a network interruption occurs, the whole process needs to be restarted. At this time, the system will first check the hardware conditions of both the server and the client: if it is found that the server has sufficient bandwidth to support fast transmission, but the memory on the client machine is limited, then each shard can be set to about 10MB; conversely, if the client's computer performance is strong and the network speed is slow, a larger shard, such as 50MB, can be selected, so that the existing network resources can be used as much as possible without affecting local performance.

[0038] Step S13: Execute the segment transmission according to the segment transmission configuration information.

[0039] The final step is to actually perform the segmented transfer according to the previously determined configuration information. This means that whether receiving or sending files from the peer device, they will be broken down into multiple small pieces. Each segment has a unique identifier to track progress and remains independent throughout the process - even if a segment fails, it will not affect the normal operation of other parts. After all segments are transferred, the system will also perform a full check to ensure that all segments have arrived at the destination correctly and can be accurately reassembled and restored to the original file.

[0040] In one embodiment, at the initial stage of data transmission, the system first needs to obtain the attribute information of the file to be transmitted, including file size, type, access rights, etc. This information is crucial for determining whether the file is suitable for fragmented transmission. For example, if the file size exceeds a preset threshold (such as 100MB), the system will consider the file suitable for fragmented transmission, thereby enabling the fragmentation mechanism.

[0041] Once it is determined that the file is suitable for fragmented transmission, the system will configure the relevant parameters for fragmented transmission. This step is automatic and does not require manual intervention. The system will automatically determine whether fragmentation is required based on the file's attribute information.

[0042] The system also needs to obtain information about current hardware resources, including CPU usage, memory usage, network bandwidth, etc. This information is crucial for dynamically adjusting the shard size because it directly affects the efficiency of data transmission.

[0043] Based on the hardware resource information obtained, the system will dynamically configure the parameters of the fragment transmission, the most critical of which is the fragment size. For example, when the network bandwidth is low or the CPU usage is high, the system may choose a smaller fragment size to reduce the pressure on the network and CPU. On the contrary, when the network bandwidth is sufficient and the CPU load is low, the system may choose a larger fragment size to improve the transmission efficiency.

[0044] After the configuration is completed, the system will perform segmented transmission according to the configuration information. For upload operations, the system will split the file into multiple small segments and upload them one by one; for download operations, the system will request specific segments of the file and download them one by one.

[0045] Suppose there is a 500MB video file that needs to be uploaded to the OBS service. The system first obtains the attribute information of the file and finds that the file size exceeds the preset 100MB threshold, so the fragment transmission mechanism is enabled. The system further checks the current hardware resource information and finds that the network bandwidth is 100Mbps, the CPU usage is 70%, and the memory usage is 60%. Based on this information, the system decides to split the video file into 10 50MB fragments for uploading.

[0046] During the upload process, the system monitors the upload status of each segment. If a segment fails to upload, the system will re-upload the segment without retransmitting the entire file. This ensures reliable data transmission even in unstable network conditions.

[0047] By dynamically adjusting the shard size, the system can effectively utilize network bandwidth, reduce memory pressure, and reduce retry costs caused by network problems. This method not only improves the efficiency of data transmission, but also improves the user experience, especially when processing large files.

[0048] In one embodiment, the file to be transmitted is a file to be sent by a peer device; the file attribute information of the file to be transmitted is obtained, and if the file attribute information satisfies a preset condition, segmented transmission is enabled, including: obtaining judgment information that the file size of the file to be transmitted exceeds a preset threshold, and enabling segmented transmission in response to a segmented transmission request of the peer device, wherein the judgment information is generated when the peer device monitors that the cache data associated with the file to be transmitted is greater than a preset threshold.

[0049] In one embodiment, the hardware resource information is obtained, and the configuration information of the fragmented transmission is configured according to the hardware resource information. The configuration information at least includes the fragment size, including: configuring the upper limit of the fragment size of the fragmented transmission according to the network status information, CPU usage, and memory usage of the current local device and the peer device, and selecting the fragment size within the determined upper limit range of the fragment size according to the file attribute information of the file to be transmitted.

[0050] In one implementation, the file to be transmitted is a file to be sent by the first peer device and is a file to be acquired by the second peer device.

[0051] In one embodiment, the obtaining of hardware resource information and configuring the configuration information for segmented transmission according to the hardware resource information, wherein the configuration information at least includes the segment size, include: respectively obtaining local hardware resource information and the hardware resource information of the first peer device, as well as local hardware resource information and the hardware resource information of the second peer device associated with the second peer device, and respectively configuring the configuration information for segmented transmission associated with the first peer device and the second peer device, wherein the segment size associated with the first peer device and the segment size associated with the second peer device are the same or different; the performing of segmented transmission according to the configuration information for segmented transmission includes: receiving the file to be transmitted from the first peer device in a segmented transmission manner according to the configuration information for segmented transmission associated with the first peer device, and sending the file to be transmitted to the second peer device in a segmented transmission manner according to the configuration information for segmented transmission associated with the second peer device.

[0052] In one implementation, the data transmission process between two end devices (i.e., the first end device and the second end device) is further optimized. Specifically, when the file to be transmitted is both the object to be sent by the first end device and the target to be obtained by the second end device, the system will not only consider the hardware resource status of a single device, but also comprehensively evaluate the resource information of the two end devices and their associated local environments to achieve more refined and efficient fragmented transmission configuration.

[0053] In order to ensure the best transmission performance, the system will collect and analyze two important information: one is the local hardware resource information associated with the first peer device and the hardware resource information of the device itself; the other is the local hardware resource information associated with the second peer device and its own hardware resource information. Here, "local" refers to the resource status of the server or intermediate node that performs the data transmission task, while "self" refers to the resource status of the client device that actually participates in the upload or download operation.

[0054] In this way, we can fully understand the real-time resource utilization of all key links in the entire transmission link, including but not limited to network bandwidth, CPU load, memory availability, etc. For example, in a complex multinational enterprise environment, data centers located in different geographical locations may have different network conditions and technical specifications, so we need to pay special attention to these differences to develop the most appropriate transmission strategy.

[0055] Based on the above-collected dual-end hardware resource information, the system will then configure the relevant parameters of the fragment transmission for each peer device separately. It is worth noting that since the physical environment and operating status of the devices at both ends may be significantly different, the fragment sizes corresponding to them are not necessarily the same. In other words, the fragment size associated with the first peer device and the fragment size associated with the second peer device can be adjusted independently according to the actual situation to achieve the best transmission effect.

[0056] For example, suppose the first peer device is located in a data center with high-speed, stable network connection and strong computing power, while the second peer device is an ordinary PC in a small office far away in a foreign country. In this case, the system may set a larger fragment size (such as 50MB) for the former to fully utilize its superior hardware conditions to quickly complete data upload; while for the latter, a smaller fragment size (such as 10MB) may be selected to avoid putting too much pressure on relatively weak local resources and reduce the risk of retransmission caused by potential network fluctuations.

[0057] Finally, after determining the specific fragment transmission configuration of each end, the system will execute the fragment transmission according to the established plan. The specific steps are as follows:

[0058] Receive the file from the first peer device: According to the segmented transmission configuration information tailored for the first peer device, the server will gradually receive the data stream from the device in the form of segments. Each segment will be given a unique identifier to track the progress of the transmission and re-request lost or damaged parts when necessary.

[0059] Send files to the second peer device: Similarly, according to the fragmented transmission configuration information designed for the second peer device, the server will also send the required data to the target device in fragments. During this process, the system will continue to monitor the resource usage of both ends, dynamically adjust the transmission rate and other related settings, and ensure an efficient and stable transmission experience.

[0060] In summary, by comprehensively and carefully analyzing the hardware resources of the devices at both ends and their associated local environments, and flexibly configuring the fragmentation transmission parameters accordingly, the present invention can significantly improve the efficiency and reliability of large file transmission across devices, and is particularly suitable for application scenarios involving multi-party collaboration and wide geographical distribution. This improvement not only helps to reduce the burden on a single device, but also better adapts to complex and changing network environments, thereby providing users with smoother data exchange services.

[0061] In one embodiment, Figure 2 This specification also provides a data transmission device, which is applied to an object storage service end, and the device includes: a first module, which is used to obtain file attribute information of a file to be transmitted, and if the file attribute information meets a preset condition, segmented transmission is enabled; a second module, which is used to obtain hardware resource information, and configure configuration information of segmented transmission according to the hardware resource information, and the configuration information at least includes the segment size; a third module, which is used to perform segmented transmission according to the configuration information of segmented transmission; the file to be transmitted is a file to be sent by a peer device and / or a file to be obtained by a peer device, and the segmented transmission includes receiving a file from a segment of a peer device and / or sending a file to a segment of a peer device.

[0062] In one embodiment, the file to be transmitted is a file to be sent by a peer device; the file attribute information of the file to be transmitted is obtained, and if the file attribute information satisfies a preset condition, segmented transmission is enabled, including: obtaining judgment information that the file size of the file to be transmitted exceeds a preset threshold, and enabling segmented transmission in response to a segmented transmission request of the peer device, wherein the judgment information is generated when the peer device monitors that the cache data associated with the file to be transmitted is greater than a preset threshold.

[0063] In one embodiment, the hardware resource information is obtained, and the configuration information of the fragmented transmission is configured according to the hardware resource information. The configuration information at least includes the fragment size, including: configuring the upper limit of the fragment size of the fragmented transmission according to the network status information, CPU usage, and memory usage of the current local device and the peer device, and selecting the fragment size within the determined upper limit range of the fragment size according to the file attribute information of the file to be transmitted.

[0064] In one implementation, the file to be transmitted is a file to be sent by the first peer device and is a file to be acquired by the second peer device.

[0065] In one embodiment, the obtaining of hardware resource information and configuring the configuration information for segmented transmission according to the hardware resource information, wherein the configuration information at least includes the segment size, include: respectively obtaining local hardware resource information and the hardware resource information of the first peer device, as well as local hardware resource information and the hardware resource information of the second peer device associated with the second peer device, and respectively configuring the configuration information for segmented transmission associated with the first peer device and the second peer device, wherein the segment size associated with the first peer device and the segment size associated with the second peer device are the same or different; the performing of segmented transmission according to the configuration information for segmented transmission includes: receiving the file to be transmitted from the first peer device in a segmented transmission manner according to the configuration information for segmented transmission associated with the first peer device, and sending the file to be transmitted to the second peer device in a segmented transmission manner according to the configuration information for segmented transmission associated with the second peer device.

[0066] In one implementation, the data transmission method is further refined, with particular attention paid to data transmission between two peer devices: the first peer device is the sender of data, and the second peer device is the receiver of data. This method not only considers the performance and network conditions of a single device, but also comprehensively considers the interaction between the two devices to achieve more efficient and stable data transmission.

[0067] The file to be transferred has a dual role: it is a file to be sent by the first peer device and a file to be obtained by the second peer device. This setting makes the data transmission process more flexible and can adapt to different application scenarios, such as cloud storage synchronization, file sharing services, etc.

[0068] In order to configure the fragmented transmission more accurately, the system not only obtains the local hardware resource information, but also obtains the hardware resource information related to the first peer device and the second peer device. This includes key parameters such as CPU usage, memory usage, network bandwidth, etc. Through this dual acquisition, the system can more comprehensively understand the resource status of the devices at both ends and provide a more accurate configuration basis for fragmented transmission.

[0069] Based on the acquired hardware resource information, the system will configure the configuration information of the fragment transmission for the first peer device and the second peer device respectively. This includes but is not limited to the setting of the fragment size, wherein the fragment size of the first peer device and the fragment size of the second peer device can be the same, or they can be different according to their respective resource conditions. This differentiated configuration allows the system to optimize the transmission efficiency according to the specific conditions of each device and reduce resource waste.

[0070] When performing segmented transmission, the system will receive the file to be transmitted from the first peer device in segmented transmission according to the configuration information related to the first peer device; at the same time, according to the configuration information related to the second peer device, the system will send the file to be transmitted to the second peer device in segmented transmission. This two-way segmented transmission mechanism ensures the efficient flow of data between the two devices, while reducing transmission interruptions and retries caused by network fluctuations or device performance differences.

[0071] Assume that an enterprise needs to synchronize a large database file from one data center (first peer device) to another remote data center (second peer device). The system first obtains the hardware resource information of the two data centers, including their respective CPU usage, memory usage, and network bandwidth.

[0072] Based on this information, the system configures different shard sizes for the two data centers. For example, if the network bandwidth of the local data center is high, the system may configure a larger shard size to improve transmission efficiency; if the network bandwidth of the remote data center is low, the system may configure a smaller shard size to reduce network congestion.

[0073] During the transmission process, the system receives files in fragments from the local data center and sends them in fragments according to the configuration information of the remote data center. In this way, even in unstable network conditions, only the failed fragments need to be retransmitted instead of the entire file, which significantly improves the efficiency and reliability of data synchronization.

[0074] In one embodiment, Figure 3 By sharding large objects and dynamically adjusting the shard size, the shard adjustment algorithm calculates the shard size based on the weights of network bandwidth, CPU usage, memory usage, and object size. This can effectively improve network bandwidth utilization, reduce local memory pressure, and reduce data packet retransmission traffic caused by network problems. This sharding strategy optimizes the overall transmission process by splitting large data objects into smaller fragments for transmission, so that each fragment can be cached and retransmitted independently.

[0075] The business components include three modules: SBU-DIR (management service), SBU-FD (client), and SBU-SD (storage engine).

[0076] SBU-DIR (Management Services): Responsible for the coordination and management of the overall work, including task scheduling, progress monitoring, and exception handling. It is the control center of the entire system, ensuring the coordination and smooth operation of various modules.

[0077] SBU-FD (client): As a client module, it is responsible for data sending and receiving operations. This module handles data upload and download in pieces, and dynamically adjusts according to the client machine performance and related network environment.

[0078] SBU-SD (Storage Engine): Responsible for the actual storage and retrieval of data. It is the core module of data persistence and integrates advanced storage technologies to support efficient data access and management.

[0079] Data services refer to a collection of services that can provide data access and storage, whether it is a local storage device, cloud storage service or a third-party data provider. All services that can perform data access operations belong to the category of data services. This abstract layer simplifies the access and management of different data sources through unified interfaces and standardized operations, allowing the system to flexibly adapt to various storage needs.

[0080] The object data archiving and restoration process starts with the initiation of SBU-DIR (management service), which retains the complete information required for all tasks, including the specific details and configuration of the tasks. When starting the process, SBU-DIR passes the task-related information to SBU-FD (client) to start the actual data processing process.

[0081] Once the task information is received, SBU-FD (client) will first try to establish a stable connection with SBU-SD (storage engine). The creation of this connection is an important prerequisite for the success of the task, because it ensures that data can be smoothly transmitted between the local cache and the storage engine. After confirming that SBU-SD (storage engine) is ready, SBU-FD begins to access data through OBS (object storage service). It is responsible for pulling the required object data and deciding whether to adopt a sharding strategy as needed. For large object data, the sharding strategy can help process data in stages, thereby effectively managing memory usage and reducing the pressure during data processing.

[0082] When data is downloaded to the memory, the system processes it as a data stream that is format-independent. This approach ensures flexibility in data formats and allows SBU-SD to perform unified storage processing regardless of the format of the data source.

[0083] In the multi-slice upload scenario, SBU-FD (client) starts to fetch partial data of the file from SBU-SD (storage engine). The fetched data is first cached in local memory for subsequent multi-slice processing and upload operations. During the caching process, SBU-FD continuously monitors the amount of cached data. When the cached data exceeds the set minimum slice threshold, or the complete data of the file has been fetched, SBU-FD creates and initiates a multi-slice upload request. Each data slice is assigned a unique identifier when it is uploaded to track the upload status and manage subsequent operations. When a slice upload is completed, SBU-FD verifies the status. This includes checking the response of the upload operation to ensure that the slice is correctly transferred to the target storage, so the identifier returned by each upload operation needs to be recorded and verified. After all slice upload operations are completed, SBU-FD initiates a slice upload completion request to summarize and confirm the upload status of the entire file. After verifying the upload of all slices, if any slice is detected to have failed or erroneous behavior, the system will trigger a rollback operation. This fallback may involve re-uploading the failed shards.

[0084] In the shard download scenario, SBU-FD (client) first initiates a request to OBS (object storage service) to calculate the current shard size to be obtained based on network bandwidth, CPU usage, memory usage, and the size of the object itself. This step usually involves calculating and locating the specific data blocks that need to be captured as needed to ensure the accuracy and effectiveness of data capture. Once a response is received from OBS, SBU-FD temporarily caches these data blocks in local memory. The cache operation is used to manage memory usage and ensure that the data has good continuity and integrity before being sent to downstream components. After caching, SBU-FD sends the data blocks to SBU-SD (storage engine). This process requires an efficient data transmission strategy to minimize latency and the possibility of data loss. After receiving the data block from SBU-FD, SBU-SD immediately persists it to local storage. This operation ensures that the data is securely stored on disk to prevent data loss and other possible data inconsistencies. When the data has been successfully written to disk, SBU-SD sends a confirmation notification back to SBU-FD. This feedback process is critical because it provides the SBU-FD with a verification signal that the data transmission and storage was successful, and allows the client to continue processing and transmitting the next data block.

[0085] The above implementation method maintains the ability to quickly respond to the upload and download of small objects. By optimizing the system architecture and transmission strategy, it achieves multi-faceted performance improvement in the upload and download process of large objects. By introducing the fragment transmission mechanism, large objects are divided into multiple small fragments for step-by-step processing, and dynamically adjusted according to the client physical environment and the object itself to achieve better transmission effect. This approach significantly reduces the memory consumption of SBU-FD (client) when transmitting large data, avoiding memory overflow or performance bottleneck caused by loading too much data at a time. The fragment strategy not only reduces the amount of data requested at a single time, but also shortens the processing time of the request. In this way, the data of each fragment can quickly respond to the transmission request, reduce the waiting time between the client and the server, and improve the overall transmission efficiency. In the case of unstable network, the fragment transmission only needs to retry the failed fragment without retransmitting the entire object. This significantly reduces the retransmission cost, saves bandwidth and time, and improves the robustness of the system. The above improvement measures lay a good foundation for subsequent business optimization. In particular, further exploration can be made such as concurrently requesting large object fragment downloads and transmitting object fragment data to SBU-SD in a pipeline manner to fully utilize the bandwidth capacity of storage hardware devices. These optimization solutions can not only improve transmission throughput, but also further improve the overall performance of the system.

[0086] In one embodiment, this specification provides an electronic device, including a processor and a readable storage medium, wherein the readable storage medium stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the aforementioned data transmission method. From a hardware level, the hardware architecture diagram can be seen in Figure 4 shown.

[0087] In one embodiment, the present specification provides a readable storage medium, wherein the readable storage medium stores machine executable instructions, and when the machine executable instructions are called and executed by a processor, the machine executable instructions prompt the processor to implement the aforementioned data transmission method.

[0088] Here, the readable storage medium can be any electronic, magnetic, optical or other physical storage device that can contain or store information, such as executable instructions, data, etc. For example, the readable storage medium can be: RAM (Radom Access Memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard disk drive), solid state drive, any type of storage disk (such as CD, DVD, etc.), or similar storage medium, or a combination thereof.

[0089] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device or a combination of any of these devices.

[0090] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0091] It will be appreciated by those skilled in the art that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may be in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may be in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0092] This specification is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0093] Moreover, these computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0094] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0095] It will be appreciated by those skilled in the art that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may be in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may be in the form of a computer program product implemented on one or more computer-usable storage media (which may include but are not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0096] The above description is only an embodiment of the present specification and is not intended to limit the present specification. For those skilled in the art, the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims of the present specification.

Claims

1. A data transmission method, characterized in that: Applied to an object storage service end, the method includes: Obtain the file attribute information of the file to be transmitted, and if the file attribute information meets the preset conditions, enable fragment transmission; Acquire hardware resource information, and configure configuration information of slice transmission according to the hardware resource information, wherein the configuration information at least includes the slice size; Execute fragment transmission according to the configuration information of fragment transmission; The file to be transmitted is a file to be sent by the peer device and / or a file to be acquired by the peer device, and the segmented transmission includes receiving a file segment from the peer device and / or sending a file segment to the peer device.

2. The method according to claim 1, characterized in that The file to be transmitted is a file to be sent by the peer device; The acquiring of file attribute information of the file to be transmitted and enabling fragment transmission if the file attribute information satisfies a preset condition includes: Obtain judgment information that the file size of the file to be transmitted exceeds a preset threshold, and enable fragmented transmission in response to a fragmented transmission request of the peer device, wherein the judgment information is generated when the peer device monitors that the cache data associated with the file to be transmitted is larger than the preset threshold.

3. The method according to claim 1, characterized in that The acquiring of hardware resource information and configuring the configuration information of the slice transmission according to the hardware resource information, wherein the configuration information at least includes the slice size, comprises: Configure the upper limit of the fragment size for fragment transmission based on the network status information, CPU usage, and memory usage of the current local device and the peer device. Select the fragment size within the determined upper limit range based on the file attribute information of the file to be transmitted.

4. The method according to claim 1, characterized in that The file to be transmitted is a file to be sent by the first peer device and is a file to be acquired by the second peer device.

5. The method according to claim 4, characterized in that The acquiring of hardware resource information and configuring the configuration information of the slice transmission according to the hardware resource information, wherein the configuration information at least includes the slice size, comprises: Respectively obtain local hardware resource information and hardware resource information of the first peer device, as well as local hardware resource information and hardware resource information of the second peer device, and respectively configure configuration information of the fragment transmission associated with the first peer device and the second peer device, wherein the fragment size associated with the first peer device and the fragment size associated with the second peer device are the same or different; The performing the fragment transmission according to the configuration information of the fragment transmission includes: According to the configuration information of the fragmented transmission associated with the first peer device, the file to be transmitted is received from the first peer device in the form of fragmented transmission, and according to the configuration information of the fragmented transmission associated with the second peer device, the file to be transmitted is sent to the second peer device in the form of fragmented transmission.

6. A data transmission device, characterized in that: Applied to an object storage service end, the device comprises: The first module is used to obtain file attribute information of the file to be transmitted, and if the file attribute information meets the preset conditions, enable fragment transmission; The second module is used to obtain hardware resource information and configure the configuration information of the slice transmission according to the hardware resource information, wherein the configuration information at least includes the slice size; The third module is used to perform fragment transmission according to the configuration information of fragment transmission; The file to be transmitted is a file to be sent by the peer device and / or a file to be acquired by the peer device, and the segmented transmission includes receiving a file segment from the peer device and / or sending a file segment to the peer device.

7. The device according to claim 6, characterized in that The file to be transmitted is a file to be sent by the peer device; The acquiring of file attribute information of the file to be transmitted and enabling fragment transmission if the file attribute information satisfies a preset condition includes: Obtain judgment information that the file size of the file to be transmitted exceeds a preset threshold, and enable fragmented transmission in response to a fragmented transmission request of the peer device, wherein the judgment information is generated when the peer device monitors that the cache data associated with the file to be transmitted is larger than the preset threshold.

8. The device according to claim 6, characterized in that The acquiring of hardware resource information and configuring the configuration information of the slice transmission according to the hardware resource information, wherein the configuration information at least includes the slice size, comprises: Configure the upper limit of the fragment size for fragment transmission based on the network status information, CPU usage, and memory usage of the current local device and the peer device. Select the fragment size within the determined upper limit range based on the file attribute information of the file to be transmitted.

9. The device according to claim 6, characterized in that The file to be transmitted is a file to be sent by the first peer device and is a file to be acquired by the second peer device.

10. The device according to claim 9, characterized in that The acquiring of hardware resource information and configuring the configuration information of the slice transmission according to the hardware resource information, wherein the configuration information at least includes the slice size, comprises: Respectively obtain local hardware resource information and hardware resource information of the first peer device, as well as local hardware resource information and hardware resource information of the second peer device, and respectively configure configuration information of the fragment transmission associated with the first peer device and the second peer device, wherein the fragment size associated with the first peer device and the fragment size associated with the second peer device are the same or different; The performing the fragment transmission according to the configuration information of the fragment transmission includes: According to the configuration information of the fragmented transmission associated with the first peer device, the file to be transmitted is received from the first peer device in the form of fragmented transmission, and according to the configuration information of the fragmented transmission associated with the second peer device, the file to be transmitted is sent to the second peer device in the form of fragmented transmission.

11. An electronic device, characterized in that: include: A processor and a readable storage medium, wherein the readable storage medium stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement any method described in claims 1-5.

12. A readable storage medium, characterized in that: The readable storage medium stores machine executable instructions, and when the machine executable instructions are called and executed by a processor, the machine executable instructions prompt the processor to implement any one of the methods of claims 1-5.

Images