Mirror image uploading method and device, electronic equipment and storage medium

Abstract

The invention relates to a mirror image uploading method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a to-be-uploaded container mirror image, and splitting the container mirror image into a plurality of mirror image layers according to the functions of internal data; extracting mirror image features of each mirror image layer, and matching a compression algorithm for the corresponding mirror image layer according to each mirror image feature; and performing compression processing on the matched mirror image layers in parallel by adopting each compression algorithm, and uploading the compressed mirror image layers to a cloud mirror image library in parallel. According to the invention, the mirror image uploading efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a mirror upload method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the rapid development of cloud computing, containerization, and virtualization technologies, container image technologies such as Docker and Kubernetes have become the core support for the deployment and operation of microservice architectures. As a standardized packaging carrier for applications and their runtime environments, the upload efficiency of container images directly determines the speed of application release and iteration, as well as the cost of operation and maintenance. Especially in microservice scenarios with frequent iterations and resource-constrained edge computing scenarios, the importance of the image upload process is becoming increasingly prominent.

[0003] Currently, the mainstream image upload solutions in the industry generally adopt a mode of overall compression followed by serial transmission. That is, the complete container image is first packaged as a whole, then compressed using traditional compression algorithms, and finally the compressed package is uploaded to the cloud image repository. Among them, gzip is the most widely used traditional compression algorithm. The core advantage of this algorithm is that it has a high compression ratio, which can reduce the image transmission size to a certain extent. Therefore, it has long been regarded as the preferred solution for image transmission compression.

[0004] However, as container image sizes continue to increase, the limitations of traditional compression algorithms, such as gzip, have become increasingly apparent. The design logic of the gzip algorithm prioritizes compression ratio over processing speed. When dealing with large images, its compression time increases exponentially. The lengthy compression process directly lengthens the pre-upload waiting period, ultimately leading to low overall image upload efficiency. Summary of the Invention

[0005] This application provides a mirror upload method, apparatus, electronic device, and storage medium to solve the problem of low overall mirror upload efficiency.

[0006] Firstly, this application provides a mirror upload method, the method comprising: Obtain the container image to be uploaded, and split the container image into multiple image layers according to the function of the internal data; Extract the image features of each image layer, and match the compression algorithm to the corresponding image layer based on each image feature; Various compression algorithms are used to perform compression processing on the matched image layers in parallel, and the compressed image layers are then uploaded to the cloud image library in parallel.

[0007] Optionally, extracting the mirror features of each mirror layer and matching compression algorithms to the corresponding mirror layer based on each mirror feature includes: Extract the mirror features of the mirror layer; Multiple triggering conditions are sorted in descending order of priority, with each triggering condition pre-bound to a compression algorithm; The mirror feature is verified using each trigger condition according to the sorting until a target trigger condition that passes the verification of the mirror feature is determined. The compression algorithm that is already bound to the target trigger condition is matched to the image layer.

[0008] Optionally, the mirrored features are verified using each trigger condition according to the aforementioned sorting until the target trigger condition that verifies the mirrored features is determined to be valid includes: The triggering condition with the highest priority is selected as the triggering condition to be verified, in descending order of priority. If the mirror feature passes the verification of the trigger condition to be verified, then the trigger condition to be verified is determined as the target trigger condition; If the mirror feature fails the verification of the trigger condition to be verified, the next priority trigger condition to be verified is selected as the new trigger condition to be verified, and the mirror feature is verified using the new trigger condition to be verified until the target trigger condition that verifies the mirror feature passes the verification is obtained.

[0009] Optionally, various compression algorithms are used to perform compression processing on the matched image layers in parallel, and the compressed image layers are uploaded to the cloud image library in parallel, including: Based on the compression algorithm matching each image layer, an independent thread group is allocated to each image layer. Each thread group performs compression and upload operations in parallel, and each thread group includes at least one compression thread and at least one upload thread. The data of the corresponding mirror layer is divided into fixed-size fragments; The at least one compression thread sequentially performs pipelined compression processing on each fragment, and the compressed fragments are stored in the buffer queue in real time. The at least one upload thread extracts compressed fragments from the buffer queue in real time and uploads them concurrently to the cloud image library using multiplexing with fragments extracted by other upload threads.

[0010] Optionally, various compression algorithms are used to perform compression processing on the matched image layers in parallel, and the compressed image layers are uploaded to the cloud image library in parallel, including: Based on the compression algorithm matching each image layer, an independent thread group is allocated to each image layer. Each thread group performs compression and upload operations in parallel, and each thread group includes at least one compression thread and at least one upload thread. The data of the corresponding mirror layer is divided into fixed-size fragments; The at least one compression thread performs pipelining compression on a portion of the data segment of the fragment, outputting continuous compressed data segments in real time and storing them in a buffer queue, wherein the data size of the compressed data segment is smaller than the data size of the fragment; The upload thread extracts the compressed data segment from the buffer queue in real time and uploads it concurrently to the cloud image library with the compressed data segment extracted by other upload threads through multiplexing.

[0011] Optionally, during the concurrent upload to the cloud image library, the method further includes: Real-time monitoring of network transmission performance during the upload process; If the network transmission performance is detected to be lower than a preset threshold, the current initial compression algorithm is switched to a set compression algorithm, and the set compression algorithm is used to compress the uncompressed fragments or data segments. If the network transmission performance is detected to recover to or above the preset threshold, then switch to the initial compression algorithm to complete the compression of the remaining data.

[0012] Optionally, before extracting the mirror features of each mirror layer, the method further includes: If this is not the first time the container image has been uploaded, then determine the hash value of each image layer; The hash values ​​of each image layer are compared with the set of hash values ​​in the cloud image library to filter out image layers whose hash values ​​do not exist in the set of hash values. The set of hash values ​​contains the hash values ​​of each image layer that have been stored in the cloud.

[0013] Secondly, this application provides a mirror upload device, the device comprising: The acquisition module is used to acquire the container image to be uploaded and split the container image into multiple image layers according to the function of the internal data. The matching module is used to extract the image features of each image layer and match the compression algorithm for the corresponding image layer based on the image features. The compression and upload module is used to perform parallel compression processing on the matching image layers using various compression algorithms, and then upload the compressed image layers to the cloud image library in parallel.

[0014] Thirdly, this application provides an electronic device, comprising: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor connected to the at least one bus; and at least one memory connected to the at least one bus.

[0015] Fourthly, this application also provides a computer storage medium storing computer-executable instructions for executing the image upload method described in any of the preceding claims of this application.

[0016] Compared with the prior art, the technical solution provided in this application has the following advantages: After splitting the container image into multiple image layers according to the function of the internal data, a compression algorithm is matched to the corresponding image layer based on the image characteristics of each image layer. By adapting the processing requirements of each image layer with a targeted compression algorithm, the problem of insufficient adaptability of a single compression algorithm is avoided, and the compression efficiency of a single image layer is accelerated. Then, a multi-image-layer parallel compression method is adopted to allow the compression operations of each layer to proceed synchronously, further reducing the overall compression time. Finally, the concurrent transmission of compressed data is achieved through a parallel upload mechanism, which improves the data transmission efficiency. This application improves the image upload efficiency by combining the selection of compression algorithms, parallel compression, and parallel upload. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0020] Figure 1 A schematic diagram of the image upload system provided in this application embodiment; Figure 2 A flowchart of a mirror upload method provided in this application embodiment; Figure 3 A schematic diagram illustrating the binding relationship between various triggering conditions and compression algorithms provided in the embodiments of this application; Figure 4 A schematic diagram of the image features and matching compression algorithms of each image layer provided in the embodiments of this application; Figure 5 This is a schematic diagram illustrating the overall process of image upload provided in an embodiment of this application; Figure 6This is a schematic diagram of the structure of a mirror upload device provided in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] The following disclosure provides numerous different embodiments or examples for implementing various structures of this application. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of this application. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0023] Optionally, in the embodiments of this application, the above-described mirror upload method can be applied to, for example... Figure 1 The hardware environment shown consists of terminal 101 and cloud 103. For example... Figure 1 As shown, the cloud 103 is connected to the terminal 101 via a network. The terminal 101 is used to split the container image into multiple image layers, perform parallel compression processing on the corresponding image layers using the compression algorithm corresponding to each image layer, and upload the compressed image layers to the cloud. A compression algorithm library 105 can be set on the terminal to provide data storage services for the terminal 101. The aforementioned network includes, but is not limited to, a wide area network, a metropolitan area network, or a local area network. The terminal 101 includes, but is not limited to, a local physical server, a virtual machine, a container client, or a local node of a microservice cluster, etc.

[0024] The following will describe in detail a mirror upload method provided in this application embodiment, taking a terminal application as an example, with specific implementation details. Figure 2 As shown, the specific steps are as follows: Step 201: Obtain the container image to be uploaded, and split the container image into multiple image layers according to the function of the internal data; Step 202: Extract the image features of each image layer, and match the compression algorithm to the corresponding image layer based on each image feature; Step 203: Perform compression processing on the matched image layers in parallel using various compression algorithms, and upload the compressed image layers to the cloud image library in parallel.

[0025] In step 201, when the terminal initiates the container image upload process, it first performs a functional-dimensional splitting operation on the entire container image to be uploaded. Specifically, the terminal first identifies the internal data of the container image and, based on the functional attributes carried by the data, splits the container image into three types of image layers: the operating system layer, the application dependency layer, and the log layer. The operating system layer contains data such as the kernel and system basic libraries that support application operation; the application dependency layer contains third-party components and runtime environment necessary for business programs; and the log layer stores various log data generated by the container operation.

[0026] In step 202, the terminal is pre-configured with a set of trigger conditions ordered by a preset priority, and each trigger condition is bound to a specific compression algorithm. After the feature extraction of each image layer is completed, the terminal will verify and compare the features of the image layer with each trigger condition in descending order of priority, until the trigger condition that has been successfully verified is selected, and the compression algorithm bound to the trigger condition is determined as the exclusive compression algorithm for that image layer.

[0027] Among them, the characteristics of the mirror layer include the layer file size, the proportion of binary files, the proportion of text files, and the file modification frequency.

[0028] The triggering conditions are as follows: the size of the layer file exceeds the preset size threshold + the proportion of binary files exceeds the first proportion threshold; the proportion of text files exceeds the second proportion threshold; and the file modification frequency exceeds the preset frequency threshold.

[0029] For example, the operating system layer is typically large and remains static for extended periods, rarely undergoing version or content updates. Therefore, the characteristics of an operating system layer image include a total file size exceeding 2GB, a binary file ratio exceeding 95%, and no content changes in the past 30 days. If the operating system layer's characteristics match the trigger condition of a file size exceeding 100MB or a binary file ratio >80%, the terminal will match the operating system layer with the zstd (Zstandard) standard algorithm based on the binding relationship.

[0030] The log layer is relatively small but generates new data frequently. Characteristics of the log layer include a total file size of less than 1MB, 60% of which are text files, and more than 10 updates per day. When a log layer's characteristics match the trigger condition of a file size <1MB or a file modification frequency >5 times / day, the terminal matches the LZ4 (LZ4 Compression Algorithm) compression algorithm to the log layer based on the binding relationship.

[0031] The application dependency layer is of medium size and has a relatively stable update frequency. Its mirror characteristics include a total file size between 20-90MB, a text file ratio exceeding 80%, and an average monthly update frequency of only once. When the application dependency layer's characteristics match the trigger condition of a text file ratio >70%, the terminal matches the application dependency layer with the default zstd algorithm plus a compression dictionary based on the binding relationship.

[0032] This application configures a dedicated compression algorithm for each image layer that is adapted to its image characteristics, abandoning the one-size-fits-all compression strategy of traditional solutions. Since different image layers differ in characteristics such as layer file size, file modification frequency, and the proportion of different file types, targeted algorithm adaptation can accurately match the processing needs of each image layer. Specifically, it matches compression logic adapted to the data structure of large-volume image layers, reducing invalid encoding steps and accelerating processing; it matches lightweight and high-speed compression algorithms for small-volume, frequently updated layers, achieving millisecond-level fast processing; and it matches dedicated encoding schemes for image layers with different file types, avoiding processing delays caused by format incompatibility, thus eliminating the time-consuming waste caused by algorithm mismatch at its root.

[0033] In step 203, the terminal allocates independent thread groups for the operating system layer, application dependency layer, and log layer, allowing the three types of image layers to perform compression operations in parallel based on their respective matching compression algorithms. For example, the operating system layer uses the zstd standard algorithm for compression, the application dependency layer uses a combination of the zstd default algorithm and a compression dictionary to balance compression speed and data volume, and the log layer's thread group uses the LZ4 algorithm to achieve millisecond-level compression. This parallel compression mode for multiple image layers ensures that the compression operations of each image layer do not block each other and proceed synchronously, shortening the compression time compared to traditional overall compression.

[0034] After data compression of each image layer is completed, the terminal immediately initiates a parallel upload mechanism. The upload modules within each thread group synchronously read the compressed data of the corresponding image layer and, relying on the HTTP / 2 multiplexing protocol, integrate the compressed data from multiple image layers into a single connection for concurrent transmission to the cloud image library. This protocol can carry multiple transmission streams simultaneously within a single connection, eliminating the extra time required for establishing multiple connections and maximizing network bandwidth utilization. This solves the resource waste problem of a single stream consuming all bandwidth while other streams queue in traditional serial uploads. The implementation of the parallel upload mechanism enables rapid batch transmission of compressed data, significantly improving the overall efficiency of the transmission process.

[0035] This application constructs a highly efficient container image upload system through the synergy of a dedicated compression algorithm, parallel compression, and parallel upload. The dedicated compression algorithm matches the optimal compression scheme for each image layer from the source, reducing the compression redundancy of a single compression algorithm; parallel compression reduces the total compression time; and parallel upload accelerates data transmission efficiency. The combined effect of these three factors significantly improves the overall upload efficiency of container images compared to traditional solutions. Actual testing shows that this application reduces end-to-end latency during the image cold start phase by 9%-16%.

[0036] In this application, the container image is split into multiple image layers according to the function of its internal data. A compression algorithm is matched to the corresponding image layer based on its image characteristics. By adapting targeted compression algorithms to the processing needs of each image layer, the problem of insufficient adaptability of a single compression algorithm is avoided, thus accelerating the compression efficiency of a single image layer. Then, a multi-image-layer parallel compression method is adopted, allowing the compression operations of each layer to proceed synchronously, further reducing the overall compression time. Finally, a parallel upload mechanism is used to achieve concurrent transmission of compressed data, improving data transmission efficiency. This application improves image upload efficiency by combining the selection of compression algorithms, parallel compression, and parallel upload.

[0037] As an optional real-time approach, in step 202, the image features of each image layer are extracted, and a compression algorithm is matched to the corresponding image layer based on each image feature, including: Step S11: Extract the mirror features of the mirror layer; Step S12: Sort multiple triggering conditions in descending order of priority, wherein each triggering condition is pre-bound to a compression algorithm; Step S13: Verify the mirror feature using each trigger condition according to the sorting until the target trigger condition that passes the verification of the mirror feature is determined; Step S14: Match the target trigger condition to the compression algorithm bound to the image layer.

[0038] The terminal extracts features from each of the split mirror layers. These features include layer file size, file type proportion, and file modification frequency. Layer file size refers to the total data volume of all files within that mirror layer; file type proportion refers to the percentage of the most important files within the mirror layer, such as the proportion of binary files or text files; and file modification frequency is the number of content changes within the most recent period.

[0039] The terminal pre-stores multiple trigger conditions bound to compression algorithms. Based on the processing requirements of the image layer, the terminal sorts these trigger conditions from highest to lowest priority. The sorting is based on the degree of correlation between the trigger condition and the business attributes of the image layer. The priority of the trigger conditions and the compression algorithms bound to each trigger condition in this application are shown below.

[0040] Set the trigger condition that the file size exceeds 100MB or the binary file ratio is >80% to the highest priority. The compression algorithm corresponding to this trigger condition is the zstd standard algorithm.

[0041] Triggering conditions with a layer file size <1M or a file modification frequency >5 times / day are given high priority, and the compression algorithm corresponding to this triggering condition is the LZ4 algorithm.

[0042] Set the trigger condition where the text file percentage is >70% as the second highest priority, and the compression algorithm corresponding to this trigger condition is the zstd default algorithm + compression dictionary.

[0043] Set the trigger conditions for other cases to the lowest priority, and the compression algorithm corresponding to the trigger condition is the default zstd algorithm.

[0044] The terminal verifies the mirror feature using various trigger conditions in a sorted order until a target trigger condition that passes the verification of the mirror feature is determined. These trigger conditions include: selecting the highest priority trigger condition as the trigger condition to be verified in descending order of priority; if the mirror feature passes the verification of the trigger condition to be verified, then the trigger condition to be verified is determined as the target trigger condition; if the mirror feature fails the verification of the trigger condition to be verified, then the next priority trigger condition is selected as the new trigger condition to be verified, and the new trigger condition to be verified is used to verify the mirror feature until a target trigger condition that passes the verification of the mirror feature is determined.

[0045] The terminal retrieves multiple trigger conditions from the database, sorted by priority from highest to lowest. It then identifies the highest-priority trigger condition and designates it as the first to be verified. The terminal uses this trigger condition to verify the image characteristics of the image layer. If the image characteristic verification passes, the trigger condition is designated as the target trigger condition. For example, if the image characteristics of the operating system layer meet the trigger condition that the file size exceeds 100MB, the operating system layer will use the zstd standard algorithm corresponding to that trigger condition for compression.

[0046] If the mirrored features fail the verification of the pending trigger conditions, the terminal switches the pending trigger conditions to the next higher priority trigger conditions and uses the new pending trigger conditions to verify the mirrored features. For example, if the log layer does not meet the first pending trigger condition (i.e., the application dependency layer does not meet the requirement of file size exceeding 100MB or binary file ratio >80%), then the terminal uses the next higher priority trigger condition (i.e., the judgment layer file size <1MB or file modification frequency >5 times / day). If the mirrored features of the log layer meet this trigger condition, then the log layer uses the LZ4 algorithm corresponding to this trigger condition for compression.

[0047] Similarly, if the application dependency layer does not meet the highest priority triggering condition and the high priority triggering condition, that is, the image characteristics of the application dependency layer do not meet any of the following: file size exceeding 100MB, binary file ratio >80%, layer file size <1M, file modification frequency >5 times / day, then the next highest priority triggering condition is used for verification, that is, text file ratio >70%. If the image characteristics of the application dependency layer meet this triggering condition, then the application dependency layer uses the zstd compression + compression dictionary algorithm corresponding to this triggering condition for compression.

[0048] Figure 3 This diagram illustrates the binding relationship between various triggering conditions and the compression algorithm. From... Figure 3 The priority order of each triggering condition and the compression algorithm bound to each triggering condition can be seen.

[0049] The compression algorithms selected in this application are determined based on their characteristics, which are shown below.

[0050] zstd standard algorithm: suitable for situations where the layer file is large or the binary file accounts for a high proportion. The zstd standard algorithm is 8%-12% more efficient than the gzip algorithm.

[0051] LZ4 algorithm: For frequently updated layers and smaller layer files, such as logs and database files, speed is prioritized, and the compression throughput can reach 500MB / s. The LZ4 algorithm is 200% more efficient than the gzip algorithm.

[0052] zstd default algorithm + compression dictionary: suitable for text-intensive layers, such as configuration files and code libraries. When text files account for ≥70%, compression dictionary optimization is enabled, and the compression ratio is improved to 15%-20%.

[0053] While the default zstd algorithm strikes a good balance between compression ratio and processing speed, it has shortcomings when dealing with small files or text-intensive data: processing small files generates independent compression metadata for each file, resulting in an excessively high metadata content; processing text-intensive data fails to adequately identify frequently repeated segments within a specific context, leading to overall low compression efficiency. However, by overlaying a compression dictionary pre-trained based on similar historical data, the default zstd algorithm prioritizes matching the data to be compressed with frequently repeated segments in the dictionary, directly replacing duplicate content with dictionary indices. This eliminates the need for redundant encoding, reducing metadata redundancy and improving the compression ratio of text-intensive data such as application dependency layers to 15%-20%.

[0054] The zstd standard and zstd default configurations are two pre-configured zstd compression settings. The zstd standard configuration includes a compression level of 6 and enables long-distance matching, while the zstd default configuration includes a compression level of 3 and disables long-distance matching. Compression level is a trade-off parameter, typically ranging from 1 to 2. Higher levels generally result in better compression ratios but slower compression speeds and higher memory and CPU consumption. Conversely, lower levels result in faster compression speeds but potentially worse compression ratios. Enabling long-distance matching is a special mode provided by zstd that allows matching duplicate data at greater distances. In ordinary compression algorithms, matching searches usually have a window limit, such as 32KB or 128KB, while long-distance matching expands this window, allowing the algorithm to discover and utilize earlier-appearing repeating patterns, even if they are far apart. The characteristics of the zstd standard are: slower compression speed, higher memory usage, but higher compression ratios, and the ability to find repeating patterns at greater distances, making it suitable for large files. zstd's default characteristics are: fast compression speed and low memory usage, but relatively low compression ratio.

[0055] Figure 4 This is a schematic diagram illustrating the compression algorithm that matches the mirror features of each mirror layer. Figure 4 As can be seen, the operating system layer has large file sizes and binary files account for >80% of the total: using the zstd standard algorithm balances compression ratio and speed. The log layer is a high-frequency update layer with files smaller than 1MB: using the LZ4 algorithm prioritizes processing speed. The application dependency layer is a text-intensive layer (text files account for >70%): using the default zstd algorithm plus a compression dictionary optimizes text processing efficiency.

[0056] In this application, after the terminal completes the matching of exclusive compression algorithms for each image layer, it initiates a collaborative processing flow of fragmentation pipeline and streaming compression upload. Specifically, the terminal allocates a dedicated independent thread group for each image layer based on the characteristics of the compression algorithm matched for each image layer, ensuring that the processing flows of different image layers do not block each other and proceed synchronously. Each thread group adopts an architecture that links compression and upload, with at least one compression thread and at least one upload thread built-in. Thread resources can be dynamically allocated according to the characteristics of the image layer.

[0057] Before initiating compression and upload, each thread group preprocesses the complete data of the corresponding image layer, dividing it into fragments according to a preset fixed size standard. The fragment size threshold is dynamically set based on the compression algorithm characteristics matched to the image layer and network bandwidth conditions. For example, for the operating system layer adapted to the zstd standard algorithm, its fragment size is set to a larger range (e.g., 5MB) to reduce the number of fragments and lower concatenation costs; for the log layer adapted to the LZ4 algorithm, its fragment size is set to a smaller range (e.g., 2MB) to speed up the transfer. At the same time, each fragment is assigned a unique fragment index to ensure that the cloud can accurately identify the data ownership and complete subsequent concatenation, providing standardized data segments for pipeline compression and streaming upload.

[0058] This application provides two implementation methods for data compression and uploading in thread groups.

[0059] In one embodiment, each fragment is sequentially compressed using at least one compression thread, and the compressed fragments are stored in a buffer queue in real time. At least one upload thread extracts the compressed fragments from the buffer queue in real time, and uploads them concurrently to the cloud image library using multiplexing and other upload threads.

[0060] The compression thread does not wait for all fragments of the entire image layer to complete processing. Instead, it performs compression operations sequentially according to the fragment order. The compression process is as follows: after the first fragment enters the compression process and completes processing, it is written to the buffer queue in real time without waiting for the processing progress of subsequent fragments. While the first fragment is stored in the buffer queue, the second fragment has already entered the processing chain of the compression thread, forming a pipeline-like continuous compression closed loop. This avoids the excessive memory consumption of large image layers due to full compression and maximizes the efficiency of the compression process by enabling the compression results to quickly enter the upload-ready state through streaming processing.

[0061] For single-fragment upload failures, the terminal has a built-in automatic retry mechanism. When a failure response is detected for a fragment upload request, the retry process is automatically triggered, and the maximum number of retries for a single fragment is limited, for example, a maximum of 3 retries. If the upload still fails after 3 retries, the terminal will put the failed fragment back into the buffer queue, wait for the current batch of upload tasks to complete, and then initiate the upload again based on the network status. This reduces the fragment upload failure rate caused by occasional network fluctuations and ensures the integrity and stability of the overall upload process.

[0062] Furthermore, the upload threads in each thread group monitor the status of the buffer queue in real time. Once a compressed fragment is detected, the extraction and upload operations are immediately initiated, forming a seamless connection between compression, buffering, and upload, enabling simultaneous compression and upload. During the upload process, the terminal invokes the HTTP / 2 multiplexing protocol to concurrently transmit the fragment data of this upload thread with the fragment data of other upload threads to the cloud image library. This reduces the time spent establishing multiple connections and maximizes the utilization of network bandwidth. Compared to the traditional serial upload mode after full compression, this application transforms the serial time of compression and upload into parallel collaboration, significantly shortening the overall link time. At the same time, multiplexing avoids the waste of bandwidth resources, thereby improving upload efficiency.

[0063] To avoid network congestion caused by multi-threaded concurrent uploads, the terminal presets a maximum limit on the number of concurrent threads, for example, ≤5 concurrent threads. Simultaneously, a token bucket algorithm is introduced to implement traffic control. The token bucket generates tokens at a preset rate, and each upload thread must acquire a token before initiating a chunked upload request. When tokens are exhausted, new upload requests will enter a queue waiting state. This rate-limiting mechanism achieves a stable distribution of upload traffic, ensuring the efficiency of multi-threaded parallel uploads while avoiding network congestion or transmission timeouts caused by excessively high instantaneous traffic.

[0064] In another embodiment, at least one compression thread compresses a portion of the data segment of the fragment, outputs continuous compressed data segments in real time and stores them in a buffer queue, wherein the data volume of the compressed data segment is less than the data volume of the fragment; the upload thread extracts the compressed data segments from the buffer queue in real time and uploads them to the cloud image library concurrently with the compressed data segments extracted by other upload threads through multiplexing.

[0065] In this embodiment, the terminal controls the compression thread to further divide the data within the fragment into smaller, continuous data segments. Then, compression operations are performed on these data segments sequentially. After each data segment is compressed, the corresponding compressed data segment is output in real time and written to a buffer queue. Since the compressed data segments are generated based on the data segments within the fragment, the data volume of each segment is much smaller than the total data volume of the entire fragment. This refined processing mode significantly improves the flexibility of data flow, avoiding the long-term memory resource occupation caused by full compression of large fragments, and allowing the compression results to quickly enter the upload-ready state with smaller granularity.

[0066] The buffer queue stores these consecutive compressed data segments using a first-in, first-out (FIFO) mechanism, decoupling the compression and uploading processes and eliminating execution blockages between the two. Upload threads within each thread group continuously monitor the data entry status of the buffer queue. Without waiting for all compressed data segments of the entire shard to be generated, they immediately initiate the extraction and uploading process as soon as an extractable compressed data segment is detected in the queue. The terminal invokes the HTTP / 2 multiplexing protocol to synchronously send the compressed data segment extracted by this upload thread, along with compressed data segments extracted by other upload threads, to the cloud image library via a concurrent transmission mechanism. This collaborative design of intra-shard data segment streaming compression and multiplexed concurrent uploading achieves finer-grained compression-as-upload, transforming the serial time consumption of compression and uploading in the traditional model into parallel collaborative time consumption. This reduces the local resource consumption pressure on the terminal and significantly shortens the container image upload time.

[0067] This application achieves a significant improvement in the overall efficiency of container image compression and uploading through the synergy of layered parallel compression, streaming compression technology, and a producer-consumer model. Layered parallel compression, relying on a multi-threaded architecture, allocates independent thread groups to each image layer, allowing different image layers such as the operating system layer, application dependency layer, and log layer to perform compression processing simultaneously. This frees up multi-core computing resources on the terminal, breaking the efficiency bottleneck of traditional single-threaded serial processing for overall compression. Streaming compression technology, combined with the producer-consumer model, treats the fragmented compression stage of each image layer as a producer, outputting small-granular compressed data in real time and storing it in a buffer queue. The uploading stage, acting as a consumer, continuously retrieves compressed data from the buffer queue and performs concurrent uploads. This results in a high degree of overlap between the compression and upload processes, eliminating the pre-upload waiting period after full compression is completed. This application reduces overall compression time through layered parallel compression and achieves seamless integration of compression and uploading through the streaming linkage of the producer-consumer model, significantly reducing the total time spent uploading container images.

[0068] Furthermore, during the fragmented upload process, the terminal records the unique index information of successfully uploaded fragments in real time and persistently stores the fragment index list locally. When an abnormality such as a network interruption causes the upload process to terminate, after the network is restored, the terminal first compares the locally stored uploaded fragment index with the fragment reception records in the cloud image library to accurately identify the missing fragments that were not uploaded or failed to be uploaded. The retransmission process is only initiated for the missing fragments, without having to re-upload the entire image layer data, reducing the impact of network abnormalities on upload efficiency and saving bandwidth resources generated by repeated transmissions.

[0069] As an optional implementation, before extracting the image features of each image layer, the method further includes: if the current upload of the container image is not the first time, determining the hash value of each image layer; comparing the hash value of each image layer with the hash value set in the cloud image library, and filtering out image layers whose hash values ​​do not exist in the hash value set, wherein the hash value set contains the hash values ​​of each image layer already stored in the cloud.

[0070] After identifying each image layer of the container image, the terminal determines that this is the first time the container image has been uploaded. At this time, there is no historical upload record of the container image locally, and there is no hash data of any image layer of the container image in the hash value set of the cloud image library. The terminal directly extracts the image features of all image layers in the container image in sequence, and performs subsequent operations such as matching the dedicated compression algorithm and performing parallel compression upload to ensure that the first uploaded image can be completely and efficiently synchronized to the cloud image library.

[0071] If the terminal determines that this is not the first time the container image has been uploaded, it will initiate a process of hash calculation and comparison with the cloud for each image layer. First, the terminal will calculate the SHA-256 hash value for each image layer within the container image, using this as a unique identifier for the image layer content. Then, the terminal will call the dedicated API interface of the registry in the cloud image repository, such as GET / v2 / <name> / blobs / <digest>The system inputs the locally calculated hash values ​​of each image layer into the interface and queries the cloud image library to see if the corresponding image layer with the hash value is already stored. This process is the hash comparison mechanism. If the cloud image library returns that the image layer corresponding to the hash value already exists, the terminal does not need to perform subsequent compression and upload operations on the image layer. It can directly reference the existing image layer in the cloud in the image description file. If it returns that the image layer does not exist, it is determined that the image layer is a new layer that has not been uploaded or a modified layer.

[0072] Through the image layer identification logic, the terminal performs a full comparison of the SHA-256 hash value of each image layer with the hash value set of the cloud image library. Ultimately, only image layers whose hash values ​​do not exist in the cloud hash value set are selected as target layers for subsequent processing. For example, if a container image originally had 10 image layers, and only the content of layer 7 has been modified, the terminal will find that only the SHA-256 hash value of layer 7 does not match the cloud record after calculating the hash value. Simultaneously, due to image layer dependencies, the image manifest file needs to be regenerated. Therefore, the terminal only needs to upload the data of image layer 7 and the new image manifest file. Compared to the traditional solution of uploading all 10 image layers, incremental uploading reduces redundant transmissions, reducing data transmission volume by approximately 90%, significantly reducing bandwidth consumption and upload time, and lowering network load by 60% in microservice scenarios.

[0073] The cloud system monitors and acquires multiple fragments uploaded concurrently by the terminal via a multiplexing protocol in real time. Each fragment uploaded to the cloud carries a fragment index, which indicates both the image layer to which the fragment belongs and its order within that image layer. The cloud system iterates through all received fragments, classifying and grouping fragments from different image layers according to the image layer identifiers in the fragment indexes, ensuring that fragments from the same image layer are assigned to the same processing queue. Subsequently, for each image layer's corresponding fragment queue, the cloud sorts the fragments according to the order indicated in the fragment indexes, and then sequentially assembles and merges the fragments according to the sorting results, ultimately obtaining the complete compressed image layer.

[0074] The cloud system reads the image layer indexes of each reconstructed image layer and retrieves the corresponding original image layer from the cloud image repository. Based on the retrieval results, it then overwrites and replaces the corresponding original image layer with the newly reconstructed image layer. During the replacement process, the cloud system synchronously updates the image manifest, writing information such as the hash value and storage path of the new image layer into the manifest to ensure that users can accurately locate the updated image layer when pulling the image later. Simultaneously, the cloud system retains historical version snapshots of the original image layers to handle rollback requirements in abnormal replacement scenarios, ensuring the consistency and traceability of image data in the cloud image repository.

[0075] If the cloud receives compressed data segments, the same logic applies: based on the database index of the compressed data segments, the shards containing those segments are located and arranged in order to form the shards. Then, the image layers are reassembled based on the shard indexes.

[0076] Optionally, during the concurrent upload to the cloud image library, the method further includes: real-time monitoring of network transmission performance during the upload process; if the network transmission performance is detected to be lower than a preset threshold, switching from the current initial compression algorithm to a set compression algorithm, and using the set compression algorithm to compress the uncompressed fragments or data segments; if the network transmission performance is detected to recover to or above the preset threshold, switching back to the initial compression algorithm to complete the compression of the remaining data.

[0077] During the concurrent data upload to the cloud mirror library, the terminal monitors the network transmission performance in real time, including network bandwidth and transmission latency. If the network transmission performance is detected to be lower than a preset threshold, a switching command is immediately triggered to terminate the initial compression algorithm currently being executed on the uncompressed fragments or data segments and switch to a set compression algorithm, such as the LZ4 algorithm. The LZ4 algorithm focuses on compression and transmission speed. After the switch is completed, the set compression algorithm is used to compress the remaining uncompressed fragments or data segments. If the network transmission performance is subsequently monitored to recover to or above the preset threshold, an algorithm revert process is automatically triggered to switch back from the set compression algorithm to the initial compression algorithm to continue compressing the remaining data, ensuring that compression ratio and transmission efficiency are balanced when network resources are sufficient.

[0078] In this application, the selection of compression algorithms can be flexibly adjusted based on network transmission performance. For example, in weak network environments where bandwidth is limited and transmission is prone to interruption, the LZ4 algorithm can be prioritized. This application employs a dynamic compression algorithm selection strategy, automatically matching the optimal compression method based on mirror layer characteristics and network conditions. This avoids excessive compression leading to computational waste, optimizes CPU utilization by 20%, and reduces system hardware load while ensuring transmission performance, thus improving system robustness.

[0079] This application achieves efficient compression and stable uploading of container images based on a Layer Analyzer, Algorithm Selector, Streaming Compressor, and Upload Scheduler. The overall process strictly adheres to the technical logic of container images. Figure 5 As shown, Figure 5 The overall process of uploading to a mirror site is illustrated below, with the specific steps as follows: Step 501: Obtain the container image to be uploaded.

[0080] The terminal obtains the container image to be uploaded. This container image is a system structure containing multiple layers of files such as code, dependencies, and configuration required for the application to run.

[0081] Step 502: Split the container image into multiple image layers.

[0082] The image resolver reads the internal data information of the container image and breaks the container image down into multiple image layers according to the function of the internal data.

[0083] Step 503: Calculate the hash of each mirror layer and compare it with the cloud.

[0084] For each split image layer, the terminal generates a unique hash value and compares the hash value with the set of hash values ​​in the cloud image library. If the hash value exists in the hash value set, the upload process for that image layer is skipped; if the hash value does not exist in the hash value set, the image feature extraction process begins.

[0085] Step 504: Extract mirror features and match compression algorithms for the corresponding mirror layers.

[0086] The mirror parser extracts the mirror features of the mirror layer, while the algorithm selector obtains multiple trigger conditions sorted by priority from high to low. The algorithm selector verifies the mirror features based on each sorted trigger condition and determines the target trigger condition that has been successfully verified. Then, it matches the compression algorithm that the target trigger condition is bound to with the mirror layer.

[0087] Operating system layer: The zstd standard algorithm is selected, which can balance compression ratio and speed.

[0088] Application dependency layer: It adopts the default algorithm of zstd + compressed dictionary, which can optimize text efficiency.

[0089] Log layer: Uses the LZ4 algorithm to prioritize processing speed.

[0090] Step 505: Perform streaming parallel compression on each mirror layer.

[0091] The streaming compression engine adopts a producer model, which divides the data of the image layer into fixed-size fragments, and then uses at least one compression thread to perform pipeline compression processing on each fragment in sequence, and stores the compressed fragments into a buffer queue in real time.

[0092] Step 506: Upload each shard concurrently to the cloud.

[0093] The fragment upload scheduler adopts a consumer model, which combines at least one upload thread to extract compressed fragments from the buffer queue in real time, and uploads them concurrently to the cloud image library through multiplexing with fragments extracted by other upload threads.

[0094] Step 507: Cloud verification and image layer reconstruction.

[0095] The cloud receives all uploaded shards and performs integrity verification. It then reassembles them into image layers according to the shard index and replaces the corresponding original image layers in the cloud image library.

[0096] This application also provides a mirror upload device, such as... Figure 6 As shown, the device includes: The acquisition module 601 is used to acquire the container image to be uploaded and split the container image into multiple image layers according to the function of the internal data. The matching module 602 is used to extract the image features of each image layer and match the compression algorithm for the corresponding image layer based on each image feature; The compression and upload module 603 is used to perform compression processing on the matching image layers in parallel using various compression algorithms, and upload the compressed image layers to the cloud image library in parallel.

[0097] Optionally, the matching module 602 is used for: Extract the mirror features of the mirror layer; Multiple triggering conditions are sorted in descending order of priority, with each triggering condition pre-bound to a compression algorithm; The mirror feature is verified using each trigger condition according to the order until the target trigger condition that passes the verification of the mirror feature is determined. The target trigger condition for matching the image layer is a compression algorithm that has been bound.

[0098] Optionally, the matching module 602 is specifically used for: The triggering condition with the highest priority is selected as the triggering condition to be verified, in descending order of priority. If the mirrored feature passes the verification of the trigger condition to be verified, then the trigger condition to be verified is determined as the target trigger condition. If the mirrored feature fails the verification of the trigger condition to be verified, the next priority trigger condition to be verified is selected as the new trigger condition to be verified, and the mirrored feature is verified using the new trigger condition until the target trigger condition that the mirrored feature passes the verification is verified.

[0099] Optionally, the compression and upload module 603 is used for: Based on the compression algorithm matched by each image layer, an independent thread group is allocated to each image layer. Each thread group performs compression and upload operations in parallel, and each thread group includes at least one compression thread and at least one upload thread. The data of the corresponding mirror layer is divided into fixed-size fragments; At least one compression thread sequentially performs pipelined compression on each fragment, and the compressed fragments are stored in a buffer queue in real time. At least one upload thread extracts compressed fragments from the buffer queue in real time, and uploads them concurrently to the cloud image library using multiplexing with fragments extracted by other upload threads.

[0100] Optionally, the compression and upload module 603 is used for: Based on the compression algorithm matched by each image layer, an independent thread group is allocated to each image layer. Each thread group performs compression and upload operations in parallel, and each thread group includes at least one compression thread and at least one upload thread. The data of the corresponding mirror layer is divided into fixed-size fragments; At least one compression thread performs pipelining compression on a portion of the data segment of the fragment, outputting continuous compressed data segments in real time and storing them in a buffer queue. The amount of data in the compressed data segment is less than the amount of data in the fragment. The upload thread extracts compressed data segments from the buffer queue in real time and uploads them concurrently to the cloud image library using multiplexing and other compressed data segments extracted by the upload thread.

[0101] Optionally, the device is also used for: Real-time monitoring of network transmission performance during the upload process; If the network transmission performance is detected to be lower than the preset threshold, the current initial compression algorithm is switched to the set compression algorithm, and the set compression algorithm is used to compress the uncompressed fragments or data segments. If the network transmission performance is detected to have recovered to or above the preset threshold, the system will switch to the initial compression algorithm to complete the compression of the remaining data.

[0102] Optionally, the device is also used for: If this is not the first time the container image has been uploaded, then determine the hash value of each image layer; The hash values ​​of each image layer are compared with the set of hash values ​​in the cloud image library to filter out image layers whose hash values ​​do not exist in the hash value set, which contains the hash values ​​of each image layer already stored in the cloud.

[0103] like Figure 7 As shown, this application provides an electronic device including a processor 701, a communication interface 702, a memory 703, and a communication bus 704, wherein the processor 701, the communication interface 702, and the memory 703 communicate with each other through the communication bus 704.

[0104] Memory 703 is used to store computer programs.

[0105] In one embodiment of this application, the processor 701, when executing a program stored in the memory 703, implements the image upload method provided in any of the foregoing method embodiments.

[0106] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the image upload method provided in any of the foregoing method embodiments.

[0107] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0108] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0109] It should be understood that the terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "a," "an," and "described" as used herein may also mean including the plural forms. The terms "comprising," "including," "containing," and "having" are inclusive and therefore indicate the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not construed as requiring them to be performed in a specific order described or illustrated unless the order of performance is explicitly indicated. It should also be understood that additional or alternative steps may be used.

[0110] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.< / digest> < / name>

Claims

1. A mirror upload method, characterized in that, The method includes: Obtain the container image to be uploaded, and split the container image into multiple image layers according to the function of the internal data; Extract the image features of each image layer, and match the compression algorithm to the corresponding image layer based on each image feature; Various compression algorithms are used to perform compression processing on the matched image layers in parallel, and the compressed image layers are then uploaded to the cloud image library in parallel.

2. The method according to claim 1, characterized in that, Extracting the mirror features of each mirror layer and matching compression algorithms to the corresponding mirror layer based on these features includes: Extract the mirror features of the mirror layer; Multiple triggering conditions are sorted in descending order of priority, with each triggering condition pre-bound to a compression algorithm; The mirror feature is verified using each trigger condition according to the sorting until a target trigger condition that passes the verification of the mirror feature is determined. The compression algorithm that is already bound to the target trigger condition is matched to the image layer.

3. The method according to claim 2, characterized in that, The mirror feature is verified using each trigger condition according to the aforementioned sorting until the target trigger condition that verifies the mirror feature is determined to be valid includes: The triggering condition with the highest priority is selected as the triggering condition to be verified, in descending order of priority. If the mirror feature passes the verification of the trigger condition to be verified, then the trigger condition to be verified is determined as the target trigger condition; If the mirror feature fails the verification of the trigger condition to be verified, the next priority trigger condition to be verified is selected as the new trigger condition to be verified, and the mirror feature is verified using the new trigger condition to be verified until the target trigger condition that verifies the mirror feature passes the verification is obtained.

4. The method according to claim 1, characterized in that, Various compression algorithms are used to perform parallel compression processing on the matched image layers, and the compressed image layers are then uploaded to the cloud image library in parallel, including: Based on the compression algorithm matching each image layer, an independent thread group is allocated to each image layer. Each thread group performs compression and upload operations in parallel, and each thread group includes at least one compression thread and at least one upload thread. The data of the corresponding mirror layer is divided into fixed-size fragments; The at least one compression thread sequentially performs pipelined compression processing on each fragment, and the compressed fragments are stored in the buffer queue in real time. The at least one upload thread extracts compressed fragments from the buffer queue in real time and uploads them concurrently to the cloud image library using multiplexing with fragments extracted by other upload threads.

5. The method according to claim 1, characterized in that, Various compression algorithms are used to perform parallel compression processing on the matched image layers, and the compressed image layers are then uploaded to the cloud image library in parallel, including: Based on the compression algorithm matching each image layer, an independent thread group is allocated to each image layer. Each thread group performs compression and upload operations in parallel, and each thread group includes at least one compression thread and at least one upload thread. The data of the corresponding mirror layer is divided into fixed-size fragments; The at least one compression thread performs pipelining compression on a portion of the data segment of the fragment, outputting continuous compressed data segments in real time and storing them in a buffer queue, wherein the data size of the compressed data segment is smaller than the data size of the fragment; The upload thread extracts the compressed data segment from the buffer queue in real time and uploads it concurrently to the cloud image library with the compressed data segment extracted by other upload threads through multiplexing.

6. The method according to claim 4 or 5, characterized in that, During the concurrent upload to the cloud image library, the method also includes: Real-time monitoring of network transmission performance during the upload process; If the network transmission performance is detected to be lower than a preset threshold, the current initial compression algorithm is switched to a set compression algorithm, and the set compression algorithm is used to compress the uncompressed fragments or data segments. If the network transmission performance is detected to recover to or above the preset threshold, then switch to the initial compression algorithm to complete the compression of the remaining data.

7. The method according to claim 1, characterized in that, Before extracting the mirror features of each mirror layer, the method further includes: If this is not the first time the container image has been uploaded, then determine the hash value of each image layer; The hash values ​​of each image layer are compared with the set of hash values ​​in the cloud image library to filter out image layers whose hash values ​​do not exist in the set of hash values. The set of hash values ​​contains the hash values ​​of each image layer that have been stored in the cloud.

8. A mirror upload device, characterized in that, The device includes: The acquisition module is used to acquire the container image to be uploaded and split the container image into multiple image layers according to the function of the internal data. The matching module is used to extract the image features of each image layer and match the compression algorithm for the corresponding image layer based on the image features. The compression and upload module is used to perform parallel compression processing on the matching image layers using various compression algorithms, and then upload the compressed image layers to the cloud image library in parallel.

9. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-7.

Images