Large-size container mirror image direct pushing method

Abstract

The embodiment of the invention provides a large-size container mirror image direct pushing method, which belongs to the technical field of data processing, and specifically comprises the following steps: receiving a container mirror image file uploaded by a user through a browser interface; writing the container mirror image file into a sandbox file system of a browser; reading the container mirror image file from the sandbox file system, decompressing the container mirror image file, and storing the decompressed file back to the sandbox file system; segmenting a mirror image layer file and a metadata file in the sandbox file system into fragmented data blocks; transmitting the fragmented data blocks in the plurality of memory buffer areas in parallel to the rear end of a mirror image warehouse for storage; until all file fragments are transmitted; synchronizing the mirroring metadata information to a mirroring warehouse; if file transmission is interrupted, uploading is restarted from the breakpoint fragment based on the recorded transmission position information; and when a preset condition is triggered, automatically cleaning the container mirror image file, the decompression file and the intermediate file in the sandbox file system. Through the scheme disclosed by the invention, the stability and reliability of the system are improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the technical field of data processing, and particularly relate to a large-size container image direct pushing method. BACKGROUND

[0002] At present, in the field of cloud mobile phones, various versions of system images based on docker (a computer operating system container runtime) need to be provided for customers. In order to meet the diverse needs of customers, the ability of uploading and pushing images to an image repository through a browser operation interface is usually provided for users. Since the image file is not a normal file, the image file needs to be pushed to a temporary file system as a normal file first, then downloaded from the temporary file system by a special server, then imported through a docker client command, and finally pushed to the image repository through a command. The whole process is relatively cumbersome, and additional storage, bandwidth costs and additional computing resources are required, and the processing time is also relatively long.

[0003] Therefore, there is an urgent need for a large-size container image direct pushing method that can improve system stability, reliability and adaptability. SUMMARY

[0004] Therefore, embodiments of the present disclosure provide a large-size container image direct pushing method to at least partially solve some problems in the prior art.

[0005] Embodiments of the present disclosure provide a large-size container image direct pushing method, comprising: Step 1: receiving a container image file uploaded by a user through a browser interface; Step 2: writing the container image file into a sandbox file system of the browser for temporary storage, the sandbox file system being based on a local disk space and not being limited by a memory of the browser; Step 3: reading the container image file from the sandbox file system and decompressing the container image file to extract image layer files and metadata files, and storing the decompressed files back to the sandbox file system; Step 4: dividing the image layer files and the metadata files in the sandbox file system into chunk data blocks, loading the chunk data blocks into a plurality of memory buffers in batches, and recording transmission position information of the current file chunk; Step 5: transmitting the chunk data blocks in the plurality of memory buffers in parallel to a backend storage of an image repository; Step 6: cyclically executing Step 4 to Step 5 until all file chunks are transmitted; Step 7: synchronizing image metadata information to the image repository to complete image uploading; Step 8: if the file transmission is interrupted, restarting the uploading based on the recorded transmission position information from the breakpoint chunk; Step 9. Automatically clean up the container image file, decompression file and intermediate file in the sandbox file system when the preset condition is triggered.

[0006] According to a specific implementation manner of the embodiment of the present disclosure, the sandbox file system is a persistent storage interface implementation provided by a browser, including File System API, IndexedDB or Origin Private File System.

[0007] According to a specific implementation manner of the embodiment of the present disclosure, the step 4 specifically includes: According to the current network bandwidth, memory buffer available space and file size, the size of the shard is dynamically adjusted.

[0008] According to a specific implementation manner of the embodiment of the present disclosure, the step 5 specifically includes: The shard data block is uploaded to the image warehouse by concurrently calling the HTTP PUT request or the shard upload protocol through a plurality of Web Worker threads.

[0009] According to a specific implementation manner of the embodiment of the present disclosure, the step 8 specifically includes: Step 8.1. Local persistent storage of shard location information and file verification information; Step 8.2. Verification of the integrity of the remaining data in the sandbox file according to the verification information after restart; Step 8.3. Resuming the upload process from the breakpoint shard location.

[0010] According to a specific implementation manner of the embodiment of the present disclosure, the preset condition includes successful synchronization of image metadata, user-initiated cancellation of uploading, network exception or failure of decompression.

[0011] The large-size container image direct pushing scheme in the embodiments of the present disclosure includes: step 1, receiving a container image file uploaded by a user through a browser interface; step 2, writing the container image file into a sandbox file system of the browser for temporary storage, the sandbox file system is based on local disk space and is not limited by browser memory; step 3, reading the container image file from the sandbox file system and decompressing, extracting image layer files and metadata files, and storing the decompressed files back to the sandbox file system; step 4, splitting the image layer files and metadata files in the sandbox file system into fragmented data blocks, loading the fragmented data blocks in multiple memory buffers in batches, and recording the transmission position information of the current file fragments; step 5, transmitting the fragmented data blocks in the multiple memory buffers to the image repository backend storage in parallel; step 6, cyclically executing steps 4 to 5 until all file fragments are transmitted; step 7, synchronizing image metadata information to the image repository to complete image uploading; step 8, if the file transmission is interrupted, restarting the uploading based on the recorded transmission position information from the breakpoint fragment; and step 9, automatically cleaning the container image file, decompressed file and intermediate file in the sandbox file system when a preset condition is triggered.

[0012] The beneficial effects of the embodiments of the present disclosure are: through the scheme of the present disclosure, by introducing a sandbox file system, the scheme is specially designed for ultra-large image files and more stringent memory limit scenarios, there is no intermediate link, the uploading time of the image is reduced by several times, and the number of user failure retransmissions is reduced through the resume mechanism; no temporary storage and processing link is needed, the hardware resource investment is reduced; no storage service and image processing service needs to be deployed, the maintenance cost of software and hardware is reduced; removing the intermediate service reduces the failure point, and improves the stability and reliability of the system. BRIEF DESCRIPTION OF DRAWINGS

[0013] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and other drawings can be obtained by those skilled in the art without creative labor.

[0014] Figure 1 A flowchart of a large-size container image direct pushing method provided by the embodiments of the present disclosure. DETAILED DESCRIPTION

[0015] The embodiments of the present disclosure will be described in detail below with reference to the drawings.

[0016] Following, the embodiments of the present disclosure are described through specific examples, and other advantages and effects of the present disclosure can be easily understood by those skilled in the art from the disclosure. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments. The present disclosure can also be implemented or applied by other different specific embodiments, and various modifications or changes can be made to the details in the specification without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present disclosure.

[0017] It should be noted that various aspects of the embodiments described below are within the scope of the appended claims. It should be apparent that the aspects described herein can be embodied in a wide variety of forms and that any specific structure and / or function described herein is merely illustrative. Based on the teachings provided herein one skilled in the art should appreciate that an aspect described herein can be implemented independently of any other aspects and that an aspect can be implemented both as any number of software, firmware, and / or hardware structures.

[0018] It should also be noted that the figures provided in the following embodiments are only to illustrate the basic concepts of the present disclosure in a schematic manner, and only show the components related to the present disclosure in the figures, not drawn according to the number, shape and size of the components in actual implementation, the actual implementation of each component can be a random change in shape, number and proportion, and the layout of the components can be more complex.

[0019] In addition, in the following description, specific details are provided in order to facilitate a thorough understanding of examples. However, one skilled in the art will understand that the aspects described can be practiced without these specific details.

[0020] The embodiments of the present disclosure provide a large-size container image direct pushing method, which can be applied to the container image direct pushing process in the cloud mobile phone scenario.

[0021] Referring to Figure 1 , a flowchart of a large-size container image direct pushing method provided by the embodiments of the present disclosure is provided. As shown in Figure 1 , the method mainly includes the following steps: Step 1: receiving a container image file uploaded by a user through a browser interface; For example, a user needs to upload a 5GB Ubuntu container image (ubuntu.tar) to a private image repository in a web browser. The browser environment is Chrome 110+, which supports the File System API.

[0022] In the file upload and sandbox storage phase, the user can log in to the image management platform and select the ubuntu.tar file through the <input type="file"> in the webpage.

[0023] The browser calls window.showOpenFilePicker() to obtain the file handle and reads the file metadata (without loading the complete content) through FileSystemFileHandle.getFile().

[0024] Step 2: Write the container image file to the browser's sandbox file system for temporary storage. The sandbox file system is based on local disk space and is not limited by browser memory. In implementation, the browser private file system interface can be called to create a temporary directory (e.g., / container / upload_task_12345 / ) on the disk, and the image file can be saved to this directory through streaming writing technology, directly occupying disk space.

[0025] Step 3: Read the container image file from the sandbox file system and decompress it, extract the image layer file and metadata file, and store the decompressed files back to the sandbox file system. In implementation, the web-based decompression library can be used to read the image file, The decompression results are: Image layer files: layer_file1.tar.gz, layer_file2.tar.gz,... Metadata files: manifest_file.json, config_file.json.

[0026] Then all the decompressed files are stored in the / decompressed_files / subdirectory of the private file system.

[0027] Step 4: Split the image layer file and metadata file in the sandbox file system into chunk data blocks, load them into multiple memory buffers in batches, and record the transmission position information of the current file chunk. In implementation, the chunking strategy can be set as: According to the current memory remaining (500MB) and network speed (100 megabit bandwidth), dynamically adjust the chunk size (initially set to 20MB).

[0028] Split the 1.2GB layer_file1.tar.gz into 60 chunks.

[0029] Then during loading and recording, 4 pieces of shards can be loaded into the memory buffer (80MB in total) each time, and the shard information is recorded through the browser database.

[0030] Step 5: After transmitting the shard data blocks in the plurality of memory buffers to the mirror warehouse backend storage in parallel, In specific implementation, 4 background work threads can be started, and each thread performs: Obtaining shard data from the memory buffer; Calling the shard upload interface of the mirror warehouse.

[0031] Step 6: Steps 4 to 5 are executed in a loop until all file shard transmissions are completed. In specific implementation, after the transmission of each batch of 4 shards is completed, the memory is immediately released and the next batch of shards is loaded, and the front end displays the progress in real time: “First layer mirror: 45 / 60 shards have been uploaded”.

[0032] Step 7: Synchronizing mirror metadata information to the mirror warehouse to complete mirror uploading; In specific implementation, after all shard uploading is completed, a shard merging request can be sent to the warehouse, and then the warehouse mirror metadata interface is called to submit the manifest file.

[0033] Step 8: If the file transmission is interrupted, the uploading is restarted from the breakpoint shard based on the recorded transmission position information. For example, the user closes the browser when uploading the 32nd shard, and then when the user accesses again, the system detects the unfinished uploading task, reads the record from the database, verifies the file integrity (compares the shard check code), and continues the transmission from the 32nd shard, skipping the successfully uploaded part.

[0034] Step 9: When a preset condition is triggered, the container mirror file, decompression file and intermediate file in the sandbox file system are automatically cleaned up.

[0035] In specific implementation, since the sandbox file system temporarily stores large-sized intermediate processing files, after any step fails or completely succeeds, the temporary mirror and decompressed files in the browser sandbox file system are cleaned up. The preset conditions can include: Successful scenario: Automatically delete temporary directory after metadata synchronization.

[0036] User cancellation: Immediately clean up generated files when manually stopped.

[0037] Decompression failure: Delete damaged files and prompt “mirror decompression error” When a preset condition is triggered, the container mirror file, decompression file and intermediate file in the sandbox file system are automatically cleaned up.

[0038] The large-size container image mirror pushing method provided by the embodiment breaks through the memory wall through the sandbox file system, optimizes the efficiency through dynamic fragmentation and parallel transmission, guarantees the breakpoint resume accuracy through the verification mechanism, and realizes zero residue through automatic cleaning, thereby solving three core problems of browser processing large-size container images at a technical level: ① memory overflow risk (from memory to disk storage); ② low transmission efficiency; and ③ unreliable abnormal recovery (fragment-level verification resume), thereby providing an enterprise-level solution for web container operation and maintenance, and having significant industrial application value.

[0039] It should be understood that parts of the present disclosure can be implemented in hardware, software, firmware, or a combination thereof.

[0040] The above is only a specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any changes or replacements that can be easily thought of by those skilled in the art within the technical scope disclosed by the present disclosure should be covered within the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims

1. A large size container mirror image push method, characterized by, The method comprises the following steps: Step 1: receiving a container image file uploaded by a user through a browser interface; Step 2: writing the container image file into a sandbox file system of the browser for temporary storage, the sandbox file system being based on local disk space and not subject to the memory limit of the browser; Step 3: reading the container image file from the sandbox file system and decompressing the same to extract image layer files and metadata files, and storing the decompressed files back into the sandbox file system; Step 4: dividing the image layer files and metadata files in the sandbox file system into chunk data blocks, loading the chunk data blocks into a plurality of memory buffers in batches, and recording the transmission position information of the current file chunk; Step 5: transmitting the chunk data blocks in the plurality of memory buffers to the image repository backend storage in parallel; Step 6: cyclically executing steps 4 to 5 until all file chunks are transmitted; Step 7: synchronizing the image metadata information to the image repository to complete the image uploading; Step 8: if the file transmission is interrupted, restarting the uploading from the breakpoint chunk based on the recorded transmission position information; Step 9: automatically cleaning up the container image file, decompressed file and intermediate file in the sandbox file system when a preset condition is triggered.

2. The method of claim 1, wherein, The sandbox file system is a persistent storage interface implementation provided by the browser, including File System API, IndexedDB or Origin Private File System.

3. The method of claim 1, wherein, Step 4 specifically comprises: Adjusting the chunk size dynamically according to the current network bandwidth, available memory buffer space and file size.

4. The method of claim 1, wherein, Step 5 specifically comprises: Concurrently invoking HTTP PUT requests or chunk upload protocols through a plurality of Web Worker threads to upload the chunk data blocks to the image repository.

5. The method of claim 1, wherein, Step 8 specifically comprises: Step 8.1: persistently storing the chunk position information and file verification information locally; Step 8.2: verifying the integrity of the remaining data in the sandbox file according to the verification information after restarting; Step 8.3: resuming the uploading process from the breakpoint chunk position.

6. The method of claim 1, wherein, The preset conditions include successful synchronization of image metadata, user-initiated cancellation of uploading, network exception or decompression failure resulting in interruption of the process.

Images