Data processing method and computer device

Abstract

The present application provides a data processing method and a computer device, which can be applied to the field of data processing. The method comprises: accurately tracking I / O requests in a container task starting process, constructing a mapping relationship between each I / O request and data (i.e., target data) necessary for container starting, and sending valid I / O requests (i.e., m I / O requests) generated in the starting process and the mapping relationship to an image repository. Each time a same container task is started, valid I / O requests generated are the same. Therefore, when the container task is started again, the number of I / O requests sent to the image repository in the container task starting process can be reduced (because m I / O requests are known, the m I / O requests can be sent to the image repository at a time after being aggregated). In addition, when the container task is started, corresponding m pieces of target data can be pulled from the image repository at a time on the basis of a pre-constructed mapping relationship, without the need for multiple pulls, thereby improving the efficiency of container task starting and reducing the data pull volume.

Description

A data processing method and computer device

[0001] This application claims priority to Chinese Patent Application No. CN202411221676.3, filed with the State Intellectual Property Office of China on August 30, 2024, entitled “A Data Processing Method and Computer Equipment”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of data processing, and more particularly to a data processing method and a computer device. Background Technology

[0003] With the development of cloud computing, the use of containers is becoming increasingly widespread. Due to their lightweight, isolation, and portability, containers have undoubtedly become an indispensable tool in cloud-native scenarios. Today, containerization has become the cornerstone of modern software infrastructure. As various applications demand higher quality of service, the rapid deployment of containerized applications has become paramount. Container startup relies on container image data, which is typically stored in an image repository and managed by a single node or cluster. An image service process provides various operations on the container image data. The node that starts the container (or the node running the container, or simply the container node) is often a different physical machine or virtual machine than the image repository. Container startup requires the use of container image data (or simply image data), thus often involving data interaction between the container node and the image repository. Container startup can generally be divided into two cases: cold start and warm start. A cold start refers to starting the container directly when the container image data is not on the container node; a warm start refers to starting the container after the container image data is already on the container node. The container startup process mainly consists of two parts: image preparation and container startup. Therefore, startup latency is primarily affected by these two parts. During image preparation, the container node checks if the container image data exists on its local machine. If not, it downloads the corresponding container image data from the image repository, decompresses it, and stores it locally. During container startup, the container node accesses this container image data to execute container tasks. Image preparation and container startup cannot run concurrently; the latter is highly dependent on the former.

[0004] Currently, there are generally two methods for starting container tasks during cold starts: 1) Full pull method: The container startup process first pulls the full data of the corresponding container image from the image repository, and then uses this data to start the container. 2) On-demand pull method: Instead of pulling the full container image data at startup, the container downloads the corresponding data from the image repository based on the data that the container depends on. Furthermore, traditional image formats (container images encapsulate the container application and its dependent file data, usually packaged according to a certain format, commonly known as the Open Container Initiative (OCI) image format) compress and package container image data, and do not support on-demand loading. Therefore, to support on-demand loading, the image format needs to be processed to support the indexability of on-demand files, but all of this must meet the OCI specification. Specifically, the on-demand retrieval method can be: Method 1, retrieve the image data of the target file on demand, because the full image data consists of multiple files, and the granularity of on-demand loading is the entire file; Method 2, retrieve large-granularity image data on demand, that is, cut the full container image data into chunks, and the granularity of on-demand loading is the chunked image data.

[0005] However, while the full-data retrieval method described above is very easy to implement and understand, it is also very inefficient. Retrieving container image data requires crossing the network, which is a very time-consuming operation. Furthermore, the larger the data volume of the container image, the longer this process takes. This cannot meet the demands of cloud-native scenarios where containers are commonly used. Additionally, this startup method relies on downloading the full container image data, but in reality, the container startup process only relies on a portion of the container image data. Research shows that during the startup process of a container task, the time spent retrieval of the full container image data accounts for 76% of the entire startup process, while the data actually used during startup only accounts for 6.4% of the full container image data. Similarly, in the on-demand retrieval method, Method 1 loads the entire file on demand, but the container task may not necessarily need the entire file's data when starting, so this method still retrieves redundant data. Likewise, in Method 2 of the on-demand retrieval method, due to technical limitations, the granularity of the image data chunks is very large, typically 1MB, which also leads to the problem of retrieving redundant data. Furthermore, the on-demand loading input / output (I / O) behavior is not user-friendly because it triggers the fetching of container image data multiple times. That is, every time the container image data to be read is not local, the on-demand loading process needs to be triggered. In other words, each I / O request triggers a loading process, which means that every subsequent I / O request for a file / data block will interact with the image repository, making the program run longer. Summary of the Invention

[0006] This application provides a data processing method and computer device for accurately tracking I / O requests during container task startup and constructing a mapping relationship between each I / O request and the data required for container startup (i.e., target data). All valid I / O requests (i.e., m I / O requests) generated during the container task startup process, along with their corresponding mapping relationships (i.e., second data), are sent to an image repository, allowing the image repository to package and store them together with the existing container image data. Since the valid I / O requests generated each time the same container task is started are the same, when other container nodes start the same container task, the number of I / O requests sent to the image repository during the container task startup process can be reduced (because the m I / O requests are known, they can be aggregated and sent to the image repository all at once). Simultaneously, during container task startup, the corresponding m target data can be pulled from the image repository at once based on the pre-constructed mapping relationship, eliminating the need for multiple pulls, thus improving the efficiency of container task startup and reducing the amount of data pulled.

[0007] Based on this, the embodiments of this application provide the following technical solutions:

[0008] Firstly, this application provides a data processing method applied to a first container node. This method specifically includes: First, the first container node determines m I / O requests generated during the startup process of the container task to be started (also called the target container task), where m ≥ 1. The set of these I / O requests is called the I / O trajectory. During the startup process of the container task, the first container node can obtain the files or data blocks (called first data) corresponding to each of the m I / O requests from the image repository. One I / O request corresponds to one piece of first data, thus a total of m pieces of first data can be obtained. These m pieces of first data belong to the container image data corresponding to the container task, and the container image data is stored in the image repository. Then, the first container node determines the target data corresponding to each I / O request from the obtained m pieces of first data, i.e., one I / O request corresponds to one piece of target data, thus obtaining a total of m pieces of target data. These m pieces of target data are the necessary data included for the successful startup of the container task, and these m pieces of target data do not contain any redundant data unrelated to startup. Therefore, the m pieces of target data are the minimum amount of data required for the successful startup of the container task. After receiving m target data items, the first container node further determines the mapping relationship between the m I / O requests and the m target data items; this mapping relationship can be called the first mapping relationship. Finally, the first container node sends second data to the image repository so that the image repository can save the second data, which includes the m I / O requests and the first mapping relationship. For example, after receiving the second data, the image repository can package and store the second data together with the original container image data.

[0009] In the above embodiments of this application, by accurately tracking I / O requests during the container task startup process and constructing a mapping relationship between each I / O request and the target data necessary for container startup, all valid m I / O requests generated during the container task startup process, along with their corresponding mapping relationships (i.e., the second data), are sent to the image repository for storage. Since the valid I / O requests generated each time the same container task is started are the same, when other container nodes start the same container task, the m I / O requests can be aggregated and sent to the image repository at once, reducing the amount of I / O requests sent to the image repository during the container task startup process. Simultaneously, when the container task starts, the corresponding m target data can be pulled from the image repository at once based on the pre-constructed mapping relationship, eliminating the need for multiple pulls and improving the efficiency of container task startup. Furthermore, since the m target data are necessary for container startup, no redundant data is pulled during the data retrieval process, thus achieving a minimum amount of data retrieval.

[0010] In one possible implementation of the first aspect, the second data may further include the m target data described above.

[0011] In the above embodiments of this application, when a container node needs to obtain m target data corresponding to the container task, the image repository can send them directly without the image repository needing to decompress the original container image data based on the m I / O requests and the first mapping relationship, then read the decompressed files or data blocks, and then extract the required m target data from the decompressed files or data blocks. Since the direct sending method eliminates the need for the image repository to parse this data from the original container image data, the container node can directly pull it, while the image repository's decompression → reading → extraction process is relatively more time-consuming.

[0012] In one possible implementation of the first aspect, before determining the m I / O requests generated by the container task during startup, the method may further determine the task identifier corresponding to the container task based on the task content during container runtime, and obtain the first manifest file from the container image data corresponding to the container task from the image repository. Before sending the second data to the image repository, the second manifest file can be obtained based on the task identifier, the m I / O requests, and the first manifest file, and included in the second data to be sent.

[0013] In the above embodiments of this application, the task identifier serves as a unique index in the I / O trajectory. When the container task is restarted (e.g., on a second container node), it is used to index the corresponding I / O trajectory and target data in the image, thereby improving the accuracy of data retrieval.

[0014] In one possible implementation of the first aspect, one way to obtain the second digest file based on the task identifier, m I / O requests, and the first digest file is as follows: when the second data does not include m target data, the first hash value corresponding to the task identifier and the second hash value corresponding to the m I / O requests are recorded in the first digest file to obtain the second digest file.

[0015] In the above embodiments of this application, a specific implementation method for obtaining a second digest file by modifying the first digest file is described. This method is obtained by additionally recording two hash values ​​and is feasible.

[0016] In one possible implementation of the first aspect, another way to obtain the second digest file based on the task identifier, m I / O requests, and the first digest file is: when the second data includes m target data, record the first hash value corresponding to the task identifier, the second hash value corresponding to the m I / O requests, and the third hash value corresponding to the m target data in the first digest file to obtain the second digest file.

[0017] In the above embodiments of this application, another implementation method for obtaining a second digest file by modifying the first digest file is specifically described. This method is obtained by additionally recording 3 hash values ​​and is feasible.

[0018] In one possible implementation of the first aspect, determining the task identifier corresponding to the container task based on the container task could be: 1) Directly generating the corresponding task identifier based on the container task, for example, the task identifier could be a generated hash value. 2) A mapping relationship between each container task and each task identifier can be pre-built, with each container task corresponding to a unique task identifier. This mapping relationship can be placed on the first container node. When the first container node wants to start the aforementioned container task (i.e., the target container task), it can directly assign the corresponding task identifier to the target container task to be started based on this mapping relationship.

[0019] In the above embodiments of this application, several implementation methods for obtaining task identifiers are specifically described, which are flexible.

[0020] In one possible implementation of the first aspect, one way to determine the m I / O requests generated by the container task during startup is as follows: First, collect the I / O requests generated by the container task during startup in real time. Assume that a total of n I / O requests are collected during the entire startup process, where n ≥ m. Then, determine the m I / O requests related to the container task from these n collected I / O requests. This way, the I / O trajectory with the minimum amount of metadata can be obtained.

[0021] In the above embodiments of this application, it is specifically described that m valid I / O requests are determined from n I / O requests collected in real time. Since these m I / O requests are valid I / O requests, the m target data corresponding to them are the necessary data required for the container task to start successfully, and do not contain any redundant data that is not needed for the container task to start, thereby minimizing the amount of data pulled by the container node.

[0022] In one possible implementation of the first aspect, determining m I / O requests from n I / O requests can be achieved by: removing the first I / O request from the n I / O requests to obtain m I / O requests, wherein the first target data corresponding to the first I / O request is included in the second target data corresponding to the second I / O request, the second I / O request is included in the n I / O requests, or the first I / O request does not belong to the I / O requests of the container task.

[0023] In the above embodiments of this application, a specific implementation method for determining m I / O requests is described, which can guarantee that all m I / O requests obtained are valid I / O requests.

[0024] In one possible implementation of the first aspect, one way to collect the n I / O requests generated during the startup process of the container task is as follows: First, the first container node pre-registers an I / O probe (also called an I / O probe point) to track the I / O requests generated during startup in real time. Specifically, the first container node can further analyze the file offset and the length of the corresponding target data of the I / O request being tracked based on the registered I / O probe, and perform page alignment processing (to improve reading efficiency). At the same time, it can find out the internal file path of the container corresponding to the I / O request and append this data as an I / O trace to the file. When the termination condition is met, the tracking stops, and the n I / O requests are the I / O requests collected during the tracking period.

[0025] In the above embodiments of this application, I / O probes are used to accurately track I / O requests during the container task startup process, thereby improving tracking accuracy and efficiency.

[0026] In one possible implementation of the first aspect, the termination conditions for the above collection include, but are not limited to: a) the container task has been started and executed; b) the execution time of the container task reaches a preset time; c) the number of I / O requests per unit time during the start-up process of the container task is lower than a preset threshold.

[0027] The above embodiments of this application specifically describe several scenarios for data collection termination conditions, which have broad applicability.

[0028] A second aspect of this application provides a method for starting a container task, applied to a second container node. The method specifically includes: First, the second container node determines a request to be sent to an image repository based on the currently scheduled container task to be started (i.e., the target container task). This request can be called a target request. After determining the target request, the target request is sent to the image repository. The target request instructs the image repository to send m I / O requests corresponding to the currently scheduled container task and m target data corresponding to these m I / O requests to the second container node. It should be noted that in this embodiment, these m I / O requests are generated by the first container node during the startup of the container task. These m target data are obtained based on a first mapping relationship, which is the mapping relationship between these m I / O requests and these m target data. Furthermore, these m target data are necessary data for the successful startup of the container task; one I / O request corresponds to one target data, and m ≥ 1. Afterward, the second container node stores the m I / O requests and m target data sent by the image repository in its kernel page cache. The second container node starts the container task and, during the startup process, collects the I / O requests generated in real time. These I / O requests are called target I / O requests. Finally, the second container node further determines whether the real-time collected target I / O request is included in the above m I / O requests. If so, it retrieves the target data corresponding to the target I / O request from its own kernel buffer.

[0029] In the above embodiments of this application, when the second container node re-executes the container task (which has previously run on the first container node and established a first mapping relationship between m I / O requests and m target data), since the effective I / O requests generated each time the same container task is started are the same, the corresponding m target data can be pulled from the image repository at once based on the pre-built first mapping relationship, eliminating the need for multiple pulls. Furthermore, the pulled data is directly imported into the kernel page cache, reducing memory copy overhead.

[0030] In one possible implementation of the second aspect, determining the target request based on the currently pending container task can be as follows: First, the second container node determines the task identifier corresponding to the currently pending container task. Then, based on the task identifier, it retrieves a summary file of the container image data corresponding to the container task from the image repository, and determines the target request based on this second summary file. The second summary file is obtained by the first container node based on the task identifier, m I / O requests, and the original first summary file. After modifying the summary file, the first container node sends it to the image repository for storage.

[0031] In the above embodiments of this application, the target request is specifically described as being determined based on the task identifier and the second digest file, which is feasible.

[0032] In one possible implementation of the second aspect, determining the target request based on the second digest file can be done by: determining the target request based on the first hash value and the second hash value in the second digest file; or, determining the target request based on the first hash value, the second hash value, and the third hash value in the second digest file; wherein the first hash value is the hash value corresponding to the task identifier, the second hash value is the hash value corresponding to m I / O requests, and the third hash value is the hash value corresponding to m target data.

[0033] In the above embodiments of this application, one specific implementation of determining the target request based on the second digest file is described, which is based on two or three hash values ​​in the digest file, and has flexibility.

[0034] In one possible implementation of the second aspect, if the second container node determines that the target I / O request for real-time collection is not included in the aforementioned m I / O requests, it needs to pull the file or data block corresponding to the target I / O request from the image repository according to the original pull process. This corresponding file or data block can be referred to as the target first data, which is included in the container image data, and the container image data is stored in the image repository. Finally, the target data corresponding to the target I / O request is determined from the target first data in a manner similar to step 303 above.

[0035] In the above embodiments of this application, if the target I / O request generated in real time does not belong to m I / O requests, the original data retrieval method is followed. This ensures that the container node can still successfully retrieve the corresponding data, improving the flexibility of execution.

[0036] A third aspect of this application provides a computer device that serves as a first container node and has the functionality to implement the method described in the first aspect or any possible implementation of the first aspect. This functionality can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the described functionality.

[0037] A fourth aspect of this application provides a computer device that serves as a second container node and has the functionality to implement the method described in the second aspect or any possible implementation of the second aspect. This functionality can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the described functionality.

[0038] The fifth aspect of this application provides a computer device, which may include a memory, a processor, and a bus system, wherein the memory is used to store a computer program (also referred to as a program or computer-readable instructions), and the processor is used to call the program stored in the memory to execute the method of the first aspect of the embodiments of this application or any possible implementation of the first aspect, or to execute the method of the second aspect of the embodiments of this application or any possible implementation of the second aspect.

[0039] The sixth aspect of this application provides a computer-readable storage medium storing instructions that, when executed on a computer, enable the computer to perform the method of the first aspect or any possible implementation thereof, or enable the computer to perform the method of the second aspect or any possible implementation thereof.

[0040] The seventh aspect of this application provides a computer program or a computer program product containing instructions that, when the computer program or computer program product is run on a computer, causes the computer to perform the method of the first aspect or any possible implementation thereof, or causes the computer to perform the method of the second aspect or any possible implementation thereof.

[0041] An eighth aspect of this application provides a chip including at least one processor and at least one interface circuit coupled to the processor. The at least one interface circuit is used to perform transceiver functions and send instructions to the at least one processor. The at least one processor is used to run computer programs or instructions, having the functionality to implement the methods described in the first aspect or any possible implementation of the first aspect, or having the functionality to implement the methods described in the second aspect or any possible implementation of the second aspect. This functionality can be implemented in hardware, software, or a combination of hardware and software, including one or more modules corresponding to the above-described functions. Furthermore, the interface circuit is used to communicate with other modules outside the chip.

[0042] In some implementations of this application, some of the one or more processors may implement some steps of the above method through dedicated hardware. For example, the processing involving neural network models may be implemented by a dedicated neural network processor or graphics processor.

[0043] The method provided in this application embodiment can be implemented by a single chip or by multiple chips working together. Attached Figure Description

[0044] Figure 1 is a schematic diagram of the system architecture provided in an embodiment of this application;

[0045] Figure 2 is another schematic diagram of the system architecture provided in the embodiment of this application;

[0046] Figure 3 is a flowchart illustrating a data processing method provided in an embodiment of this application.

[0047] Figure 4 is a flowchart illustrating a method for starting a container task according to an embodiment of this application.

[0048] Figure 5 is a schematic flowchart of the data retrieval process provided in an embodiment of this application;

[0049] Figure 6 is a component structure diagram provided in an embodiment of this application;

[0050] Figure 7 is another component structure diagram provided in an embodiment of this application;

[0051] Figure 8 is a schematic flowchart of a file system opening operation provided in an embodiment of this application;

[0052] Figure 9 is a flowchart illustrating a file reading / file mapping operation provided in an embodiment of this application.

[0053] Figure 10 is a schematic diagram of a computer device provided in an embodiment of this application;

[0054] Figure 11 is another structural schematic diagram of the computer device provided in an embodiment of this application;

[0055] Figure 12 is another structural schematic diagram of the computer device provided in an embodiment of this application. Detailed Implementation

[0056] This application provides a data processing method and computer device for accurately tracking I / O requests during container task startup and constructing a mapping relationship between each I / O request and the target data required for container startup. All m valid I / O requests generated during the container task startup process, along with their corresponding mapping relationships (i.e., second data), are sent to an image repository for storage. Since the valid I / O requests generated each time the same container task is started are the same, when other container nodes start the same container task, the m I / O requests can be aggregated and sent to the image repository all at once, reducing the amount of I / O requests sent to the image repository during container task startup. Simultaneously, during container task startup, the corresponding m target data can be pulled from the image repository at once based on the pre-constructed mapping relationship, eliminating the need for multiple pulls and improving the efficiency of container task startup. Furthermore, since the m target data are necessary for container startup, no redundant data is pulled during the data retrieval process, thus achieving a minimum amount of data pulled.

[0057] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0058] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0059] To better understand the solutions of the embodiments of this application, the relevant terms and concepts that may be involved in the embodiments of this application will be introduced below. It should be understood that the explanations of the relevant terms and concepts may be limited due to the specific circumstances of the embodiments of this application, but this does not mean that this application can only be limited to that specific circumstance. The specific circumstances of different embodiments may also differ, and no specific limitation is made here.

[0060] (1) Container image

[0061] A lightweight, portable software packaging method for packaging an application and all its runtime environment, system libraries, and dependencies into a single file. This file can be loaded and run by containerized platforms (such as Docker, Kubernetes, etc.), enabling rapid deployment and execution of the application.

[0062] (2) Container image data

[0063] Container image data, also known as image data, refers to a series of data used in containerization technology to define the container's runtime environment and application. One container image can correspond to one or more container tasks, the difference being that each container task uses different amounts and types of data from that container image. When a container node starts a container task, a unique task identifier is generated or determined for that task. Based on this task identifier, it is possible to directly identify which container image corresponds to that container task.

[0064] (3) OCI image format

[0065] A standard container image format that defines an image as containing a manifest file, an image index (optional), a set of file system layers, and a configuration file.

[0066] (4) I / O trajectory

[0067] This refers to the set of all I / O requests triggered during program execution. I / O requests include reading / writing files and the corresponding file offsets and lengths.

[0068] (5) Page caching

[0069] This refers to a segment of memory in the kernel, measured in physical pages, used to accelerate file access.

[0070] (6) Missing page

[0071] When the target page table entry is empty or marked as read-only, the memory subsystem triggers a page fault operation to map a block of physical memory address to the corresponding page table entry.

[0072] First, the system architecture and overall process of the method used in the embodiments of this application will be described. Please refer to Figure 1 for details. Figure 1 is a schematic diagram of the system architecture provided in the embodiments of this application. The system architecture of this application has been modified accordingly on both the container node 100 and the image repository 200 side. Specifically, on the container node 100 side, an I / O trajectory module 110 is mainly introduced to track the I / O trajectory during the execution of container tasks and load the relevant data (i.e., the m target data) corresponding to the I / O trajectory into the page cache of the kernel file system of the target file. On the image repository 200 side, the container image data 210 with embedded I / O trajectories will embed the I / O trajectories of different container tasks, or embed the I / O trajectories of different container tasks and the corresponding m target data. Whether only I / O trajectories are embedded or I / O trajectories and the corresponding m target data are embedded depends on the specific application scenario, and this application does not limit this.

[0073] Based on this system architecture, the method of this application mainly includes two aspects, which are described below:

[0074] I. I / O Track Management

[0075] I / O trajectory management tracks I / O requests at the file system level. This approach shields the container's underlying storage driver, allowing for precise collection of I / O trajectories and corresponding target data for container tasks across any on-demand loading method. The I / O trajectory management described in this application operates at the container task level. Different container tasks using the same container image data will generate different I / O trajectories. These I / O trajectories are associated with the container tasks and embedded together into the container image, ensuring compatibility between the added I / O trajectories and the original container image. Similarly, in other application scenarios, these I / O trajectories can also be associated with container tasks and embedded along with m corresponding target data points into the container image, maintaining compatibility between the added I / O trajectories and the original container image.

[0076] II. Container Task Launch Based on I / O Trajectory

[0077] This application differs from existing on-demand loading of container image data in its container task startup process based on I / O trajectories. Traditional on-demand loading fetches files or data blocks as needed during startup, but its I / O behavior is inefficient because it triggers multiple fetches of container image data each time the data is not locally available. This means that every I / O request triggers a loading process, resulting in subsequent I / O requests for a single file / data block requiring interaction with the image repository, thus lengthening the program's execution time. In contrast, this application fetches the corresponding data (i.e., the aforementioned m target data) from the I / O trajectory all at once during startup. This is because the application orchestrates the I / O requests of the container task, achieving I / O aggregation. Therefore, during container task startup, the corresponding m target data can be fetched from the image repository at once through I / O aggregation, eliminating the need for multiple fetches by the container node and improving I / O efficiency during startup.

[0078] After the m target data corresponding to the I / O trajectory are pulled to the container node, since these target data also have page alignment characteristics and are the data that the container startup process actually needs to read, this invention loads the m target data corresponding to the I / O trajectory into the kernel page cache. I / O requests during the container task startup process can directly hit the kernel page cache, thereby realizing I / O pass-through between the page cache and the image repository and reducing the I / O access path (in the existing on-demand pull method, the container image data needs to go through multiple memory copies from the image repository to the local image cache and finally to the kernel page cache of the container node, which makes the read I / O stack during the container task startup process longer and brings certain data copy overhead).

[0079] It should be noted that, in this embodiment, the system architecture in Figure 1 is merely illustrative, and the deployment method of each unit / module is not limited. Furthermore, each device in Figure 1 may contain more modules / sub-modules to decouple functions. For example, the I / O trajectory module 101 in container node 100 can be decoupled into a tracking sub-module 1001 and a loading sub-module 1002. The tracking sub-module 1001 tracks the I / O trajectory during container task execution, and the loading sub-module 1002 loads the relevant data corresponding to the I / O trajectory (i.e., the m target data) into the page cache of the kernel file system of the target file. Similarly, each device in Figure 1 may contain fewer modules / sub-modules to achieve centralized data processing. This application does not limit the number and function of modules / sub-modules contained in each device; these can be determined by specific application requirements.

[0080] Based on the above system architecture, an embodiment of the method provided in this application will be described below. Please refer to Figure 3 for details. Figure 3 is a flowchart of a data processing method provided in this application. This data processing method corresponds to the above-mentioned I / O trajectory management and is applied to the first container node. Specifically, it may include the following steps:

[0081] 301. Determine the m I / O requests generated by the container task during startup, where m ≥ 1.

[0082] First, the first container node determines the m I / O requests generated during the startup process of the container task to be started (also known as the target container task), where m ≥ 1. The set of these I / O requests can be called the I / O trajectory.

[0083] It should be noted that, in the embodiments of this application, the m I / O requests refer to the m I / O requests related to the container task, that is, the m valid I / O requests.

[0084] As an example, the method to obtain the m I / O requests can be as follows: First, collect the I / O requests generated by the container task during startup in real time. Assume that a total of n I / O requests are collected during the entire startup process, where n ≥ m. Then, determine the m I / O requests related to the container task from these n collected I / O requests. This yields the I / O trajectory with the minimum amount of metadata. Next, the corresponding m target data can be aggregated using this I / O trajectory. These aggregated m target data are the necessary data for the successful startup of the container task, and do not contain any redundant data not required for the container task's startup.

[0085] It should also be noted that, in some embodiments of this application, the methods for determining m I / O requests from n I / O requests include, but are not limited to:

[0086] a. Merge multiple I / O requests from n I / O requests.

[0087] If the first target data corresponding to one of the n I / O requests (referred to as the first I / O request) is contained in the second target data corresponding to another I / O request (referred to as the second I / O request), that is, the first target data is a subset of the second target data, then the two I / O requests can be merged. The merging method can be to keep only the second I / O request and filter out the first I / O request.

[0088] Similarly, if there are multiple I / O requests that need to be merged among the n I / O requests, they can be merged in the manner described above, which will not be elaborated here.

[0089] b. Eliminate one or more I / O requests from among the n I / O requests.

[0090] If one or more of the n I / O requests do not belong to the current container task (i.e. invalid I / O requests), these one or more I / O requests can be directly removed.

[0091] It should also be noted that, in some embodiments of this application, the first container node can achieve real-time collection of I / O requests in the following way: First, the first container node pre-registers an I / O probe (also called an I / O probe point) to track I / O requests generated during startup in real time. Specifically, the first container node can further analyze the file offset and the length of the corresponding target data of the I / O request being tracked based on the registered I / O probe, and perform page alignment processing (to improve reading efficiency). At the same time, it can find out the internal file path of the container corresponding to the I / O request and append this data as an I / O trace to the file.

[0092] For example, in the embodiment shown in Figure 1, the tracing submodule 1001 can be loaded into the kernel first. I / O probes can be registered within this tracing submodule 1001 for real-time collection of I / O requests during subsequent container task tracing. This process can be called the initialization process of the tracing submodule 1001. Then, during the collection process, it is determined whether the collection termination condition has been met. Once the collection termination condition is met, the tracing will stop (e.g., by removing the I / O probes). The n I / O requests are then the I / O requests collected during this entire tracing period.

[0093] Specifically, the first container node starts the container task based on the container startup parameters input by the user. These startup parameters reveal which container image data is being used and what type of container task is being run. Before the container task starts, real-time I / O request collection is enabled, so that all I / O requests during the container task's startup process are recorded until the collection termination condition is met.

[0094] It should be noted that, in some embodiments of this application, the termination conditions for the above-mentioned data collection include, but are not limited to:

[0095] a. The container task has been started and executed successfully;

[0096] b. The container task execution startup time reaches the preset time;

[0097] c. The number of I / O requests per unit time during the startup process of the container task is lower than the preset threshold.

[0098] It should be noted that in some embodiments of this application, after the above initialization, the first container node can further clean up the cached data of files in the current kernel page cache under a specified directory (e.g., which can be specified by the user). This is to prevent inaccurate tracking and to narrow down the tracking scope. Furthermore, to accurately track kernel page fault behavior, the first container node can also disable the kernel fault around feature.

[0099] It should also be noted that, in this embodiment, when the first container node starts the container task according to the container startup parameters input by the user, it can further determine the task identifier corresponding to the container task based on the task content during container runtime, and obtain the summary file (i.e., manifest file) from the container image data corresponding to the container task from the image repository. This summary file can be called the first summary file or the first manifest file. The task identifier serves as a unique index in the I / O trajectory and is used to index the corresponding I / O trajectory and target data in the image when the container task is subsequently started again (e.g., started on a second container node).

[0100] It should be noted that, in some embodiments of this application, the implementation methods for determining the task identifier corresponding to the container task may include, but are not limited to:

[0101] a. Generate a task identifier based on the container task.

[0102] In this case, a corresponding task identifier can be directly generated based on the container task. For example, the task identifier can be a generated hash value.

[0103] b. Assign corresponding task identifiers to container tasks based on the pre-built mapping relationship.

[0104] In this scenario, a mapping relationship between each container task and each task identifier can be pre-built, with each container task corresponding to a unique task identifier. This mapping relationship can be placed on the first container node. When the first container node wants to start the container task (i.e. the target container task) mentioned above, it can directly assign the corresponding task identifier to the target container task to be started based on this mapping relationship.

[0105] 302. During startup, the first data corresponding to each of the m I / O requests is obtained from the image repository, resulting in a total of m first data. These m first data belong to the container image data corresponding to the container task, and the container image data is stored in the image repository.

[0106] During the startup process of this container task, the first container node can obtain the files or data blocks (referred to as first data) corresponding to each of the m I / O requests from the image repository. One I / O request corresponds to one piece of first data, so a total of m pieces of first data can be obtained. These m pieces of first data belong to the container image data corresponding to this container task, and the container image data is stored in the image repository.

[0107] 303. Determine the target data corresponding to each of the m I / O requests from the m first data, and obtain a total of m target data. These m target data are the necessary data included for the successful startup of the container task.

[0108] Next, the first container node determines the target data corresponding to each I / O request from the m pieces of initial data obtained. That is, one I / O request corresponds to one piece of target data, resulting in a total of m pieces of target data. These m pieces of target data are the necessary data included for the successful startup of the container task, and they do not contain any redundant data unrelated to the startup. Therefore, the m pieces of target data represent the minimum amount of data required for the successful startup of the container task.

[0109] 304. Determine the first mapping relationship between m I / O requests and m target data.

[0110] After obtaining m target data, the first container node further determines the mapping relationship between the m I / O requests and the m target data. This mapping relationship can be called the first mapping relationship.

[0111] 305. Send the second data to the mirror repository so that the mirror repository saves the second data, which includes m I / O requests and the first mapping relationship.

[0112] Finally, the first container node sends second data to the image repository so that the image repository can save the second data. This second data includes m I / O requests and a first mapping relationship. For example, after receiving the second data, the image repository can package and store it together with the existing container image data.

[0113] It should be noted that in some embodiments of this application, the second data may further include the aforementioned m target data. Thus, when a container node needs to obtain the m target data corresponding to its container task, the image repository can send it directly without requiring the image repository to decompress the original container image data based on the m I / O requests and the first mapping relationship, then read the decompressed files or data blocks, and finally extract the required m target data from the decompressed files or data blocks. Since the direct sending method eliminates the need for the image repository to parse this data from the original container image data, the container node can directly pull it, whereas the image repository's decompression → reading → extraction process is relatively more time-consuming.

[0114] It should also be noted that in some embodiments of this application, since the first container node can further determine the task identifier corresponding to the container task based on the task content of the container runtime, and obtain the first digest file (i.e., the first manifest file) in the container image data corresponding to the container task from the image repository, in this case, the first container node also needs to further modify the original first manifest file according to the task identifier and the m I / O requests to obtain the second digest file, i.e., the second manifest file, and include the second manifest file in the above-mentioned second data and send it to the image repository.

[0115] It should be noted that, in this embodiment of the application, how to modify the first manifest file to obtain the second manifest file is determined based on the specific content included in the second data, which will be explained below:

[0116] a) The second set of data does not include the m target data.

[0117] In this case, it is only necessary to record the hash value corresponding to the task identifier (which can be called the first hash value; when the task identifier is directly generated, the task identifier is this hash value) and the hash value corresponding to the m I / O requests (which can be called the second hash value) in the first manifest file. The first manifest file that additionally records these two hash values ​​is the second manifest file.

[0118] b. The second set of data includes m target data points.

[0119] In this case, in addition to recording the first hash value corresponding to the task identifier and the second hash value corresponding to the m I / O requests in the first manifest file, it is also necessary to record the hash value (which can be called the third hash value) corresponding to the m target data. The first manifest file that additionally records these three hash values ​​is the second manifest file mentioned above.

[0120] To better understand the difference between the first and second manifest files, the following example illustrates the distinction:

[0121] In the original container image data, the manifest file (i.e., the first manifest file) is assumed to have the following format:

[0122] {

[0123] Xxx

[0124] }

[0125] Therefore, in the method of this application embodiment, the format of the manifest file (i.e., the second manifest file) in the container image data is as follows:

[0126] As mentioned above, an I / O trace record was added to the second manifest file. Here, `${task_id}` is the task identifier of the container task, and `${sha256}` is the hash value calculated from `m` I / O requests, used as the filename for storing the `m` I / O request files. This additional file, `${sha256}`, records the information for the `m` I / O requests. Since the image repository can store a file according to certain rules, such as naming it `${sha256}`, it can automatically determine the location of this file on the server host.

[0127] If the second data also includes m target data, then the format of the second manifest file can be:

[0128] In this case, two additional files are added, each with its own content hash value as its filename: one to record I / O requests and the other to record the target data.

[0129] It should be noted that the formats of the first and second manifest files mentioned above are for illustrative purposes only, and this application does not impose any restrictions on the format of the manifest files.

[0130] Based on the above system architecture, another embodiment of the method provided in this application will be described below. Please refer to Figure 4 for details. Figure 4 is a flowchart of a container task startup method provided in this application. This container task startup method corresponds to the above-mentioned container task startup based on I / O trajectory. This method is applied to the second container node and may specifically include the following steps:

[0131] 401. Determine the target request based on the container task to be started, and send the target request to the image repository. The target request is used to instruct the image repository to send m I / O requests corresponding to the container task and m target data corresponding to the m I / O requests to the second container node. The m I / O requests are generated by the first container node during the start-up of the container task. The m target data are obtained based on the first mapping relationship. The first mapping relationship is the mapping relationship between the m I / O requests and the m target data. The m target data are the necessary data included for the successful start of the container task. One I / O request corresponds to one target data, and m≥1.

[0132] First, the second container node determines a request to be sent to the image repository based on the currently pending container task (i.e., the target container task). This request can be called a target request. After determining the target request, it sends it to the image repository. The target request instructs the image repository to send m I / O requests corresponding to the currently pending container task and m target data corresponding to these m I / O requests to the second container node. It is important to note that in this embodiment, these m I / O requests are generated by the first container node during the startup of the container task. These m target data are obtained based on a first mapping relationship, which is the mapping relationship between these m I / O requests and these m target data. Furthermore, these m target data are the necessary data included for the successful startup of the container task; one I / O request corresponds to one target data, and m ≥ 1.

[0133] It should be noted that, in some embodiments of this application, one way to determine the target request based on the currently pending container task is as follows: First, the second container node determines the task identifier corresponding to the currently pending container task. Then, based on the task identifier, it retrieves the manifest file of the container image data corresponding to the container task from the image repository, and determines the target request based on the second manifest file. The second manifest file is obtained by the first container node based on the task identifier, m I / O requests, and the original first manifest file (the specific method of obtaining this manifest file can be found in the above embodiments, and will not be repeated here). After modifying the manifest file, the first container node sends it to the image repository for storage.

[0134] It should be noted that in some embodiments of this application, when the second manifest file only includes the first hash value and the second hash value, the target request is determined based on the first hash value and the second hash value in the second manifest file; when the second manifest file includes a third hash value in addition to the first and second hash values, the target request is determined based on the first hash value, the second hash value, and the third hash value in the second manifest file. Here, the first hash value is the hash value corresponding to the task identifier (e.g., the task identifier itself can be a hash value), the second hash value is the hash value corresponding to m I / O requests, and the third hash value is the hash value corresponding to m target data.

[0135] Specifically, to facilitate understanding of the above process, the following example illustrates the process of starting the container task: The second container node first obtains the manifest file from the image repository based on the image specified in the container task to be started, which is the manifest file modified by the first container node in the above embodiment (i.e., the second manifest file). Then, it parses the second manifest file. Since the first container node previously wrote the hash value of the task identifier into the second manifest file, after parsing the second manifest file, the second container node can obtain the index value of the corresponding I / O request based on the container task identifier in it, and use the index value to construct a file pull request (i.e., the target request mentioned above). Then, it can pull m target data corresponding to the I / O request from the image repository based on the target request.

[0136] It's important to note that in this embodiment, the container task to be started on the second container node is the same as the target container task started on the first container node. However, in some practical applications, the container task to be started on the second container node can be a similar task to the target container task started on the first container node, since similar container tasks typically share most of their I / O requests. For the differing I / O requests, the image data can be pulled using the original pull method. Overall, this still improves pull efficiency and reduces the amount of data pulled.

[0137] 402. Store the m I / O requests and m target data sent by the image repository in the kernel page cache.

[0138] Then, the second container node stores the m I / O requests and m target data sent by the image repository in its kernel page cache.

[0139] 403. Start the container task and collect the target I / O requests generated during the startup process.

[0140] The second container node starts the container task and collects the I / O requests generated during the startup process in real time. These I / O requests are called target I / O requests.

[0141] 404. If the target I / O request is contained in m I / O requests, retrieve the target data corresponding to the target I / O request from the kernel buffer.

[0142] Finally, the second container node will further determine whether the target I / O request collected in real time is included in the above m I / O requests. If so, it will retrieve the target data corresponding to the target I / O request from its own kernel buffer.

[0143] It should be noted that in some other embodiments of this application, if the second container node determines that the target I / O request collected in real time is not included in the above m I / O requests, it needs to pull the file or data block corresponding to the target I / O request from the image repository according to the original pull process. This corresponding file or data block can be referred to as the target first data. The target first data is included in the container image data, and the container image data is stored in the image repository. Finally, the target data corresponding to the target I / O request is determined from the target first data in a manner similar to step 303 above.

[0144] It should also be noted that in the embodiments of this application, the data processing method corresponding to Figure 3 and the container task startup method corresponding to Figure 4 are generally performed on different container nodes. That is, the first container node and the second container node are generally different container nodes. However, in some practical applications, the first container node and the second container node can also be the same container node, and this application does not limit this.

[0145] Furthermore, to facilitate understanding of the data retrieval process of the second container node described above, a specific example is used below for illustration. Please refer to Figure 5 for details. Figure 5 is a flowchart illustrating the data retrieval process provided in this application embodiment. This retrieval process is completed based on the system architecture corresponding to Figure 1 or Figure 2 above. The retrieval process may specifically include the following steps:

[0146] 501. Initialize the I / O trajectory module.

[0147] First, this step initializes the I / O trajectory module 110 shown in Figure 1 (or the loading submodule 1002 shown in Figure 2 if it is Figure 2). Specifically, the I / O trajectory module 110 (or loading submodule 1002) is loaded into the kernel. This module can provide some capabilities required for kernel buffer pass-through.

[0148] 502. Start the container task and generate a unique task identifier based on the container task.

[0149] In this step, the second container node will start the container task and generate a task identifier to uniquely identify the container task. This process is similar to the method of identifying the task identifier in the embodiment corresponding to Figure 3 above, and will not be described in detail here.

[0150] 503. Obtain the manifest and config files from the mirror repository and parse them.

[0151] Afterwards, the second container node will retrieve the manifest and config files from the image repository based on the generated task identifier and parse them. The manifest file is the second manifest file obtained by modification as described above.

[0152] 504. Based on the parsing results, determine whether there is an I / O trajectory corresponding to the task identifier. If yes, proceed to step 505; otherwise, proceed to step 509.

[0153] After parsing the above file, it can be determined whether there is a corresponding I / O trace based on the parsing result. If yes, proceed to step 505; otherwise, proceed to step 509.

[0154] 505. Retrieve the corresponding I / O trace from the image repository based on the task identifier and save it to the kernel cache.

[0155] The second container node will pull the corresponding I / O trace from the image repository based on the generated task identifier and save it to the kernel cache.

[0156] 506. The container task begins creation and execution.

[0157] 507. Determine whether the I / O request is satisfied in the I / O trajectory. If yes, proceed to step 508; otherwise, proceed to step 510.

[0158] The second container node determines whether the currently collected I / O request (i.e. the target I / O request) is satisfied in the I / O trajectory. If yes, it executes step 508; otherwise, it executes step 510.

[0159] 508. Return the corresponding target data from the kernel cache.

[0160] The second container node returns the target data corresponding to the real-time collected target I / O request from its own kernel cache.

[0161] 509. The container task begins creation and execution.

[0162] 510. Load target data from the mirror repository.

[0163] The second container node loads the target data corresponding to the target I / O request from the image repository.

[0164] 511. Container task completed.

[0165] The container task is executed based on the target data returned from its own kernel cache, or the target data loaded from the image repository, until the container task starts successfully.

[0166] To facilitate understanding of the I / O trajectory management and I / O trajectory-based container task startup process described above, a specific example will be provided below. This example adapts the container image generation process to the open-source container management tool nerdctl, while the container task startup process incorporates I / O trajectory-based container startup functionality into the open-source nydus solution.

[0167] I. Container Image Building Phase

[0168] This stage corresponds to the I / O trajectory management process described above. Please refer to Figure 6 for details. Figure 6 is a component structure diagram provided in this embodiment. The container client used in the container image building stage is nerdctl. In the execution flow of nerdctl sending the image conversion command (convert) to the container service process, the trace option is added. An I / O trajectory tracing module called rio_tracker is introduced (i.e., the I / O trajectory module 110 described in the embodiment corresponding to Figure 1, or the tracing submodule 1001 described in the embodiment corresponding to Figure 2). The rio_tracker module uses EBPF technology to register I / O probes. In this example, only read I / O is traced; therefore, the kernel functions that need to be stubbed include, but are not limited to, vfs_read, kernel_read, and filemap_fault. Referring to the container I / O trajectory management process of this application, the main process of building a container image can include:

[0169] a. Load the rio_tracker module into the kernel.

[0170] b. Clear the cache in the specified directory and turn off the fault_around switch.

[0171] c. Send a command to the rio_tracker module to enable the IO trajectory tracking function.

[0172] d. Execute the image conversion command in nerdctl, specifying the target container task, to convert the traditional image. This process will run the target container task using the traditional image.

[0173] e. If a termination condition is specified in d above (e.g., a preset duration or a specified time), then the container task will be stopped after the termination condition is met, and a command to disable I / O track tracking will be sent to the rio_tracker module. Otherwise, wait for the container task started in d to finish running, or wait for the frequency of kernel-level I / O reads to fall below x times / minute (e.g., x = 5), where the threshold can be specified by the user, and then disable the I / O track tracking function.

[0174] f. Step e above records I / O requests from containers started based on traditional images to a file. These I / O requests may contain duplicates, overlaps, or invalid requests. This step restarts the container task and obtains the rootfs path of the new container. Through this path, file paths within the container can be accessed. Invalid I / O requests are filtered out by comparing the paths within the container with the files detected in the I / O records. For duplicates and overlaps, merging and deduplication are performed directly, ultimately resulting in the most concise I / O trajectory metadata. In this example, the format of the I / O trajectory metadata can be...<fullpath,offset,len> A series of sets, with offset and len both page-aligned, are used to represent a specific segment of a file read during an I / O operation. This metadata is then used to read data from the target file in the container task, ultimately merging the data corresponding to these I / O requests (i.e., the target data) into a large file, forming the I / O trace, before the target container is stopped.

[0175] g. Obtain the content of the target container task specified in d above and calculate its hash value; also calculate the hash value of the I / O trace generated in f above, form key / value pairs with this information and write them into the manifest file of this image, and repackage the I / O trace into the container image.

[0176] h. Push the container image with embedded I / O traces to the image repository.

[0177] It is important to note that in this example, the I / O trajectory includes not only I / O requests but also the corresponding target data. In the embodiment corresponding to Figure 3-5 above, the I / O trajectory only includes I / O requests.

[0178] II. Container Task Startup Phase

[0179] This stage corresponds to the container task startup process based on I / O trajectories described above. Please refer to Figure 7 for details. Figure 7 is another component structure diagram provided in this embodiment. This example is implemented based on the existing Nydus solution, in which an I / O trajectory startup path is added. An I / O trajectory loading module called io_trace is introduced (i.e., the I / O trajectory module 110 described in the embodiment corresponding to Figure 1 above, or the loading submodule 1002 described in the embodiment corresponding to Figure 2). Referring to the container task startup process based on I / O trajectories described above in this application, the main process of container task startup in this example may include:

[0180] a. Load the io_trace module into the kernel.

[0181] b. Execute the container task start command in nerdctl (i.e., the container client) and specify the container image data corresponding to the I / O trajectory.

[0182] c. The container service process downloads the manifest file from the image repository based on the image address, parses the I / O request information in the file, calculates the hash value using the task specified in b above, and then retrieves the hash value corresponding to the I / O request from the manifest file. If the corresponding I / O request information is not found, it reverts to the original container task startup process and downloads the config file from the image metadata. Otherwise, the I / O trajectory is added to the concurrent download process, and the downloaded I / O trajectory is stored in the kernel cache. It should be noted that in this example, the I / O trajectory includes not only the I / O request but also the corresponding target data. In the embodiment corresponding to Figure 3-5 above, the I / O trajectory only includes the I / O request.

[0183] d. After the container service process finishes downloading data, it creates a container task. Once the container task is created, it sends a command to the io_trace module to trigger trace metadata parsing. Trace metadata refers to...<fullpath,offset,len> The collection of data is used to build a hash table based on the full path. By using the full path, all I / O requests corresponding to that full path can be retrieved. Once the trace metadata index is built, the container task enters the running phase.

[0184] e. During the runtime phase, the image file will be opened, which will trigger the open operation. The open operation process is shown in Figure 8: First, the full path of the open file is obtained, and then the corresponding set of I / O requests is searched in the trace metadata through the full path. If it exists, the set of I / O requests is recorded in the private structure of the inode.

[0185] f. During the runtime phase, the image file is read, triggering file read and mmap operations. The execution flow is shown in Figure 9: For the read operation, a range match is performed on the I / O requests obtained during the open phase based on its offset and length. If the data is covered by the trace, the data in the trace kernel cache is directly copied to user space, and then the data reading is completed. For the mmap operation, the data reading process originates from a page fault. The offset within the file is calculated from the kernel file structure based on the virtual address (addr) of the page fault location. Then, a range match is performed on the I / O requests according to the page-aligned length. If the data is covered by the trace, the page in the trace kernel cache is directly mapped to the page fault virtual address, and then the mmap page fault process is completed. If the I / O requests do not match, the original process is executed.

[0186] g. After the target data is read, proceed with the container startup process until the container starts successfully.

[0187] Based on the above embodiments, in order to better implement the above-described solution of this application embodiment, related equipment for implementing the above solution is also provided below. Specifically, refer to Figure 10, which is a schematic diagram of a computer device provided in an embodiment of this application. The computer device 1000 serves as a first container node and may specifically include: a first determining module 1001, a pulling module 1002, a second determining module 1003, a third determining module 1004, and a sending module 1005. The first determining module 1001 is used to determine m I / O requests generated during the startup process of the container task, where m ≥ 1; the pulling module 1002 is used to obtain the first data corresponding to each of the m I / O requests from the image repository during the startup process, obtaining a total of m first data items. These m first data items belong to... The container image data corresponding to the container task is stored in the image repository; the second determining module 1003 is used to determine the target data corresponding to each of the m I / O requests from the m first data, resulting in a total of m target data, which are the necessary data included for the successful startup of the container task; the third determining module 1004 is used to determine the first mapping relationship between the m I / O requests and the m target data; the sending module 1005 is used to send second data to the image repository so that the image repository saves the second data, which includes the m I / O requests and the first mapping relationship.

[0188] In one possible design, the second data also includes: the m target data.

[0189] In one possible design, the first determining module 1001 is further configured to: determine a task identifier corresponding to the container task based on the container task before determining the m I / O requests generated during the startup process of the container task, and obtain a first digest file from the container image data from the image repository; the third determining module 1004 is further configured to: obtain a second digest file based on the task identifier, the m I / O requests and the first digest file before the sending module 1005 sends the second data to the image repository, and the second digest file is included in the second data.

[0190] In one possible design, the third determining module 1004 is specifically used to: when the second data does not include the m target data, record the first hash value corresponding to the task identifier and the second hash value corresponding to the m I / O requests in the first digest file to obtain the second digest file.

[0191] In one possible design, the third determining module 1004 is specifically used to: when the second data includes the m target data, record the first hash value corresponding to the task identifier, the second hash value corresponding to the m I / O requests, and the third hash value corresponding to the m target data in the first digest file to obtain the second digest file.

[0192] In one possible design, the first determining module 1001 is specifically used to: generate a task identifier corresponding to the container task based on the container task; or, assign a corresponding task identifier to the container task based on a pre-built second mapping relationship between each container task and each task identifier.

[0193] In one possible design, the first determining module 1001 is specifically used to: collect n I / O requests generated during the startup process of the container task, where n ≥ m; and determine m I / O requests from these n I / O requests.

[0194] In one possible design, the first determining module 1001 is specifically used to: remove the first I / O request from the n I / O requests to obtain m I / O requests; wherein the first target data corresponding to the first I / O request is included in the second target data corresponding to the second I / O request, the second I / O request is included in the n I / O requests, or the first I / O request does not belong to the I / O requests of the container task.

[0195] In one possible design, the first determining module 1001 is specifically used to: track the I / O requests generated during the startup process using an I / O probe, which is pre-registered on the first container node; and stop the tracking when a termination condition is met, wherein the n I / O requests are the I / O requests collected during the tracking period.

[0196] In one possible design, the termination condition includes: the container task has completed its startup execution; or, the startup duration of the container task has reached a preset duration; or, the I / O request volume of the container task per unit time during the startup process is lower than a preset threshold.

[0197] It should be noted that the information interaction and execution process between the modules / units in the computer device 1000 are based on the same concept as the embodiment corresponding to Figure 3 in this application. For details, please refer to the description in the method embodiment shown above in this application, which will not be repeated here.

[0198] This application embodiment also provides a computer device. Please refer to Figure 11. Figure 11 is a schematic diagram of another structure of the computer device provided in this application embodiment. The computer device 1100 serves as a second container node and may specifically include: a determining module 1101, a saving module 1102, a collecting module 1103, and a retrieving module 1104. The determining module 1101 is used to determine a target request based on the currently scheduled container task to be started, and send the target request to the image repository. The target request instructs the image repository to send m I / O requests corresponding to the container task and m target data corresponding to the m I / O requests to the second container node. The m I / O requests are generated by the first container node when starting the container task. The process generates m target data, which are obtained based on a first mapping relationship. This first mapping relationship is the mapping relationship between the m I / O requests and the m target data. The m target data are the necessary data included for the successful startup of the container task. One I / O request corresponds to one target data, and m ≥ 1. The storage module 1102 is used to store the m I / O requests and the m target data sent by the image repository in the kernel page cache. The acquisition module 1103 is used to start the container task and acquire the target I / O requests generated during the startup process. The retrieval module 1104 is used to retrieve the target data corresponding to the target I / O request from the kernel buffer when the target I / O request is included in the m I / O requests.

[0199] In one possible design, the determining module 1101 is specifically used for: determining a task identifier corresponding to the container task based on the currently scheduled container task to be started; obtaining a second digest file of the container image data corresponding to the container task from the image repository based on the task identifier, the second digest file being obtained by the first container node based on the task identifier, the m I / O requests and the first digest file, the second digest file being sent to the image repository by the first container node; and determining the target request based on the second digest file.

[0200] In one possible design, the determining module 1101 is specifically used to: determine the target request based on the first hash value and the second hash value in the second digest file; or, determine the target request based on the first hash value, the second hash value, and the third hash value in the second digest file; wherein the first hash value is the hash value corresponding to the task identifier, the second hash value is the hash value corresponding to the m I / O requests, and the third hash value is the hash value corresponding to the m target data.

[0201] In one possible design, the computer device 1100 further includes a pull module 1105, which is used to retrieve target first data corresponding to the target I / O request from the image repository when the target I / O request is not included in the m I / O requests. The target first data is included in the container image data and the container image data is stored in the image repository. The target data corresponding to the target I / O request is determined from the target first data.

[0202] It should be noted that the information interaction and execution process between the modules / units in the computer device 1100 are based on the same concept as the embodiments corresponding to Figures 4 and 5 in this application. For details, please refer to the description in the method embodiments shown above in this application, which will not be repeated here.

[0203] Next, we will introduce another computer device provided in the embodiments of this application. Please refer to Figure 12. Figure 12 is a schematic diagram of the structure of a computer device provided in the embodiments of this application. The computer device 1200 may deploy the computer device 1000 described in the embodiment corresponding to Figure 10, for implementing the functions of each module in the embodiment corresponding to Figure 10. The computer device 1200 may also deploy the computer device 1100 described in the embodiment corresponding to Figure 11, for implementing the functions of each module in the embodiment corresponding to Figure 10. Specifically, the computer device 1200 is implemented by one or more servers. The computer device 1200 may vary considerably due to different configurations or performance. It may include one or more central processing units (CPUs) 1222 and memory 1232, and one or more storage media 1230 (e.g., one or more mass storage devices) for storing application programs 1242 or data 1244. The memory 1232 and storage media 1230 may be temporary storage or persistent storage. The program stored in storage medium 1230 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on computer device 1200. Furthermore, central processing unit 1222 may be configured to communicate with storage medium 1230 and execute the series of instruction operations in storage medium 1230 on computer device 1200.

[0204] Computer device 1200 may also include one or more power supplies 1226, one or more wired or wireless network interfaces 1250, one or more input / output interfaces 1258, and / or one or more operating systems 1241, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0205] In this embodiment, the central processing unit 1222 is used to execute the steps performed by the container node in the embodiments corresponding to Figures 3 to 5, or Figures 8 to 9. For example, the central processing unit 1222 can be used to: first, determine m I / O requests generated during the startup process of the container task, where m ≥ 1; then, during the startup process of the container task, obtain the first data corresponding to each of the m I / O requests from the image repository, resulting in a total of m first data, which belong to the container image data corresponding to the container task, and the container image data is stored in the image repository; then, determine the target data corresponding to each of the m I / O requests from the m first data, resulting in a total of m target data, and determine the first mapping relationship between the m I / O requests and the m target data, wherein the m target data are the necessary data included for the successful startup of the container task; finally, send second data to the image repository so that the image repository saves the second data, which includes the m I / O requests and the first mapping relationship.

[0206] It should be noted that the specific way in which the central processing unit 1222 executes the above steps is based on the same concept as the method embodiments corresponding to Figures 3 to 5 or Figures 8 to 9 in this application, and the resulting technical effects are also the same as those in the above embodiments of this application. For details, please refer to the description in the method embodiments shown above in this application, which will not be repeated here.

[0207] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0208] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0209] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0210] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A data processing method, characterized in that, Applied to the first container node, including: Determine the m I / O requests generated by the container task during startup, where m ≥ 1; During the startup process, the first data corresponding to each of the m I / O requests is obtained from the image repository, resulting in a total of m first data. The m first data belong to the container image data corresponding to the container task, and the container image data is stored in the image repository. From the m first data, the target data corresponding to each of the m I / O requests is determined, resulting in a total of m target data. The m target data are the necessary data included for the successful startup of the container task. Determine a first mapping relationship between the m I / O requests and the m target data; Send second data to the image repository so that the image repository saves the second data, the second data including the m I / O requests and the first mapping relationship.

2. The method according to claim 1, characterized in that, The second data also includes: The m target data.

3. The method according to any one of claims 1-2, characterized in that, Before determining the m I / O requests generated by the container task during startup, the method further includes: The task identifier corresponding to the container task is determined based on the container task, and the first digest file in the container image data is obtained from the image repository; Before sending the second data to the mirror repository, the method further includes: A second digest file is obtained based on the task identifier, the m I / O requests, and the first digest file, and the second digest file is contained in the second data.

4. The method according to claim 3, characterized in that, The step of obtaining the second digest file based on the task identifier, the m I / O requests, and the first digest file includes: If the second data does not include the m target data, the first hash value corresponding to the task identifier and the second hash value corresponding to the m I / O requests are recorded in the first digest file to obtain the second digest file.

5. The method according to claim 3, characterized in that, The step of obtaining the second digest file based on the task identifier, the m I / O requests, and the first digest file includes: When the second data includes the m target data, the first hash value corresponding to the task identifier, the second hash value corresponding to the m I / O requests, and the third hash value corresponding to the m target data are recorded in the first digest file to obtain the second digest file.

6. The method according to any one of claims 3-5, characterized in that, The step of determining the task identifier corresponding to the container task based on the container task includes: Based on the container task, generate a task identifier corresponding to the container task; or, Based on the pre-constructed second mapping relationship between each container task and each task identifier, a corresponding task identifier is assigned to each container task.

7. The method according to any one of claims 1-6, characterized in that, The determination of the m I / O requests generated by the container task during startup includes: Collect n I / O requests generated by the container task during startup, where n ≥ m; Determine m I / O requests from the n I / O requests.

8. The method according to claim 7, characterized in that, The step of determining m I / O requests from the n I / O requests includes: The first I / O request among the n I / O requests is removed to obtain m I / O requests; Wherein, the first target data corresponding to the first I / O request is included in the second target data corresponding to the second I / O request, the second I / O request is included in the n I / O requests, or the first I / O request does not belong to the I / O request of the container task.

9. The method according to any one of claims 6-7, characterized in that, The n I / O requests generated during the startup process of the data acquisition container task include: The I / O requests generated during the startup process are tracked using I / O probes, which are pre-registered with the first container node; If the termination condition is met, the tracing is stopped, and the n I / O requests are the I / O requests collected during the tracing.

10. The method according to claim 9, characterized in that, The termination conditions include: The container task has been started and executed. or, The execution time of the container task reaches the preset time. or, The number of I / O requests per unit time during the startup process of the container task is lower than a preset threshold.

11. A method for starting a container task, characterized in that, Applied to the second container node, including: The target request is determined based on the container task to be started, and the target request is sent to the image repository. The target request is used to instruct the image repository to send m I / O requests corresponding to the container task and m target data corresponding to the m I / O requests to the second container node. The m I / O requests are generated by the first container node during the start of the container task. The m target data are obtained based on a first mapping relationship. The first mapping relationship is the mapping relationship between the m I / O requests and the m target data. The m target data are the necessary data included for the successful start of the container task. One I / O request corresponds to one target data, and m≥1. The m I / O requests and m target data sent by the image repository are stored in the kernel page cache; Start the container task and collect the target I / O requests generated during the startup process; If the target I / O request is included in the m I / O requests, the target data corresponding to the target I / O request is retrieved from the kernel buffer.

12. The method according to claim 11, characterized in that, The process of determining the target request based on the currently pending container task includes: Determine the task identifier corresponding to the container task based on the container task to be started; According to the task identifier, a second digest file of the container image data corresponding to the container task is obtained from the image repository. The second digest file is obtained by the first container node based on the task identifier, the m I / O requests and the first digest file. The second digest file is sent to the image repository by the first container node. The target request is determined based on the second summary file.

13. The method according to claim 12, characterized in that, The step of determining the target request based on the second digest file includes: The target request is determined based on the first hash value and the second hash value in the second digest file; or, The target request is determined based on the first hash value, the second hash value, and the third hash value in the second digest file; Wherein, the first hash value is the hash value corresponding to the task identifier, the second hash value is the hash value corresponding to the m I / O requests, and the third hash value is the hash value corresponding to the m target data.

14. The method according to any one of claims 11-13, characterized in that, The method further includes: If the target I / O request is not included in the m I / O requests, the target first data corresponding to the target I / O request is obtained from the image repository. The target first data is included in the container image data, and the container image data is stored in the image repository. The target data corresponding to the target I / O request is determined from the target first data.

15. A computer device, characterized in that, The computer device, serving as the first container node, includes: The first determining module is used to determine the m I / O requests generated by the container task during startup, where m ≥ 1; The pull module is used to retrieve the first data corresponding to each of the m I / O requests from the image repository during the startup process, resulting in a total of m first data. The m first data belong to the container image data corresponding to the container task, and the container image data is stored in the image repository. The second determining module is used to determine the target data corresponding to each of the m I / O requests from the m first data, and obtain a total of m target data. The m target data are the necessary data included for the successful startup of the container task. The third determining module is used to determine the first mapping relationship between the m I / O requests and the m target data; The sending module is used to send second data to the image repository so that the image repository saves the second data, the second data including the m I / O requests and the first mapping relationship.

16. The device according to claim 15, characterized in that, The second data also includes: The m target data.

17. The device according to any one of claims 15-16, characterized in that, The first determining module is further configured to: Before determining the m I / O requests generated by the container task during startup, a task identifier corresponding to the container task is determined based on the container task, and a first digest file in the container image data is obtained from the image repository. The third determining module is further configured to obtain a second digest file based on the task identifier, the m I / O requests, and the first digest file before the sending module sends the second data to the mirror repository, wherein the second digest file is contained in the second data.

18. The device according to claim 17, characterized in that, The third determining module is specifically used for: If the second data does not include the m target data, the first hash value corresponding to the task identifier and the second hash value corresponding to the m I / O requests are recorded in the first digest file to obtain the second digest file.

19. The device according to claim 17, characterized in that, The third determining module is specifically used for: When the second data includes the m target data, the first hash value corresponding to the task identifier, the second hash value corresponding to the m I / O requests, and the third hash value corresponding to the m target data are recorded in the first digest file to obtain the second digest file.

20. The device according to any one of claims 17-19, characterized in that, The first determining module is specifically used for: Based on the container task, generate a task identifier corresponding to the container task; or, Based on the pre-constructed second mapping relationship between each container task and each task identifier, a corresponding task identifier is assigned to each container task.

21. The device according to any one of claims 15-20, characterized in that, The first determining module is specifically used for: Collect n I / O requests generated by the container task during startup, where n ≥ m; Determine m I / O requests from the n I / O requests.

22. The device according to claim 21, characterized in that, The first determining module is specifically used for: The first I / O request among the n I / O requests is removed to obtain m I / O requests; Wherein, the first target data corresponding to the first I / O request is included in the second target data corresponding to the second I / O request, the second I / O request is included in the n I / O requests, or the first I / O request does not belong to the I / O request of the container task.

23. The device according to any one of claims 20-21, characterized in that, The first determining module is specifically used for: The I / O requests generated during the startup process are tracked using I / O probes, which are pre-registered with the first container node; If the termination condition is met, the tracing is stopped, and the n I / O requests are the I / O requests collected during the tracing.

24. The device according to claim 23, characterized in that, The termination conditions include: The container task has been started and executed. or, The execution time of the container task reaches the preset time. or, The number of I / O requests per unit time during the startup process of the container task is lower than a preset threshold.

25. A computer device, characterized in that, The computer device, serving as the second container node, includes: The determination module is used to determine the target request based on the container task to be started, and send the target request to the image repository. The target request is used to instruct the image repository to send m I / O requests corresponding to the container task and m target data corresponding to the m I / O requests to the second container node. The m I / O requests are generated by the first container node during the start of the container task. The m target data are obtained based on a first mapping relationship. The first mapping relationship is the mapping relationship between the m I / O requests and the m target data. The m target data are the necessary data included for the successful start of the container task. One I / O request corresponds to one target data, and m≥1. A storage module is used to store the m I / O requests and the m target data sent by the image repository in the kernel page cache; The acquisition module is used to start the container task and acquire the target I / O requests generated during the startup process; The retrieval module is used to retrieve target data corresponding to the target I / O request from the kernel buffer when the target I / O request is included in the m I / O requests.

26. The device according to claim 25, characterized in that, The determining module is specifically used for: Determine the task identifier corresponding to the container task based on the container task to be started; According to the task identifier, a second digest file of the container image data corresponding to the container task is obtained from the image repository. The second digest file is obtained by the first container node based on the task identifier, the m I / O requests and the first digest file. The second digest file is sent to the image repository by the first container node. The target request is determined based on the second summary file.

27. The device according to claim 26, characterized in that, The determining module is specifically used for: The target request is determined based on the first hash value and the second hash value in the second digest file; or, The target request is determined based on the first hash value, the second hash value, and the third hash value in the second digest file; Wherein, the first hash value is the hash value corresponding to the task identifier, the second hash value is the hash value corresponding to the m I / O requests, and the third hash value is the hash value corresponding to the m target data.

28. The device according to any one of claims 25-27, characterized in that, The device also includes: a pull module; The pull module is used to retrieve target first data corresponding to the target I / O request from the image repository when the target I / O request is not included in the m I / O requests. The target first data is included in the container image data, and the container image data is stored in the image repository. The target data corresponding to the target I / O request is determined from the target first data.

29. A computer device comprising a processor and a memory, the processor being coupled to the memory, characterized in that, The memory is used to store programs; The processor is configured to execute a program in the memory, causing the computer device to perform the method as described in any one of claims 1-14.

30. A computer storage medium, characterized in that, The device stores computer-readable instructions, which, when executed by a processor, implement the method as described in any one of claims 1-14.

31. A computer program product, characterized in that, The computer program product includes computer-readable instructions that, when executed by a processor, implement the method as described in any one of claims 1-14.

32. A chip, the chip comprising a processor and a data interface, characterized in that, The processor reads instructions stored in the memory through the data interface and executes the method as described in any one of claims 1-14.

Images