Data processing method and computer equipment

Abstract

The invention provides a data processing method and computer equipment, which can be applied to the field of data processing, and comprises the following steps: accurately tracking I / O (Input / Output) requests in a container task starting process, constructing a mapping relationship between each I / O request and data (namely target data) necessary for starting a container, effective I / O requests (namely m I / O requests) generated in the starting process and the mapping relation are sent to the mirror image warehouse; the effective I / O requests generated when the same container task is started each time are the same, so that when the container task is started again, the I / O request quantity sent to the mirror image warehouse in the container task starting process can be reduced (m I / O requests are known and can be aggregated to be sent to the mirror image warehouse at one time), and the task starting efficiency is improved. Meanwhile, when the container task is started, the corresponding m target data can be pulled from the mirror image warehouse on the basis of the pre-constructed mapping relation at a time, repeated pulling is not needed, the container task starting efficiency is improved, and the data pulling amount is reduced.

Description

Technical Field

[0001] This application relates to the field of data processing, and more particularly to a data processing method and a computer device. Background Technology

[0002] With the development of cloud computing, the use of containers is becoming increasingly widespread. Due to their lightweight, isolation, and portability, containers have undoubtedly become an indispensable tool in cloud-native scenarios. Today, containerization has become the cornerstone of modern software infrastructure. As various applications demand higher quality of service, the rapid deployment of containerized applications has become paramount. Container startup relies on container image data, which is typically stored in an image repository and managed by a single node or cluster. An image service process provides various operations on the container image data. The node that starts the container (or the node running the container, or simply the container node) is often a different physical machine or virtual machine than the image repository. Container startup requires the use of container image data (or simply image data), thus often involving data interaction between the container node and the image repository. Container startup can generally be divided into two cases: cold start and warm start. A cold start refers to starting the container directly when the container image data is not on the container node; a warm start refers to starting the container after the container image data is already on the container node. The container startup process mainly consists of two parts: image preparation and container startup. Therefore, startup latency is primarily affected by these two parts. During image preparation, the container node checks if the container image data exists on its local machine. If not, it downloads the corresponding container image data from the image repository, decompresses it, and stores it locally. During container startup, the container node accesses this container image data to execute container tasks. Image preparation and container startup cannot run concurrently; the latter is highly dependent on the former.

[0003] Currently, there are generally two methods for starting container tasks during cold starts: 1) Full pull method: The container startup process first pulls the full data of the corresponding container image from the image repository, and then uses this data to start the container. 2) On-demand pull method: Instead of pulling the full container image data at startup, the container downloads the corresponding data from the image repository based on the data that the container depends on. Furthermore, traditional image formats (container images encapsulate the container application and its dependent file data, usually packaged according to a certain format, commonly known as the Open Container Initiative (OCI) image format) compress and package container image data, and do not support on-demand loading. Therefore, to support on-demand loading, the image format needs to be processed to support the indexability of on-demand files, but all of this must meet the OCI specification. Specifically, the on-demand retrieval method can be: Method 1, retrieve the image data of the target file on demand, because the full image data consists of multiple files, and the granularity of on-demand loading is the entire file; Method 2, retrieve large-granularity image data on demand, that is, cut the full container image data into chunks, and the granularity of on-demand loading is the chunked image data.

[0004] However, while the full-data retrieval method described above is very easy to implement and understand, it is also very inefficient. Retrieving container image data requires crossing the network, which is a very time-consuming operation. Furthermore, the larger the data volume of the container image, the longer this process takes. This cannot meet the demands of cloud-native scenarios where containers are commonly used. Additionally, this startup method relies on downloading the full container image data, but in reality, the container startup process only relies on a portion of the container image data. Research shows that during the startup process of a container task, the time spent retrieval of the full container image data accounts for 76% of the entire startup process, while the data actually used during startup only accounts for 6.4% of the full container image data. Similarly, in the on-demand retrieval method, Method 1 loads the entire file on demand, but the container task may not necessarily need the entire file's data when starting, so this method still retrieves redundant data. Likewise, in Method 2 of the on-demand retrieval method, due to technical limitations, the granularity of the image data chunks is very large, typically 1MB, which also leads to the problem of retrieving redundant data. Furthermore, the on-demand loading input / output (I / O) behavior is not user-friendly because it triggers the fetching of container image data multiple times. That is, every time the container image data to be read is not local, the on-demand loading process needs to be triggered. In other words, each I / O request triggers a loading process, which means that every subsequent I / O request for a file / data block will interact with the image repository, making the program run longer. Summary of the Invention

[0005] This application provides a data processing method and computer device for accurately tracking I / O requests during container task startup and constructing a mapping relationship between each I / O request and the data required for container startup (i.e., target data). All valid I / O requests (i.e., m I / O requests) generated during the container task startup process, along with their corresponding mapping relationships (i.e., second data), are sent to an image repository, allowing the image repository to package and store them together with the existing container image data. Since the valid I / O requests generated each time the same container task is started are the same, when other container nodes start the same container task, the number of I / O requests sent to the image repository during the container task startup process can be reduced (because the m I / O requests are known, they can be aggregated and sent to the image repository all at once). Simultaneously, during container task startup, the corresponding m target data can be pulled from the image repository at once based on the pre-constructed mapping relationship, eliminating the need for multiple pulls, thus improving the efficiency of container task startup and reducing the amount of data pulled.

[0006] Based on this, the embodiments of this application provide the following technical solutions:

[0007] Firstly, this application provides a data processing method applied to a first container node. This method specifically includes: First, the first container node determines m I / O requests generated during the startup process of the container task to be started (also called the target container task), where m ≥ 1. The set of these I / O requests is called the I / O trajectory. During the startup process of the container task, the first container node can obtain the files or data blocks (called first data) corresponding to each of the m I / O requests from the image repository. One I / O request corresponds to one piece of first data, thus a total of m pieces of first data can be obtained. These m pieces of first data belong to the container image data corresponding to the container task, and the container image data is stored in the image repository. Then, the first container node determines the target data corresponding to each I / O request from the obtained m pieces of first data, i.e., one I / O request corresponds to one piece of target data, thus obtaining a total of m pieces of target data. These m pieces of target data are the necessary data included for the successful startup of the container task, and these m pieces of target data do not contain any redundant data unrelated to startup. Therefore, the m pieces of target data are the minimum amount of data required for the successful startup of the container task. After receiving m target data items, the first container node further determines the mapping relationship between the m I / O requests and the m target data items; this mapping relationship can be called the first mapping relationship. Finally, the first container node sends second data to the image repository so that the image repository can save the second data, which includes the m I / O requests and the first mapping relationship. For example, after receiving the second data, the image repository can package and store the second data together with the original container image data.

[0008] In the above embodiments of this application, by accurately tracking I / O requests during the container task startup process and constructing a mapping relationship between each I / O request and the target data necessary for container startup, all valid m I / O requests generated during the container task startup process, along with their corresponding mapping relationships (i.e., the second data), are sent to the image repository for storage. Since the valid I / O requests generated each time the same container task is started are the same, when other container nodes start the same container task, the m I / O requests can be aggregated and sent to the image repository at once, reducing the amount of I / O requests sent to the image repository during the container task startup process. Simultaneously, when the container task starts, the corresponding m target data can be pulled from the image repository at once based on the pre-constructed mapping relationship, eliminating the need for multiple pulls and improving the efficiency of container task startup. Furthermore, since the m target data are necessary for container startup, no redundant data is pulled during the data retrieval process, thus achieving a minimum amount of data retrieval.

[0009] In one possible implementation of the first aspect, the second data may further include the m target data described above.

[0010] In the above embodiments of this application, when a container node needs to obtain m target data corresponding to the container task, the image repository can send them directly without the image repository needing to decompress the original container image data based on the m I / O requests and the first mapping relationship, then read the decompressed files or data blocks, and then extract the required m target data from the decompressed files or data blocks. Since the direct sending method eliminates the need for the image repository to parse this data from the original container image data, the container node can directly pull it, while the image repository's decompression → reading → extraction process is relatively more time-consuming.

[0011] In one possible implementation of the first aspect, before determining the m I / O requests generated by the container task during startup, the method may further determine the task identifier corresponding to the container task based on the task content during container runtime, and obtain the first manifest file from the container image data corresponding to the container task from the image repository. Before sending the second data to the image repository, the second manifest file can be obtained based on the task identifier, the m I / O requests, and the first manifest file, and included in the second data to be sent.

[0012] In the above embodiments of this application, the task identifier serves as a unique index in the I / O trajectory. When the container task is restarted (e.g., on a second container node), it is used to index the corresponding I / O trajectory and target data in the image, thereby improving the accuracy of data retrieval.

[0013] In one possible implementation of the first aspect, one way to obtain the second digest file based on the task identifier, m I / O requests, and the first digest file is as follows: when the second data does not include m target data, the first hash value corresponding to the task identifier and the second hash value corresponding to the m I / O requests are recorded in the first digest file to obtain the second digest file.

[0014] In the above embodiments of this application, a specific implementation method for obtaining a second digest file by modifying the first digest file is described. This method is obtained by additionally recording two hash values ​​and is feasible.

[0015] In one possible implementation of the first aspect, another way to obtain the second digest file based on the task identifier, m I / O requests, and the first digest file is: when the second data includes m target data, record the first hash value corresponding to the task identifier, the second hash value corresponding to the m I / O requests, and the third hash value corresponding to the m target data in the first digest file to obtain the second digest file.

[0016] In the above embodiments of this application, another implementation method for obtaining a second digest file by modifying the first digest file is specifically described. This method is obtained by additionally recording 3 hash values ​​and is feasible.

[0017] In one possible implementation of the first aspect, determining the task identifier corresponding to the container task based on the container task could be: 1) Directly generating the corresponding task identifier based on the container task, for example, the task identifier could be a generated hash value. 2) A mapping relationship between each container task and each task identifier can be pre-built, with each container task corresponding to a unique task identifier. This mapping relationship can be placed on the first container node. When the first container node wants to start the aforementioned container task (i.e., the target container task), it can directly assign the corresponding task identifier to the target container task to be started based on this mapping relationship.

[0018] In the above embodiments of this application, several implementation methods for obtaining task identifiers are specifically described, which are flexible.

[0019] In one possible implementation of the first aspect, one way to determine the m I / O requests generated by the container task during startup is as follows: First, collect the I / O requests generated by the container task during startup in real time. Assume that a total of n I / O requests are collected during the entire startup process, where n ≥ m. Then, determine the m I / O requests related to the container task from these n collected I / O requests. This way, the I / O trajectory with the minimum amount of metadata can be obtained.

[0020] In the above embodiments of this application, it is specifically described that m valid I / O requests are determined from n I / O requests collected in real time. Since these m I / O requests are valid I / O requests, the m target data corresponding to them are the necessary data required for the container task to start successfully, and do not contain any redundant data that is not needed for the container task to start, thereby minimizing the amount of data pulled by the container node.

[0021] In one possible implementation of the first aspect, determining m I / O requests from n I / O requests can be achieved by: removing the first I / O request from the n I / O requests to obtain m I / O requests, wherein the first target data corresponding to the first I / O request is included in the second target data corresponding to the second I / O request, the second I / O request is included in the n I / O requests, or the first I / O request does not belong to the I / O requests of the container task.

[0022] In the above embodiments of this application, a specific implementation method for determining m I / O requests is described, which can guarantee that all m I / O requests obtained are valid I / O requests.

[0023] In one possible implementation of the first aspect, one way to collect the n I / O requests generated during the startup process of the container task is as follows: First, the first container node pre-registers an I / O probe (also called an I / O probe point) to track the I / O requests generated during startup in real time. Specifically, the first container node can further analyze the file offset and the length of the corresponding target data of the I / O request being tracked based on the registered I / O probe, and perform page alignment processing (to improve reading efficiency). At the same time, it can find out the internal file path of the container corresponding to the I / O request and append this data as an I / O trace to the file. When the termination condition is met, the tracking stops, and the n I / O requests are the I / O requests collected during the tracking period.

[0024] In the above embodiments of this application, I / O probes are used to accurately track I / O requests during the container task startup process, thereby improving tracking accuracy and efficiency.

[0025] In one possible implementation of the first aspect, the termination conditions for the above collection include, but are not limited to: a) the container task has been started and executed; b) the execution time of the container task reaches a preset time; c) the number of I / O requests per unit time during the start-up process of the container task is lower than a preset threshold.

[0026] The above embodiments of this application specifically describe several scenarios for data collection termination conditions, which have broad applicability.

[0027] A second aspect of this application provides a method for starting a container task, applied to a second container node. The method specifically includes: First, the second container node determines a request to be sent to an image repository based on the currently scheduled container task to be started (i.e., the target container task). This request can be called a target request. After determining the target request, the target request is sent to the image repository. The target request instructs the image repository to send m I / O requests corresponding to the currently scheduled container task and m target data corresponding to these m I / O requests to the second container node. It should be noted that in this embodiment, these m I / O requests are generated by the first container node during the startup of the container task. These m target data are obtained based on a first mapping relationship, which is the mapping relationship between these m I / O requests and these m target data. Furthermore, these m target data are necessary data for the successful startup of the container task; one I / O request corresponds to one target data, and m ≥ 1. Afterward, the second container node stores the m I / O requests and m target data sent by the image repository in its kernel page cache. The second container node starts the container task and, during the startup process, collects the I / O requests generated in real time. These I / O requests are called target I / O requests. Finally, the second container node further determines whether the real-time collected target I / O request is included in the above m I / O requests. If so, it retrieves the target data corresponding to the target I / O request from its own kernel buffer.

[0028] In the above embodiments of this application, when the second container node re-executes the container task (which has previously run on the first container node and established a first mapping relationship between m I / O requests and m target data), since the effective I / O requests generated each time the same container task is started are the same, the corresponding m target data can be pulled from the image repository at once based on the pre-built first mapping relationship, eliminating the need for multiple pulls. Furthermore, the pulled data is directly imported into the kernel page cache, reducing memory copy overhead.

[0029] In one possible implementation of the second aspect, determining the target request based on the currently pending container task can be as follows: First, the second container node determines the task identifier corresponding to the currently pending container task. Then, based on the task identifier, it retrieves a summary file of the container image data corresponding to the container task from the image repository, and determines the target request based on this second summary file. The second summary file is obtained by the first container node based on the task identifier, m I / O requests, and the original first summary file. After modifying the summary file, the first container node sends it to the image repository for storage.

[0030] In the above embodiments of this application, the target request is specifically described as being determined based on the task identifier and the second digest file, which is feasible.

[0031] In one possible implementation of the second aspect, determining the target request based on the second digest file can be done by: determining the target request based on the first hash value and the second hash value in the second digest file; or, determining the target request based on the first hash value, the second hash value, and the third hash value in the second digest file; wherein the first hash value is the hash value corresponding to the task identifier, the second hash value is the hash value corresponding to m I / O requests, and the third hash value is the hash value corresponding to m target data.

[0032] In the above embodiments of this application, one specific implementation of determining the target request based on the second digest file is described, which is based on two or three hash values ​​in the digest file, and has flexibility.

[0033] In one possible implementation of the second aspect, if the second container node determines that the target I / O request for real-time collection is not included in the aforementioned m I / O requests, it needs to pull the file or data block corresponding to the target I / O request from the image repository according to the original pull process. This corresponding file or data block can be referred to as the target first data, which is included in the container image data, and the container image data is stored in the image repository. Finally, the target data corresponding to the target I / O request is determined from the target first data in a manner similar to step 303 above.

[0034] In the above embodiments of this application, if the target I / O request generated in real time does not belong to m I / O requests, the original data retrieval method is followed. This ensures that the container node can still successfully retrieve the corresponding data, improving the flexibility of execution.

[0035] A third aspect of this application provides a computer device that serves as a first container node and has the functionality to implement the method described in the first aspect or any possible implementation of the first aspect. This functionality can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the described functionality.

[0036] A fourth aspect of this application provides a computer device that serves as a second container node and has the functionality to implement the method described in the second aspect or any possible implementation of the second aspect. This functionality can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the described functionality.

[0037] The fifth aspect of this application provides a computer device, which may include a memory, a processor, and a bus system, wherein the memory is used to store a computer program (also referred to as a program or computer-readable instructions), and the processor is used to call the program stored in the memory to execute the method of the first aspect of the embodiments of this application or any possible implementation of the first aspect, or to execute the method of the second aspect of the embodiments of this application or any possible implementation of the second aspect.

[0038] The sixth aspect of this application provides a computer-readable storage medium storing instructions that, when executed on a computer, enable the computer to perform the method of the first aspect or any possible implementation thereof, or enable the computer to perform the method of the second aspect or any possible implementation thereof.

[0039] The seventh aspect of this application provides a computer program or a computer program product containing instructions that, when the computer program or computer program product is run on a computer, causes the computer to perform the method of the first aspect or any possible implementation thereof, or causes the computer to perform the method of the second aspect or any possible implementation thereof.

[0040] An eighth aspect of this application provides a chip including at least one processor and at least one interface circuit coupled to the processor. The at least one interface circuit is used to perform transceiver functions and send instructions to the at least one processor. The at least one processor is used to run computer programs or instructions, having the functionality to implement the methods described in the first aspect or any possible implementation of the first aspect, or having the functionality to implement the methods described in the second aspect or any possible implementation of the second aspect. This functionality can be implemented in hardware, software, or a combination of hardware and software, including one or more modules corresponding to the above-described functions. Furthermore, the interface circuit is used to communicate with other modules outside the chip.

[0041] In some implementations of this application, some of the one or more processors may implement some steps of the above method through dedicated hardware. For example, the processing involving neural network models may be implemented by a dedicated neural network processor or graphics processor.

[0042] The method provided in this application embodiment can be implemented by a single chip or by multiple chips working together. Attached Figure Description

[0043] Figure 1 A schematic diagram of the system architecture provided in the embodiments of this application;

[0044] Figure 2 Another schematic diagram of the system architecture provided in the embodiments of this application;

[0045] Figure 3 A flowchart illustrating the data processing method provided in this application embodiment;

[0046] Figure 4 A flowchart illustrating a method for starting a container task provided in an embodiment of this application;

[0047] Figure 5 A flowchart illustrating the data retrieval process provided in this application embodiment;

[0048] Figure 6 A component structure diagram provided for an embodiment of this application;

[0049] Figure 7 Another component structure diagram provided for an embodiment of this application;

[0050] Figure 8 A schematic flowchart illustrating the file system opening operation provided in this application embodiment;

[0051] Figure 9A flowchart illustrating a file reading / file mapping operation provided in an embodiment of this application;

[0052] Figure 10 A schematic diagram of the structure of a computer device provided in an embodiment of this application;

[0053] Figure 11 Another structural schematic diagram of the computer device provided in the embodiments of this application;

[0054] Figure 12 Another structural schematic diagram of the computer device provided in the embodiments of this application. Detailed Implementation

[0055] This application provides a data processing method and computer device for accurately tracking I / O requests during container task startup and constructing a mapping relationship between each I / O request and the target data required for container startup. All m valid I / O requests generated during the container task startup process, along with their corresponding mapping relationships (i.e., second data), are sent to an image repository for storage. Since the valid I / O requests generated each time the same container task is started are the same, when other container nodes start the same container task, the m I / O requests can be aggregated and sent to the image repository all at once, reducing the amount of I / O requests sent to the image repository during container task startup. Simultaneously, during container task startup, the corresponding m target data can be pulled from the image repository at once based on the pre-constructed mapping relationship, eliminating the need for multiple pulls and improving the efficiency of container task startup. Furthermore, since the m target data are necessary for container startup, no redundant data is pulled during the data retrieval process, thus achieving a minimum amount of data pulled.

[0056] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0057] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0058] To better understand the solutions of the embodiments of this application, the relevant terms and concepts that may be involved in the embodiments of this application will be introduced below. It should be understood that the explanations of the relevant terms and concepts may be limited due to the specific circumstances of the embodiments of this application, but this does not mean that this application can only be limited to that specific circumstance. The specific circumstances of different embodiments may also differ, and no specific limitation is made here.

[0059] (1) Container image

[0060] A lightweight, portable software packaging method for packaging an application and all its runtime environment, system libraries, and dependencies into a single file. This file can be loaded and run by containerized platforms (such as Docker, Kubernetes, etc.), enabling rapid deployment and execution of the application.

[0061] (2) Container image data

[0062] Container image data, also known as image data, refers to a series of data used in containerization technology to define the container's runtime environment and application. One container image can correspond to one or more container tasks, the difference being that each container task uses different amounts and types of data from that container image. When a container node starts a container task, a unique task identifier is generated or determined for that task. Based on this task identifier, it is possible to directly identify which container image corresponds to that container task.

[0063] (3) OCI image format

[0064] A standard container image format that defines an image as containing a manifest file, an image index (optional), a set of file system layers, and a configuration file.

[0065] (4) I / O trajectory

[0066] This refers to the set of all I / O requests triggered during program execution. I / O requests include reading / writing files and the corresponding file offsets and lengths.

[0067] (5) Page caching

[0068] This refers to a segment of memory in the kernel, measured in physical pages, used to accelerate file access.

[0069] (6) Missing page

[0070] When the target page table entry is empty or marked as read-only, the memory subsystem triggers a page fault operation to map a block of physical memory address to the corresponding page table entry.

[0071] First, the system architecture and overall process applied in the methods of the embodiments of this application will be described. For details, please refer to [link / reference needed]. Figure 1 , Figure 1 This is a schematic diagram of the system architecture provided in this application embodiment. The system architecture of this application has been modified on both the container node 100 and the image repository 200 sides. Specifically, on the container node 100 side, an I / O trajectory module 110 is mainly introduced to track the I / O trajectory during the execution of container tasks and load the relevant data corresponding to the I / O trajectory (i.e., the m target data) into the page cache of the kernel file system of the target file. On the image repository 200 side, the container image data 210 embedding the I / O trajectory will embed the I / O trajectory of different container tasks, or embed the I / O trajectory of different container tasks and the corresponding m target data. Whether only the I / O trajectory is embedded, or whether the I / O trajectory and the corresponding m target data are embedded, depends on the specific application scenario, and this application does not limit this.

[0072] Based on this system architecture, the method of this application mainly includes two aspects, which are described below:

[0073] I. I / O Track Management

[0074] I / O trajectory management tracks I / O requests at the file system level. This approach shields the container's underlying storage driver, allowing for precise collection of I / O trajectories and corresponding target data for container tasks across any on-demand loading method. The I / O trajectory management described in this application operates at the container task level. Different container tasks using the same container image data will generate different I / O trajectories. These I / O trajectories are associated with the container tasks and embedded together into the container image, ensuring compatibility between the added I / O trajectories and the original container image. Similarly, in other application scenarios, these I / O trajectories can also be associated with container tasks and embedded along with m corresponding target data points into the container image, maintaining compatibility between the added I / O trajectories and the original container image.

[0075] II. Container Task Launch Based on I / O Trajectory

[0076] This application differs from existing on-demand loading of container image data in its container task startup process based on I / O trajectories. Traditional on-demand loading fetches files or data blocks as needed during startup, but its I / O behavior is inefficient because it triggers multiple fetches of container image data each time the data is not locally available. This means that every I / O request triggers a loading process, resulting in subsequent I / O requests for a single file / data block requiring interaction with the image repository, thus lengthening the program's execution time. In contrast, this application fetches the corresponding data (i.e., the aforementioned m target data) from the I / O trajectory all at once during startup. This is because the application orchestrates the I / O requests of the container task, achieving I / O aggregation. Therefore, during container task startup, the corresponding m target data can be fetched from the image repository at once through I / O aggregation, eliminating the need for multiple fetches by the container node and improving I / O efficiency during startup.

[0077] After the m target data corresponding to the I / O trajectory are pulled to the container node, since these target data also have page alignment characteristics and are the data that the container startup process actually needs to read, this invention loads the m target data corresponding to the I / O trajectory into the kernel page cache. I / O requests during the container task startup process can directly hit the kernel page cache, thereby realizing I / O pass-through between the page cache and the image repository and reducing the I / O access path (in the existing on-demand pull method, the container image data needs to go through multiple memory copies from the image repository to the local image cache and finally to the kernel page cache of the container node, which makes the read I / O stack during the container task startup process longer and brings certain data copy overhead).

[0078] It should be noted that, in the embodiments of this application, Figure 1 The system architecture shown is for illustrative purposes only, and there are no restrictions on the deployment methods of each unit / module. Furthermore, Figure 1 Each device can contain more modules / submodules to decouple functions; see details below. Figure 2 For example, the I / O trajectory module 101 in container node 100 can be decoupled into a tracking submodule 1001 and a loading submodule 1002. The tracking submodule 1001 tracks the I / O trajectory during container task execution, and the loading submodule 1002 loads the relevant data corresponding to the I / O trajectory (i.e., the m target data) into the page cache of the kernel file system of the target file. Similarly, Figure 1Each device may also contain fewer modules / sub-modules to achieve centralized data processing. This application does not limit the number and function of modules / sub-modules contained in each device lock, which can be determined by specific application requirements.

[0079] Based on the above system architecture, an embodiment method provided by this application will be described below. Please refer to the following for details. Figure 3 , Figure 3 This is a flowchart illustrating a data processing method provided in an embodiment of this application. This data processing method corresponds to the aforementioned I / O trajectory management and is applied to a first container node. Specifically, it may include the following steps:

[0080] 301. Determine the m I / O requests generated by the container task during startup, where m ≥ 1.

[0081] First, the first container node determines the m I / O requests generated during the startup process of the container task to be started (also known as the target container task), where m ≥ 1. The set of these I / O requests can be called the I / O trajectory.

[0082] It should be noted that, in the embodiments of this application, the m I / O requests refer to the m I / O requests related to the container task, that is, the m valid I / O requests.

[0083] As an example, the method to obtain the m I / O requests can be as follows: First, collect the I / O requests generated by the container task during startup in real time. Assume that a total of n I / O requests are collected during the entire startup process, where n ≥ m. Then, determine the m I / O requests related to the container task from these n collected I / O requests. This yields the I / O trajectory with the minimum amount of metadata. Next, the corresponding m target data can be aggregated using this I / O trajectory. These aggregated m target data are the necessary data for the successful startup of the container task, and do not contain any redundant data not required for the container task's startup.

[0084] It should also be noted that, in some embodiments of this application, the methods for determining m I / O requests from n I / O requests include, but are not limited to:

[0085] a. Merge multiple I / O requests from n I / O requests.

[0086] If the first target data corresponding to one of the n I / O requests (referred to as the first I / O request) is contained in the second target data corresponding to another I / O request (referred to as the second I / O request), that is, the first target data is a subset of the second target data, then the two I / O requests can be merged. The merging method can be to keep only the second I / O request and filter out the first I / O request.

[0087] Similarly, if there are multiple I / O requests that need to be merged among the n I / O requests, they can be merged in the manner described above, which will not be elaborated here.

[0088] b. Eliminate one or more I / O requests from among the n I / O requests.

[0089] If one or more of the n I / O requests do not belong to the current container task (i.e. invalid I / O requests), these one or more I / O requests can be directly removed.

[0090] It should also be noted that, in some embodiments of this application, the first container node can achieve real-time collection of I / O requests in the following way: First, the first container node pre-registers an I / O probe (also called an I / O probe point) to track I / O requests generated during startup in real time. Specifically, the first container node can further analyze the file offset and the length of the corresponding target data of the I / O request being tracked based on the registered I / O probe, and perform page alignment processing (to improve reading efficiency). At the same time, it can find out the internal file path of the container corresponding to the I / O request and append this data as an I / O trace to the file.

[0091] For example, in Figure 1 In the illustrated embodiment, the tracing submodule 1001 can be loaded into the kernel first. I / O probes can be registered within this submodule for real-time collection of I / O requests during subsequent container task tracing. This process can be referred to as the initialization process of the tracing submodule 1001. During the subsequent collection process, it is determined whether the collection termination condition has been met. Once the termination condition is met, the tracing will stop (e.g., by removing the I / O probes). The n I / O requests are then the I / O requests collected during this entire tracing period.

[0092] Specifically, the first container node starts the container task based on the container startup parameters input by the user. These startup parameters reveal which container image data is being used and what type of container task is being run. Before the container task starts, real-time I / O request collection is enabled, so that all I / O requests during the container task's startup process are recorded until the collection termination condition is met.

[0093] It should be noted that, in some embodiments of this application, the termination conditions for the above-mentioned data collection include, but are not limited to:

[0094] a. The container task has been started and executed successfully;

[0095] b. The container task execution startup time reaches the preset time;

[0096] c. The number of I / O requests per unit time during the startup process of the container task is lower than the preset threshold.

[0097] It should be noted that in some embodiments of this application, after the above initialization, the first container node can further clean up the cached data of files in the current kernel page cache under a specified directory (e.g., which can be specified by the user). This is to prevent inaccurate tracking and to narrow down the tracking scope. Furthermore, to accurately track kernel page fault behavior, the first container node can also disable the kernel fault around feature.

[0098] It should also be noted that, in this embodiment, when the first container node starts the container task according to the container startup parameters input by the user, it can further determine the task identifier corresponding to the container task based on the task content during container runtime, and obtain the summary file (i.e., manifest file) from the container image data corresponding to the container task from the image repository. This summary file can be called the first summary file or the first manifest file. The task identifier serves as a unique index in the I / O trajectory and is used to index the corresponding I / O trajectory and target data in the image when the container task is subsequently started again (e.g., started on a second container node).

[0099] It should be noted that, in some embodiments of this application, the implementation methods for determining the task identifier corresponding to the container task may include, but are not limited to:

[0100] a. Generate a task identifier based on the container task.

[0101] In this case, a corresponding task identifier can be directly generated based on the container task. For example, the task identifier can be a generated hash value.

[0102] b. Assign corresponding task identifiers to container tasks based on the pre-built mapping relationship.

[0103] In this scenario, a mapping relationship between each container task and each task identifier can be pre-built, with each container task corresponding to a unique task identifier. This mapping relationship can be placed on the first container node. When the first container node wants to start the container task (i.e. the target container task) mentioned above, it can directly assign the corresponding task identifier to the target container task to be started based on this mapping relationship.

[0104] 302. During startup, the first data corresponding to each of the m I / O requests is obtained from the image repository, resulting in a total of m first data. These m first data belong to the container image data corresponding to the container task, and the container image data is stored in the image repository.

[0105] During the startup process of this container task, the first container node can obtain the files or data blocks (referred to as first data) corresponding to each of the m I / O requests from the image repository. One I / O request corresponds to one piece of first data, so a total of m pieces of first data can be obtained. These m pieces of first data belong to the container image data corresponding to this container task, and the container image data is stored in the image repository.

[0106] 303. Determine the target data corresponding to each of the m I / O requests from the m first data, and obtain a total of m target data. These m target data are the necessary data included for the successful startup of the container task.

[0107] Next, the first container node determines the target data corresponding to each I / O request from the m pieces of initial data obtained. That is, one I / O request corresponds to one piece of target data, resulting in a total of m pieces of target data. These m pieces of target data are the necessary data included for the successful startup of the container task, and they do not contain any redundant data unrelated to the startup. Therefore, the m pieces of target data represent the minimum amount of data required for the successful startup of the container task.

[0108] 304. Determine the first mapping relationship between m I / O requests and m target data.

[0109] After obtaining m target data, the first container node further determines the mapping relationship between the m I / O requests and the m target data. This mapping relationship can be called the first mapping relationship.

[0110] 305. Send the second data to the mirror repository so that the mirror repository saves the second data, which includes m I / O requests and the first mapping relationship.

[0111] Finally, the first container node sends second data to the image repository so that the image repository can save the second data. This second data includes m I / O requests and a first mapping relationship. For example, after receiving the second data, the image repository can package and store it together with the existing container image data.

[0112] It should be noted that in some embodiments of this application, the second data may further include the aforementioned m target data. Thus, when a container node needs to obtain the m target data corresponding to its container task, the image repository can send it directly without requiring the image repository to decompress the original container image data based on the m I / O requests and the first mapping relationship, then read the decompressed files or data blocks, and finally extract the required m target data from the decompressed files or data blocks. Since the direct sending method eliminates the need for the image repository to parse this data from the original container image data, the container node can directly pull it, whereas the image repository's decompression → reading → extraction process is relatively more time-consuming.

[0113] It should also be noted that in some embodiments of this application, since the first container node can further determine the task identifier corresponding to the container task based on the task content of the container runtime, and obtain the first digest file (i.e., the first manifest file) in the container image data corresponding to the container task from the image repository, in this case, the first container node also needs to further modify the original first manifest file according to the task identifier and the m I / O requests to obtain the second digest file, i.e., the second manifest file, and include the second manifest file in the above-mentioned second data and send it to the image repository.

[0114] It should be noted that, in this embodiment of the application, how to modify the first manifest file to obtain the second manifest file is determined based on the specific content included in the second data, which will be explained below:

[0115] a) The second set of data does not include the m target data.

[0116] In this case, it is only necessary to record the hash value corresponding to the task identifier (which can be called the first hash value; when the task identifier is directly generated, the task identifier is this hash value) and the hash value corresponding to the m I / O requests (which can be called the second hash value) in the first manifest file. The first manifest file that additionally records these two hash values ​​is the second manifest file.

[0117] b. The second set of data includes m target data points.

[0118] In this case, in addition to recording the first hash value corresponding to the task identifier and the second hash value corresponding to the m I / O requests in the first manifest file, it is also necessary to record the hash value (which can be called the third hash value) corresponding to the m target data. The first manifest file that additionally records these three hash values ​​is the second manifest file mentioned above.

[0119] To better understand the difference between the first and second manifest files, the following example illustrates the distinction:

[0120] In the original container image data, the manifest file (i.e., the first manifest file) is assumed to have the following format:

[0121] {

[0122] Xxx

[0123] }

[0124] Therefore, in the method of this application embodiment, the format of the manifest file (i.e., the second manifest file) in the container image data is as follows:

[0125] {

[0126] Xxx

[0127] "${task_id}_meta":${sha256}

[0128] }

[0129] As mentioned above, an I / O trace record was added to the second manifest file. Here, `${task_id}` is the task identifier of the container task, and `${sha256}` is the hash value calculated from `m` I / O requests, used as the filename for storing the `m` I / O request files. This additional file, `${sha256}`, records the information for the `m` I / O requests. Since the image repository can store a file according to certain rules, such as naming it `${sha256}`, it can automatically determine the location of this file on the server host.

[0130] If the second data also includes m target data, then the format of the second manifest file can be:

[0131] {

[0132] Xxx

[0133] ${task_id}_meta:${sha256},

[0134] ${task_id}_data:${sha256'}

[0135] }

[0136] In this case, two additional files are added, each using the hash value of its own content as its filename: one to record I / O requests and the other to record the target data.

[0137] It should be noted that the formats of the first and second manifest files mentioned above are for illustrative purposes only, and this application does not impose any restrictions on the format of the manifest files.

[0138] Based on the above system architecture, another embodiment of the method provided in this application will be described below. Please refer to the following for details. Figure 4 , Figure 4 This is a flowchart illustrating a method for starting a container task provided in an embodiment of this application. This method corresponds to the aforementioned container task starting method based on I / O trajectories. The method is applied to a second container node and may specifically include the following steps:

[0139] 401. Determine the target request based on the container task to be started, and send the target request to the image repository. The target request is used to instruct the image repository to send m I / O requests corresponding to the container task and m target data corresponding to the m I / O requests to the second container node. The m I / O requests are generated by the first container node during the start-up of the container task. The m target data are obtained based on the first mapping relationship. The first mapping relationship is the mapping relationship between the m I / O requests and the m target data. The m target data are the necessary data included for the successful start of the container task. One I / O request corresponds to one target data, and m≥1.

[0140] First, the second container node determines a request to be sent to the image repository based on the currently pending container task (i.e., the target container task). This request can be called a target request. After determining the target request, it sends it to the image repository. The target request instructs the image repository to send m I / O requests corresponding to the currently pending container task and m target data corresponding to these m I / O requests to the second container node. It is important to note that in this embodiment, these m I / O requests are generated by the first container node during the startup of the container task. These m target data are obtained based on a first mapping relationship, which is the mapping relationship between these m I / O requests and these m target data. Furthermore, these m target data are the necessary data included for the successful startup of the container task; one I / O request corresponds to one target data, and m ≥ 1.

[0141] It should be noted that, in some embodiments of this application, one way to determine the target request based on the currently pending container task is as follows: First, the second container node determines the task identifier corresponding to the currently pending container task. Then, based on the task identifier, it retrieves the manifest file of the container image data corresponding to the container task from the image repository, and determines the target request based on the second manifest file. The second manifest file is obtained by the first container node based on the task identifier, m I / O requests, and the original first manifest file (the specific method of obtaining this manifest file can be found in the above embodiments, and will not be repeated here). After modifying the manifest file, the first container node sends it to the image repository for storage.

[0142] It should be noted that in some embodiments of this application, when the second manifest file only includes the first hash value and the second hash value, the target request is determined based on the first hash value and the second hash value in the second manifest file; when the second manifest file includes a third hash value in addition to the first and second hash values, the target request is determined based on the first hash value, the second hash value, and the third hash value in the second manifest file. Here, the first hash value is the hash value corresponding to the task identifier (e.g., the task identifier itself can be a hash value), the second hash value is the hash value corresponding to m I / O requests, and the third hash value is the hash value corresponding to m target data.

[0143] Specifically, to facilitate understanding of the above process, the following example illustrates the process of starting the container task: The second container node first obtains the manifest file from the image repository based on the image specified in the container task to be started, which is the manifest file modified by the first container node in the above embodiment (i.e., the second manifest file). Then, it parses the second manifest file. Since the first container node previously wrote the hash value of the task identifier into the second manifest file, after parsing the second manifest file, the second container node can obtain the index value of the corresponding I / O request based on the container task identifier in it, and use the index value to construct a file pull request (i.e., the target request mentioned above). Then, it can pull m target data corresponding to the I / O request from the image repository based on the target request.

[0144] It's important to note that in this embodiment, the container task to be started on the second container node is the same as the target container task started on the first container node. However, in some practical applications, the container task to be started on the second container node can be a similar task to the target container task started on the first container node, since similar container tasks typically share most of their I / O requests. For the differing I / O requests, the image data can be pulled using the original pull method. Overall, this still improves pull efficiency and reduces the amount of data pulled.

[0145] 402. Store the m I / O requests and m target data sent by the image repository in the kernel page cache.

[0146] Then, the second container node stores the m I / O requests and m target data sent by the image repository in its kernel page cache.

[0147] 403. Start the container task and collect the target I / O requests generated during the startup process.

[0148] The second container node starts the container task and collects the I / O requests generated during the startup process in real time. These I / O requests are called target I / O requests.

[0149] 404. If the target I / O request is contained in m I / O requests, retrieve the target data corresponding to the target I / O request from the kernel buffer.

[0150] Finally, the second container node will further determine whether the target I / O request collected in real time is included in the above m I / O requests. If so, it will retrieve the target data corresponding to the target I / O request from its own kernel buffer.

[0151] It should be noted that in some other embodiments of this application, if the second container node determines that the target I / O request collected in real time is not included in the above m I / O requests, it needs to pull the file or data block corresponding to the target I / O request from the image repository according to the original pull process. This corresponding file or data block can be referred to as the target first data. The target first data is included in the container image data, and the container image data is stored in the image repository. Finally, the target data corresponding to the target I / O request is determined from the target first data in a manner similar to step 303 above.

[0152] It should also be noted that, in the embodiments of this application, Figure 3 Corresponding data processing methods and Figure 4The corresponding container task startup method is generally performed on different container nodes; that is, the first container node and the second container node are usually different container nodes. However, in some practical applications, the first container node and the second container node can also be the same container node, and this application does not limit this.

[0153] Furthermore, to facilitate understanding of the data retrieval process of the second container node described above, a specific example is provided below. Please refer to [link / reference needed] for details. Figure 5 , Figure 5 This is a schematic flowchart illustrating the data retrieval process provided in an embodiment of this application. This retrieval process is based on the above... Figure 1 or Figure 2 Once the corresponding system architecture is completed, the retrieval process can specifically include the following steps:

[0154] 501. Initialize the I / O trajectory module.

[0155] First, this step will... Figure 1 The I / O trajectory module 110 described above is initialized (if...). Figure 2 That is to Figure 2 The loading submodule 1002 described in the document is initialized. Specifically, the I / O trajectory module 110 (or loading submodule 1002) will be loaded into the kernel. This module can provide some capabilities required for kernel buffer pass-through.

[0156] 502. Start the container task and generate a unique task identifier based on the container task.

[0157] In this step, the second container node will start the container task and generate a task identifier to uniquely identify the container task. This process is similar to that described above. Figure 3 The method of identifying body shape tasks in the corresponding embodiments is similar and will not be described in detail here.

[0158] 503. Obtain the manifest and config files from the mirror repository and parse them.

[0159] Afterwards, the second container node will retrieve the manifest and config files from the image repository based on the generated task identifier and parse them. The manifest file is the second manifest file obtained by modification as described above.

[0160] 504. Based on the parsing results, determine whether there is an I / O trajectory corresponding to the task identifier. If yes, proceed to step 505; otherwise, proceed to step 509.

[0161] After parsing the above file, it can be determined whether there is a corresponding I / O trace based on the parsing result. If yes, proceed to step 505; otherwise, proceed to step 509.

[0162] 505. Retrieve the corresponding I / O trace from the image repository based on the task identifier and save it to the kernel cache.

[0163] The second container node will pull the corresponding I / O trace from the image repository based on the generated task identifier and save it to the kernel cache.

[0164] 506. The container task begins creation and execution.

[0165] 507. Determine whether the I / O request is satisfied in the I / O trajectory. If yes, proceed to step 508; otherwise, proceed to step 510.

[0166] The second container node determines whether the currently collected I / O request (i.e. the target I / O request) is satisfied in the I / O trajectory. If yes, it executes step 508; otherwise, it executes step 510.

[0167] 508. Return the corresponding target data from the kernel cache.

[0168] The second container node returns the target data corresponding to the real-time collected target I / O request from its own kernel cache.

[0169] 509. The container task begins creation and execution.

[0170] 510. Load target data from the mirror repository.

[0171] The second container node loads the target data corresponding to the target I / O request from the image repository.

[0172] 511. Container task completed.

[0173] The container task is executed based on the target data returned from its own kernel cache, or the target data loaded from the image repository, until the container task starts successfully.

[0174] To facilitate understanding of the I / O trajectory management and I / O trajectory-based container task startup process described above, a specific example will be provided below. This example adapts the container image generation process to the open-source container management tool nerdctl, while the container task startup process incorporates I / O trajectory-based container startup functionality into the open-source nydus solution.

[0175] I. Container Image Building Phase

[0176] This stage corresponds to the I / O trajectory management process described above; please refer to [link / reference needed] for details. Figure 6 , Figure 6 This application provides a component structure diagram. In the container image building phase, the container client uses nerdctl. The trace option is added to the execution flow where nerdctl sends the image conversion command (convert) to the container service process. An I / O trajectory tracking module called rio_tracker (i.e., the one mentioned above) is introduced. Figure 1 Corresponding to the I / O trajectory module 110 described in the embodiment, or, Figure 2 Corresponding to the tracking submodule 1001 described in the embodiment, the rio_tracker module uses EBPF technology to register I / O probes. In this example, only read I / O is tracked; therefore, the kernel functions that need to be stubbed include, but are not limited to, vfs_read, kernel_read, and filemap_fault. Referring to the container I / O trajectory management process of this application, the main process of building a container image may include:

[0177] a. Load the rio_tracker module into the kernel.

[0178] b. Clear the cache in the specified directory and turn off the fault_around switch.

[0179] c. Send a command to the rio_tracker module to enable the IO trajectory tracking function.

[0180] d. Execute the image conversion command in nerdctl, specifying the target container task, to convert the traditional image. This process will run the target container task using the traditional image.

[0181] e. If a termination condition is specified in d above (e.g., a preset duration or a specified time), then the container task will be stopped after the termination condition is met, and a command to disable I / O track tracking will be sent to the rio_tracker module. Otherwise, wait for the container task started in d to finish running, or wait for the frequency of kernel-level I / O reads to fall below x times / minute (e.g., x = 5), where the threshold can be specified by the user, and then disable the I / O track tracking function.

[0182] f. Step e above records I / O requests from containers started based on traditional images to a file. These I / O requests may contain duplicates, overlaps, or invalid requests. This step restarts the container task and obtains the rootfs path of the new container. Through this path, file paths within the container can be accessed. Invalid I / O requests are filtered out by comparing the paths within the container with the files detected in the I / O records. For duplicates and overlaps, merging and deduplication are performed directly, ultimately resulting in the most concise I / O trajectory metadata. In this example, the format of the I / O trajectory metadata can be...<fullpath,offset,len> A series of sets, with offset and len both page-aligned, are used to represent a specific segment of a file read during an I / O operation. This metadata is then used to read data from the target file in the container task, ultimately merging the data corresponding to these I / O requests (i.e., the target data) into a large file, forming the I / O trace, before the target container is stopped.

[0183] g. Obtain the content of the target container task specified in d above and calculate its hash value; also calculate the hash value of the I / O trace generated in f above, form key / value pairs with this information and write them into the manifest file of this image, and repackage the I / O trace into the container image.

[0184] h. Push the container image with embedded I / O traces to the image repository.

[0185] It's important to note that in this example, the I / O trajectory includes not only I / O requests but also the corresponding target data. And in the aforementioned... Figure 3-5 In the corresponding embodiment, the I / O trajectory only contains I / O requests.

[0186] II. Container Task Startup Phase

[0187] This stage corresponds to the container task startup process based on I / O trajectories described above. Please refer to [link / reference] for details. Figure 7 , Figure 7 This is another component structure diagram provided in this application embodiment. This example is implemented based on the existing Nydus solution, and an I / O trajectory startup path is added to the Nydus solution. An I / O trajectory loading module called io_trace (i.e., the one mentioned above) is introduced. Figure 1 Corresponding to the I / O trajectory module 110 described in the embodiment, or, Figure 2 (Corresponding to the loading submodule 1002 in the embodiment). Referring to the container task startup process based on I / O trajectory described above in this application, the main process of container task startup in this example may include:

[0188] a. Load the io_trace module into the kernel.

[0189] b. Execute the container task start command in nerdctl (i.e., the container client) and specify the container image data corresponding to the I / O trajectory.

[0190] c. The container service process downloads the manifest file from the image repository based on the image address, parses the I / O request information in the file, calculates the hash value using the task specified in b above, and then retrieves the corresponding hash value for the I / O request from the manifest file. If the corresponding I / O request information is not found, it reverts to the original container task startup process and downloads the config file from the image metadata. Otherwise, the I / O trace is added to the concurrent download process, and the downloaded I / O trace is stored in the kernel cache. It's important to note that in this example, the I / O trace includes not only the I / O request but also the corresponding target data. And in the above... Figure 3-5 In the corresponding embodiment, the I / O trajectory only contains I / O requests.

[0191] d. After the container service process finishes downloading data, it creates a container task. Once the container task is created, it sends a command to the io_trace module to trigger trace metadata parsing. Trace metadata refers to...<fullpath,offset,len> The collection of data is used to build a hash table based on the full path. By using the full path, all I / O requests corresponding to that full path can be retrieved. Once the trace metadata index is built, the container task enters the running phase.

[0192] e. During the runtime phase, the image file will be opened, which will trigger an open operation. The open operation process is as follows: Figure 8 As shown: First, obtain the full path of the open file, then search for the corresponding set of I / O requests in the trace metadata using the full path. If it exists, record the set of I / O requests in the private structure of the inode.

[0193] f. During the runtime phase, the image file is read, which triggers the file read and mmap operations. The execution flow is as follows: Figure 9As shown: For a read operation, a range match is performed on the I / O requests obtained during the open phase based on its offset and length. If the range is covered by the trace, the data in the trace kernel cache is directly copied to user space, and then the data reading is completed. For an mmap operation, the data reading process enters from the page fault, i.e., the fault process. At this time, the file offset is calculated from the kernel file structure based on the virtual address addr of the page fault location, and then a range match is performed on the I / O requests according to the page alignment length. If the range is covered by the trace, the page in the trace kernel cache is directly mapped to the page fault virtual address, and then the mmap page fault process is completed. If the I / O requests do not match, the original process is executed.

[0194] g. After the target data is read, proceed with the container startup process until the container starts successfully.

[0195] Based on the above embodiments, in order to better implement the above solutions of this application, related equipment for implementing the above solutions is also provided below. See details. Figure 10 , Figure 10 This is a schematic diagram of a computer device provided in an embodiment of this application. The computer device 1000 serves as a first container node and may specifically include: a first determining module 1001, a pulling module 1002, a second determining module 1003, a third determining module 1004, and a sending module 1005. The first determining module 1001 is used to determine m I / O requests generated during the startup process of the container task, where m ≥ 1. The pulling module 1002 is used to retrieve the first data corresponding to each of the m I / O requests from the image repository during the startup process, obtaining a total of m first data items. These m first data items belong to the container task... The first module 1003 is used to determine the target data corresponding to each of the m I / O requests from the m first data, resulting in a total of m target data, which are the necessary data for the successful startup of the container task; the second module 1004 is used to determine the first mapping relationship between the m I / O requests and the m target data; and the third module 1005 is used to send second data to the image repository so that the image repository can save the second data, which includes the m I / O requests and the first mapping relationship.

[0196] In one possible design, the second data also includes: the m target data.

[0197] In one possible design, the first determining module 1001 is further configured to: determine a task identifier corresponding to the container task based on the container task before determining the m I / O requests generated during the startup process of the container task, and obtain a first digest file from the container image data from the image repository; the third determining module 1004 is further configured to: obtain a second digest file based on the task identifier, the m I / O requests and the first digest file before the sending module 1005 sends the second data to the image repository, and the second digest file is included in the second data.

[0198] In one possible design, the third determining module 1004 is specifically used to: when the second data does not include the m target data, record the first hash value corresponding to the task identifier and the second hash value corresponding to the m I / O requests in the first digest file to obtain the second digest file.

[0199] In one possible design, the third determining module 1004 is specifically used to: when the second data includes the m target data, record the first hash value corresponding to the task identifier, the second hash value corresponding to the m I / O requests, and the third hash value corresponding to the m target data in the first digest file to obtain the second digest file.

[0200] In one possible design, the first determining module 1001 is specifically used to: generate a task identifier corresponding to the container task based on the container task; or, assign a corresponding task identifier to the container task based on a pre-built second mapping relationship between each container task and each task identifier.

[0201] In one possible design, the first determining module 1001 is specifically used to: collect n I / O requests generated during the startup process of the container task, where n ≥ m; and determine m I / O requests from these n I / O requests.

[0202] In one possible design, the first determining module 1001 is specifically used to: remove the first I / O request from the n I / O requests to obtain m I / O requests; wherein the first target data corresponding to the first I / O request is included in the second target data corresponding to the second I / O request, the second I / O request is included in the n I / O requests, or the first I / O request does not belong to the I / O requests of the container task.

[0203] In one possible design, the first determining module 1001 is specifically used to: track the I / O requests generated during the startup process using an I / O probe, which is pre-registered on the first container node; and stop the tracking when a termination condition is met, wherein the n I / O requests are the I / O requests collected during the tracking period.

[0204] In one possible design, the termination condition includes: the container task has completed its startup execution; or, the startup duration of the container task has reached a preset duration; or, the I / O request volume of the container task per unit time during the startup process is lower than a preset threshold.

[0205] It should be noted that the information interaction and execution process between the various modules / units in the computer device 1000 are the same as those described in this application. Figure 3 The corresponding embodiments are based on the same concept, and the specific details can be found in the description of the method embodiments shown above in this application, which will not be repeated here.

[0206] This application also provides a computer device; please refer to [link / reference]. Figure 11 , Figure 11 This is another schematic diagram of the computer device provided in an embodiment of this application. The computer device 1100 serves as a second container node and specifically includes: a determining module 1101, a saving module 1102, a collecting module 1103, and a retrieving module 1104. The determining module 1101 is used to determine a target request based on the currently scheduled container task to be started, and send the target request to the image repository. The target request instructs the image repository to send m I / O requests corresponding to the container task and m target data corresponding to the m I / O requests to the second container node. The m I / O requests are generated by the first container node during the startup of the container task, and the m target data are based on... The first mapping relationship is obtained from the first mapping relationship, which is the mapping relationship between the m I / O requests and the m target data. The m target data are the necessary data included in the successful startup of the container task. One I / O request corresponds to one target data, and m≥1. The storage module 1102 is used to store the m I / O requests and the m target data sent by the image repository in the kernel page cache. The acquisition module 1103 is used to start the container task and acquire the target I / O requests generated during the startup process. The retrieval module 1104 is used to retrieve the target data corresponding to the target I / O request from the kernel buffer when the target I / O request is included in the m I / O requests.

[0207] In one possible design, the determining module 1101 is specifically used for: determining a task identifier corresponding to the container task based on the currently scheduled container task to be started; obtaining a second digest file of the container image data corresponding to the container task from the image repository based on the task identifier, the second digest file being obtained by the first container node based on the task identifier, the m I / O requests and the first digest file, the second digest file being sent to the image repository by the first container node; and determining the target request based on the second digest file.

[0208] In one possible design, the determining module 1101 is specifically used to: determine the target request based on the first hash value and the second hash value in the second digest file; or, determine the target request based on the first hash value, the second hash value, and the third hash value in the second digest file; wherein the first hash value is the hash value corresponding to the task identifier, the second hash value is the hash value corresponding to the m I / O requests, and the third hash value is the hash value corresponding to the m target data.

[0209] In one possible design, the computer device 1100 further includes a pull module 1105, which is used to retrieve target first data corresponding to the target I / O request from the image repository when the target I / O request is not included in the m I / O requests. The target first data is included in the container image data and the container image data is stored in the image repository. The target data corresponding to the target I / O request is determined from the target first data.

[0210] It should be noted that the information interaction and execution process between the various modules / units in the computer device 1100 are the same as those described above in this application. Figure 4 and Figure 5 The corresponding embodiments are based on the same concept, and the specific details can be found in the description of the method embodiments shown above in this application, which will not be repeated here.

[0211] Next, we will introduce another computer device provided in the embodiments of this application. Please refer to [link / reference]. Figure 12 , Figure 12 This is a schematic diagram of a computer device provided in an embodiment of this application. The computer device 1200 may be equipped with... Figure 10 The computer device 1000 described in the corresponding embodiment is used to implement Figure 10 Corresponding to the functions of each module in the embodiment, the computer device 1200 can also be deployed with Figure 11 The computer device 1100 described in the corresponding embodiment is used to implement Figure 10The functions of each module in the corresponding embodiment are described. Specifically, the computer device 1200 is implemented by one or more servers. The computer device 1200 can vary significantly due to different configurations or performance, and may include one or more central processing units (CPUs) 1222 and memory 1232, and one or more storage media 1230 (e.g., one or more mass storage devices) for storing application programs 1242 or data 1244. The memory 1232 and storage media 1230 can be temporary or persistent storage. The program stored in the storage media 1230 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the computer device 1200. Furthermore, the CPU 1222 may be configured to communicate with the storage media 1230 and execute the series of instruction operations in the storage media 1230 on the computer device 1200.

[0212] Computer device 1200 may also include one or more power supplies 1226, one or more wired or wireless network interfaces 1250, one or more input / output interfaces 1258, and / or one or more operating systems 1241, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0213] In this embodiment of the application, the central processing unit 1222 is used to execute... Figures 3 to 5 ,or Figures 8 to 9 The steps executed by the container node in the corresponding embodiment are as follows. For example, the central processing unit 1222 can be used to: First, determine m I / O requests generated during the startup process of the container task, where m ≥ 1; then, during the startup process of the container task, obtain the first data corresponding to each of the m I / O requests from the image repository, resulting in a total of m first data, which belong to the container image data corresponding to the container task, and the container image data is stored in the image repository; then, determine the target data corresponding to each of the m I / O requests from the m first data, resulting in a total of m target data, and determine the first mapping relationship between the m I / O requests and the m target data, wherein the m target data are the necessary data included for the successful startup of the container task; finally, send second data to the image repository so that the image repository saves the second data, which includes the m I / O requests and the first mapping relationship.

[0214] It should be noted that the specific manner in which the central processing unit 1222 executes the above steps is different from that in this application. Figures 3 to 5 ,or Figures 8 to 9The corresponding method embodiments are based on the same concept, and the technical effects they bring are the same as those in the above embodiments of this application. For details, please refer to the description in the method embodiments shown above in this application, which will not be repeated here.

[0215] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0216] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0217] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0218] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A method of data processing, characterized by, The method is applied to a first container node, and comprises the following steps: determining m I / O requests generated by a container task in a starting process, m≥1; obtaining, in the starting process, first data corresponding to the m I / O requests respectively from an image warehouse, so as to obtain m first data, wherein the m first data belong to container image data corresponding to the container task, and the container image data is stored in the image warehouse; determining target data corresponding to the m I / O requests respectively from the m first data, so as to obtain m target data, wherein the m target data are necessary data included in the container task starting successfully; determining a first mapping relationship between the m I / O requests and the m target data; sending second data to the image warehouse, so that the image warehouse saves the second data, wherein the second data comprises the m I / O requests and the first mapping relationship.

2. The method of claim 1, wherein, The second data further comprises: the m target data.

3. The method according to any one of claims 1-2, characterized in that, Before the step of determining the m I / O requests generated by the container task in the starting process, the method further comprises the following steps: determining a task identifier corresponding to the container task according to the container task, and obtaining a first digest file in the container image data from the image warehouse; before the step of sending the second data to the image warehouse, the method further comprises the following steps: obtaining a second digest file according to the task identifier, the m I / O requests and the first digest file, wherein the second digest file is contained in the second data.

4. The method of claim 3, wherein, The step of obtaining the second digest file according to the task identifier, the m I / O requests and the first digest file comprises the following steps: in the case that the second data does not comprise the m target data, recording a first hash value corresponding to the task identifier and a second hash value corresponding to the m I / O requests in the first digest file, so as to obtain the second digest file.

5. The method of claim 3, wherein, The step of obtaining the second digest file according to the task identifier, the m I / O requests and the first digest file comprises the following steps: in the case that the second data comprises the m target data, recording a first hash value corresponding to the task identifier, a second hash value corresponding to the m I / O requests and a third hash value corresponding to the m target data in the first digest file, so as to obtain the second digest file.

6. The method according to any one of claims 3-5, characterized in that, The step of determining the task identifier corresponding to the container task according to the container task comprises the following steps: generating the task identifier corresponding to the container task according to the container task; or, allocating the corresponding task identifier to the container task according to a second mapping relationship between each container task and each task identifier constructed in advance.

7. The method according to any one of claims 1 to 6, characterized in that, The step of determining the m I / O requests generated by the container task in the starting process comprises the following steps: collecting n I / O requests generated by the container task in the starting process, n≥m; determining the m I / O requests from the n I / O requests.

8. The method of claim 7, wherein, The step of determining the m I / O requests from the n I / O requests comprises the following steps: eliminating a first I / O request in the n I / O requests, so as to obtain the m I / O requests; The first target data corresponding to the first I / O request is contained in second target data corresponding to a second I / O request, the second I / O request is contained in the n I / O requests, or the first I / O request is not an I / O request of the container task.

9. The method according to any one of claims 6-7, characterized by, The n I / O requests generated in the starting process of the collection container task include: Tracking the I / O requests generated in the starting process by an I / O probe, the I / O probe being pre-registered in the first container node; In the case where a termination condition is reached, stopping the tracking, the n I / O requests being I / O requests collected during the tracking.

10. The method of claim 9, wherein, The termination condition includes: The starting execution of the container task is completed; Or, The duration of the starting execution of the container task reaches a preset duration; Or, The I / O request amount of the container task in a unit duration in the starting process is lower than a preset threshold.

11. A method of container task initiation, characterized by, Applied to a second container node, including: Determining a target request according to a container task to be started currently, and sending the target request to a mirror repository, the target request being used to instruct the mirror repository to send m I / O requests corresponding to the container task and m target data corresponding to the m I / O requests to the second container node, the m I / O requests being generated by a first container node in a process of starting the container task, the m target data being obtained based on a first mapping relationship, the first mapping relationship being a mapping relationship between the m I / O requests and the m target data, the m target data being necessary data included in a successful starting of the container task, one I / O request corresponding to one target data, and m≥1; Saving the m I / O requests and the m target data sent by the mirror repository in a kernel page cache; Starting the container task, and collecting target I / O requests generated in a starting process; In the case where the target I / O request is contained in the m I / O requests, calling target data corresponding to the target I / O request from the kernel buffer.

12. The method of claim 11, wherein, The determining of the target request according to the container task to be started currently includes: Determining a task identifier corresponding to the container task according to the container task to be started currently; Obtaining a second digest file of container mirror data corresponding to the container task from the mirror repository according to the task identifier, the second digest file being obtained by the first container node based on the task identifier, the m I / O requests, and a first digest file, the second digest file being sent to the mirror repository by the first container node; Determining a target request according to the second digest file.

13. The method of claim 12, wherein, The determining of the target request according to the second digest file includes: Determining a target request according to a first hash value and a second hash value in the second digest file; Or, Determining the target request according to the first hash value, the second hash value, and a third hash value in the second digest file. The first hash value is a hash value corresponding to the task identifier, the second hash value is a hash value corresponding to the m I / O requests, and the third hash value is a hash value corresponding to the m target data.

14. The method according to any one of claims 11-13, characterized in that, The method further includes: In a case where the target I / O request is not included in the m I / O requests, obtaining target first data corresponding to the target I / O request from the mirror repository, the target first data being included in the container image data, the container image data being stored in the mirror repository; Determining target data corresponding to the target I / O request from the target first data.

15. A computer device, comprising: The computer device, as a first container node, includes: A first determining module configured to determine m I / O requests generated by a container task in a starting process, m being greater than or equal to 1; A pulling module configured to obtain, in the starting process, m first data corresponding to the m I / O requests respectively from a mirror repository, the m first data being included in container image data corresponding to the container task, the container image data being stored in the mirror repository; A second determining module configured to determine m target data corresponding to the m I / O requests respectively from the m first data, the m target data being necessary data included in a successful start of the container task; A third determining module configured to determine a first mapping relationship between the m I / O requests and the m target data; A sending module configured to send second data to the mirror repository, so that the mirror repository stores the second data, the second data including the m I / O requests and the first mapping relationship.

16. The apparatus of claim 15, wherein, The second data further includes: The m target data.

17. The apparatus of any one of claims 15-16, wherein, The first determining module is further configured to: Before determining the m I / O requests generated by the container task in the starting process, determine a task identifier corresponding to the container task according to the container task, and obtain a first summary file in the container image data from the mirror repository; The third determining module is further configured to, before the sending module sends the second data to the mirror repository, obtain a second summary file according to the task identifier, the m I / O requests, and the first summary file, the second summary file being included in the second data.

18. The apparatus of claim 17, wherein, The third determining module is specifically configured to: In a case where the second data does not include the m target data, record a first hash value corresponding to the task identifier and a second hash value corresponding to the m I / O requests in the first summary file, to obtain a second summary file.

19. The apparatus of claim 17, wherein, The third determining module is specifically configured to: In a case where the second data includes the m target data, record a first hash value corresponding to the task identifier, a second hash value corresponding to the m I / O requests, and a third hash value corresponding to the m target data in the first summary file, to obtain a second summary file.

20. The apparatus of any one of claims 17-19, wherein, The first determining module is specifically configured to: Generate a task identifier corresponding to the container task according to the container task; Or, According to a second mapping relationship between the container task and the task identifier, the container task is assigned a corresponding task identifier.

21. The apparatus of any one of claims 15-20, wherein, The first determining module is specifically configured to: Collect n I / O requests generated in a starting process of the container task, n≥m; Determine m I / O requests from the n I / O requests.

22. The apparatus of claim 21, wherein, The first determining module is specifically configured to: Eliminate a first I / O request in the n I / O requests to obtain m I / O requests; The first I / O request corresponds to first target data, the second I / O request corresponds to second target data, and the second target data contains the first target data, or the first I / O request does not belong to the I / O requests of the container task.

23. The apparatus of any one of claims 20-21, wherein, The first determining module is specifically configured to: Track the I / O requests generated in the starting process through an I / O probe, and the I / O probe is pre-registered in the first container node; In the case where a termination condition is reached, the tracking is stopped, and the n I / O requests are I / O requests collected during tracking.

24. The apparatus of claim 23, wherein, The termination condition includes: The starting execution of the container task is completed; Or, The time length of the starting execution of the container task reaches a preset time length; Or, The I / O request amount of the container task in a unit time length in the starting process is lower than a preset threshold.

25. A computer device, comprising: The computer device as a second container node includes: A determining module configured to determine a target request according to a container task to be started currently, and send the target request to a mirror warehouse, the target request being used to instruct the mirror warehouse to send m I / O requests corresponding to the container task and m target data corresponding to the m I / O requests to the second container node, the m I / O requests being generated by a first container node in a process of starting the container task, the m target data being obtained based on a first mapping relationship, the first mapping relationship being a mapping relationship between the m I / O requests and the m target data, the m target data being necessary data included in a successful starting of the container task, one I / O request corresponding to one target data, and m≥1; A saving module configured to save the m I / O requests and the m target data sent by the mirror warehouse in a kernel page cache; A collecting module configured to start the container task and collect target I / O requests generated in a starting process; A calling module configured to call target data corresponding to the target I / O request from the kernel buffer in a case where the target I / O request is included in the m I / O requests.

26. The apparatus of claim 25, wherein, The determining module is specifically configured to: Determine a task identifier corresponding to the container task according to the container task to be started currently; Obtain a second digest file of container mirror data corresponding to the container task from the mirror warehouse according to the task identifier, the second digest file being obtained by the first container node based on the task identifier, the m I / O requests, and a first digest file, and the second digest file being sent to the mirror warehouse by the first container node. According to the second summary file, determine a target request.

27. The apparatus of claim 26, wherein, The determining module is specifically configured to: According to the first hash value and the second hash value in the second summary file, determine the target request; Or, According to the first hash value, the second hash value and the third hash value in the second summary file, determine the target request; The first hash value is a hash value corresponding to the task identifier, the second hash value is a hash value corresponding to the m I / O requests, and the third hash value is a hash value corresponding to the m target data.

28. The apparatus of any one of claims 25-27, wherein, The device further comprises a pulling module. In the case that the target I / O request is not included in the m I / O requests, the pulling module is configured to obtain target first data corresponding to the target I / O request from the mirror repository, the target first data being included in the container image data, the container image data being stored in the mirror repository. From the target first data, determine target data corresponding to the target I / O request.

29. A computer device, comprising a processor and a memory, the processor being coupled with the memory, characterized in that: The memory is configured to store a program. The processor is configured to execute the program in the memory, so that the computer device executes the method according to any one of claims 1-14.

30. A computer storage medium, comprising, Computer readable instructions are stored, and the computer readable instructions implement the method according to any one of claims 1-14 when executed by a processor.

31. A computer program product, characterised in that, The computer program product comprises computer readable instructions, which implement the method according to any one of claims 1-14 when executed by a processor.

32. A chip comprising a processor and a data interface, characterized in that The processor reads the instructions stored on the memory through the data interface, and executes the method according to any one of claims 1-14.

Images