Data and container file reading method, device, system, medium and program product

Abstract

The embodiment of the present application provides a method, device, system, medium and program product for reading data and container files. In the embodiment of the present application, data with the same content has the same data fingerprint, and multiple index nodes corresponding to multiple identifiers with the same data fingerprint are mapped to the same virtual index node, so that multiple index nodes can share the target memory space corresponding to the virtual index node, that is, multiple index nodes can share the data stored in the target memory space of the virtual index node, without caching a copy of the target data in each index node, thereby reducing the occupation of memory resources.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, system, medium and program product for reading data and container files. Background Art

[0002] A read-only file system (ROFS) is a file system that does not allow write operations on files. This means that in a read-only file system, files can only be read, and cannot be modified, deleted, or new files created. Some read-only file systems, such as the Enhanced Read-Only File System (EROFS), will generate multiple caches in memory when reading files with the same content but different identifiers. For example, in a container scenario, assuming that the image files of two containers have the same shared library file, since different containers have different mount points, the same shared library file has different file paths for the two containers. In this way, when the read-only file system reads the shared library file in the image files of the two containers, it will generate two caches in memory for the same shared library file, which undoubtedly causes a waste of memory resources. Summary of the invention

[0003] Multiple aspects of the present application provide a method, device, system, medium and program product for reading data and container files, so as to reduce memory usage when reading data.

[0004] The present application provides a data reading method, including:

[0005] In the case where the read-only file system performs a read operation on the target data of the multiple identifiers, obtaining metadata of the target data of the multiple identifiers; the metadata of the target data includes a data fingerprint of the target data;

[0006] Acquire a plurality of index nodes according to metadata of target data of the plurality of identifiers; the plurality of identifiers correspond one to one to the plurality of index nodes;

[0007] According to the data fingerprint of the target data, mapping the multiple index nodes to the same target virtual index node, so that the multiple index nodes share the target memory space of the target virtual index node;

[0008] The target data is read from the target memory space.

[0009] The embodiment of the present application also provides a container file reading method, which is applicable to a service node, wherein the service node is deployed with multiple containers, the image files of the multiple containers have target files with the same content, and the file paths of the target files of the multiple containers are different; the method comprises:

[0010] In a case where the read-only file system performs a read operation on the target files of the multiple containers, obtaining metadata of the target files of the multiple containers; the target files of the multiple containers include data fingerprints of contents of the target files;

[0011] According to the metadata of the target files of the multiple containers, a plurality of index nodes are obtained; each file path corresponds to an index node;

[0012] According to the data fingerprint of the content of the target file, the multiple index nodes are mapped to the same target virtual index node, so that the multiple index nodes share the target memory space of the target virtual index node;

[0013] The target file is read from the target memory space.

[0014] The embodiment of the present application also provides a data processing system, including: a service node and a storage node, wherein the storage node stores data and metadata of the data; the metadata includes a data fingerprint of the data; the service node reads the data through a read-only file system, and creates an independent index node for target data with the same content but different identifiers;

[0015] The service node is used to execute the steps of the aforementioned data reading method and / or container file reading method.

[0016] The embodiment of the present application further provides an electronic device, comprising: a memory and a processor; wherein the memory is used to store a computer program;

[0017] The processor is coupled to the memory, and is configured to execute the computer program to perform the steps of the aforementioned data reading method and / or container file reading method.

[0018] An embodiment of the present application also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed by one or more processors, the one or more processors are caused to execute the steps of the aforementioned data reading method and / or container file reading method.

[0019] An embodiment of the present application also provides a computer program product, including a computer program. When the computer program is executed by one or more processors, the one or more processors are caused to execute the steps of the aforementioned data reading method and / or container file reading method.

[0020] In the embodiment of the present application, data with the same content has the same data fingerprint. Multiple index nodes corresponding to multiple identifiers with the same data fingerprint are mapped to the same virtual index node, so that multiple index nodes can share the target memory space corresponding to the virtual index node, that is, multiple index nodes can share the data stored in the target memory space of the virtual index node, without caching a copy of the target data in each index node, thereby reducing the occupation of memory resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0022] Figure 1 A schematic diagram of the data reading process of a traditional container image reading solution;

[0023] Figure 2 and Figure 3 A schematic diagram of the structure of a data processing system provided in an embodiment of the present application;

[0024] Figure 4 A schematic diagram of a file reading process in a container scenario provided in an embodiment of the present application;

[0025] Figure 5 A schematic diagram of a data reading method according to an embodiment of the present invention;

[0026] Figure 6 A flowchart of a container file reading method provided in an embodiment of the present application;

[0027] Figure 7 A schematic diagram of the file life cycle when reading data provided in an embodiment of the present application;

[0028] Figure 8 A schematic diagram of the life cycle of an index node when reading data provided in an embodiment of the present application;

[0029] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0031] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0032] Before describing the solutions provided in the embodiments of the present application, the terms or concepts involved in the embodiments of the present application are first explained.

[0033] Enhanced Read-Only File System (EROFS): EROFS is a read-only file system in an operating system (such as Linux). EROFS can be used for firmware, read-only data partitions, and other scenarios that require efficient data reading.

[0034] Page Cache: Page cache is a memory cache mechanism used by the operating system to temporarily store data read from disk or to be written to disk. Its main purpose is to improve system performance and response speed.

[0035] Container Image: A container image is a software package that contains everything needed to run an application, including code, runtime libraries, system tools, system libraries, and configurations. A container image is a static, immutable template for a container that defines the container's operating environment and startup instructions.

[0036] Container: A container is an instance running based on a container image. It is a lightweight and independent execution environment. It uses virtualization technology at the operating system layer to allow applications and their dependencies to run in an isolated user space while sharing the kernel of the host machine.

[0037] Inode: An index node is a data structure in the file system that is used to store file metadata, including file size, file permissions, file type, owner and group, creation time, last modification time, last access time, and a pointer to the file data.

[0038] Hash Function: A hash function, also known as a hash function, is a mathematical function that maps input data (also called a "key") to a fixed-length value or hash value (usually an integer).

[0039] Extended File Attributes: Extended file attributes refer to additional information that can be defined by users or applications in addition to the traditional metadata of the file (such as the metadata included in the aforementioned index node). They provide a flexible way for the file system to store and retrieve additional information without embedding this information into the file content itself. In the embodiment of the present application, extended file attributes also belong to the metadata of the file.

[0040] At present, when a read-only file system, such as the EROFS file system, reads multiple files with different identifiers but the same content, multiple page caches will be generated in the memory. Each identifier corresponds to a page cache. Among them, the file identifier includes: file path and file name. For example, the different file identifiers can be different file paths but the same file name, the same file path but different file name, or different file paths and different file names. Figure 1 The process of the EROFS file system reading the image file of the container is exemplified.

[0041] like Figure 1 As shown, the image A of container A and the image B of container B have the same shared library file libc.so.6. After containers A and B are running, since the mount points of image A and image B are different, the mount point of image A is "mnt_a" and the mount point of image B is "mnt_b", the file paths of the shared library file libc.so.6 are " / mnt_a / libc.so.6" and " / mnt_b / libc.so.6" respectively. Therefore, when container A and container B read the shared library file libc.so.6 through the EROFS file system, two different index nodes, index node A and index node B, will be generated. Each index node has its own page cache, so when container A and container B read the shared library file libc.so.6 through index node A and index node B respectively, two page caches with the same content will be generated in the memory, that is, Figure 1 Page cache A and page cache B in the , resulting in a waste of memory resources.

[0042] In response to the above-mentioned technical problem of repeated storage of files or data with the same content in the memory, in some embodiments of the present application, data with the same content has the same data fingerprint, and multiple index nodes corresponding to multiple identifiers with the same data fingerprint are mapped to the same virtual index node, so that multiple index nodes can share the target memory space corresponding to the virtual index node, that is, multiple index nodes can share the data stored in the target memory space of the virtual index node, and there is no need to cache a copy of the target data in each index node, thereby reducing the occupancy of memory resources.

[0043] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0044] It should be noted that the same reference numerals denote the same objects in the following drawings and embodiments, and therefore, once an object is defined in one drawing or embodiment, it does not need to be further discussed in the subsequent drawings and embodiments.

[0045] Figure 2 and Figure 3 This is a schematic diagram of the structure of the data processing system provided in the embodiment of the present application. Figure 2 and Figure 3 As shown, the data processing system includes: a service node 10 and a storage node 20.

[0046] In this embodiment, the service node 10 refers to a computer device that can perform data management. The service node 10 can be a single server device, a cloud server array, or a virtual machine (VM) running in a cloud server array. In addition, the service node can also refer to other computing devices with corresponding service capabilities, such as mobile phones, computers and other terminal devices (running service programs), etc.

[0047] The storage node 20 refers to a device or equipment that can provide persistent storage resources. The service node 10 and the storage node 20 can be located in the same physical machine or in different physical machines. In an embodiment where the storage node 20 and the service node 10 are located in the same physical machine, the storage node 20 can be implemented as a persistent storage medium of the service node 10, such as a disk, a hard disk, etc. In an embodiment where the storage node 20 and the service node 10 are located in different physical machines, the storage node 20 can be implemented as a storage resource provided by cloud storage, such as a storage resource in a database, a mirror library, etc.

[0048] In this example, the service node 10 runs a read-only file system and reads data through the read-only file system. The read-only file system may be an EROFS file system, a read-only memory file system (ROMFS) or a compressed read-only memory file system (CROMFS), etc., but is not limited thereto. The data read by the service node 10 through the read-only file system may be data stored in the form of files.

[0049] In this embodiment, the storage node 20 stores data and metadata of the data. The metadata of the data may include: the size of the data, data permissions, the file type of the file to which the data belongs, the owner of the data, the creation time, the last modification time, the last access time and the storage location information of the data. The storage location information of the data may include the starting storage location of the data in the storage node and the data length. The starting storage location of the data in the storage node may be represented by a pointer pointing to the data. The storage location information is generally the physical address information of the data in the storage node.

[0050] The read-only file system can read data based on the index node of the data. For target data with the same content but different identifiers, multiple index nodes are created in the memory, where one identifier corresponds to one index node. Multiple refers to more than 2 (including 2). For example, Figure 2 and Figure 3 The identifiers of the target data are identifiers 1 and 2, and two index nodes corresponding to identifiers 1 and 2 can be created in the memory. The index node corresponding to the target data of each identifier can be the currently created one or the one that has been created before and has not been deleted.

[0051] The identifier of the data is an identifier used by the read-only file system to distinguish and identify the data, and may include: the file path of the data and the file name corresponding to the data. The identifiers of target data with the same content are different, and may include: the file paths of the target data are different and / or the file names of the target data are different, that is, the file paths of the target data are different but the file names are the same, the file paths of the target data are the same but the file names are different, and the file paths of the target data are different and the file names are different. Any of these situations means that the identifiers of the target data are different. In the traditional solution, each index node is independent of each other and has an independent cache space in the memory, resulting in a waste of memory resources.

[0052] In an embodiment of the present application, in order to reduce the occupation of memory resources, a cache sharing mechanism is proposed, that is, multiple index nodes of target data with different identifiers but the same content are mapped to the same virtual index node through data content. These multiple index nodes share the cache corresponding to the virtual index node, so that target data with the same content retains a cache in the memory, which can save memory resources. Among them, the virtual index node is visible to the read-only file system and is invisible to other systems outside the read-only file system. For the convenience of description, the same virtual index node to which multiple index nodes are mapped is defined as the target virtual index node.

[0053] Among them, Figure 2 As shown in , the target virtual index node may be an index node created in addition to the multiple index nodes. Figure 3As shown, an index node determined from multiple index nodes may also be used as the target virtual index node. In an embodiment where the target virtual index node is one of multiple index nodes, any index node from the multiple index nodes may be selected as the target virtual index node. Alternatively, the first or last created index node may be selected as the target virtual index node, and so on. Figure 3 The middle index node 30 - 1 is illustrated as the target virtual index node.

[0054] In order to map multiple index nodes of target data with the same content to the same virtual index node, and data with different contents will not be mapped to the same virtual index node, in the embodiment of the present application, the metadata of the data (such as files, etc.) can be extended, and the customized metadata can be called file extension attributes. The name of the extended attribute can be called "fingerprint", where the data fingerprints of data with the same content are the same, and the data fingerprints of data with different contents are different. Different virtual index nodes correspond to different data fingerprints.

[0055] Among them, data fingerprint is a technology used to identify the uniqueness of a data set or file. It is usually a fixed-length string generated by an algorithm, and this string is calculated based on the content of the data. Data fingerprints can use hash functions, checksum functions, or digital signatures to determine the data fingerprint of data. For example, a hash function can be used to calculate the hash value of data as the data fingerprint of the data. The hash function can be Message-Digest Algorithm 5 (MD5), Secure Hash Algorithms1 (SHA)1, that is, SHA-1, SHA-224, SHA-256, SHA-384, or SHA-512, etc.

[0056] Among them, MD5 can convert data of any length into a 128-bit (16-byte) hexadecimal string. SHA-1 can convert data of any length into a 160-bit (20-byte) string. SHA-224, SHA-256, SHA-384, and SHA-512 are all part of the SHA-2 family, and can convert data of any length into 224-bit, 256-bit, 384-bit, and 512-bit strings, respectively.

[0057] Checksum is a simple form of data fingerprint. It is a numerical value obtained by performing mathematical operations on the data as the data fingerprint of the data. If the data changes, the checksum will also change. The checksum can be a cyclic redundancy check (CRC) algorithm, etc.

[0058] Digital signatures can combine public key encryption technology and hash functions to encrypt data and obtain the data fingerprint. If the data changes, the digital signature will also change.

[0059] Based on the above analysis, in the embodiment of the present application, the data fingerprint of the data stored in the storage node 20 can be determined by the data fingerprint technology, and the data fingerprint of the data is stored. Among them, the data fingerprint is also a kind of metadata of the data. For example, for a container image, the data fingerprint of each file in the container image can be determined by the data fingerprint technology, and the data fingerprint of the file is stored. Among them, the data fingerprint of the data can be stored in the extended attribute list.

[0060] Since data with the same content has a unique data fingerprint, and different data has different data fingerprints, it is possible to distinguish whether the content of the data is the same based on the data fingerprint. Figure 2 and Figure 3 When the read-only file system performs a read operation on target data of multiple identifiers, the service node 10 can read metadata of the target data of multiple identifiers. The multiple identifiers refer to multiple different identifiers. The metadata of the target data includes a data fingerprint of the target data.

[0061] Further, the service node 10 may obtain multiple index nodes 30-n according to the metadata of the target data of the multiple identifiers, where n=1, 2, ... N. N represents the number of identifiers corresponding to the target data, and N≥2, and is an integer. Figure 2 and Figure 3 Only N=2 is used for illustration, but it is not a limitation. Each index node 30-n corresponds to an identifier. The index node corresponding to the target data of each identifier can be created in real time, or can be created before and not yet deleted.

[0062] Furthermore, multiple index nodes 30-n may be mapped to the same virtual index node 40 (defined as the target virtual index node 40) according to the data fingerprint of the target data. Multiple index nodes 30-n share the memory space (defined as the target memory space) corresponding to the target virtual index node 40. In this way, the service node 10 may read the target data from the target memory space.

[0063] In this embodiment, data with the same content has the same data fingerprint through data fingerprint technology, and multiple index nodes corresponding to multiple identifiers with the same data fingerprint are mapped to the same virtual index node, so that multiple index nodes can share the target memory space corresponding to the virtual index node, that is, multiple index nodes can share the data stored in the target memory space of the virtual index node, without caching a copy of the target data in each index node, thereby reducing the occupation of memory resources. The saved memory resources can be used to store other data to improve memory usage efficiency.

[0064] The target memory space of the virtual index node can be managed in the form of memory pages. Memory pages refer to the basic units used by computer operating systems to manage physical memory. Memory is organized and managed in the form of memory pages. Physical memory is divided into a series of fixed-size blocks, each of which is called a memory page. The memory space corresponding to the virtual index node stores data in the form of memory pages. Therefore, the data stored in the memory space corresponding to the virtual index node can also be called page cache.

[0065] The process of mapping the index node corresponding to each identifier to the same virtual index node is the same or similar. The following takes any identifier X among the multiple identifiers as an example to exemplify the specific process of mapping multiple index nodes corresponding to the multiple identifiers to the same virtual index node.

[0066] Specifically, for any target data of identifier X, the service node 10 can obtain the metadata of the target data of identifier X when performing a read operation on the target data of identifier X in a read-only file system; further, the index node 30-n of the target data of identifier X can be obtained based on the metadata of the target data of identifier X.

[0067] In some embodiments, the index node 30-n of the target data of the identifier X is created in real time. Accordingly, the index node 30-n of the target data of the identifier X can be created according to the metadata of the target data of the identifier X. Since the index node 30-n is a data structure, the data structure corresponding to the index node 30-n can be initialized according to the metadata of the target data of the identifier X to obtain the index node 30-n of the target data of the identifier X. Specifically, the metadata of the target data of the identifier X can be written into the corresponding field contained in the data structure corresponding to the index node 30-n to realize the initialization of the data structure corresponding to the index node 30-n, and then obtain the index node 30-n of the target data of the identifier X. In the same way, multiple index nodes 30-n of the target data of multiple identifiers can be obtained.

[0068] In other embodiments, when the read-only file system performs a read operation on the target data of the identifier X, the index node 30-n of the target data of the identifier X has been created, and the index node 30-n of the target data of the identifier X can be obtained according to the metadata of the target data of the identifier X. Specifically, according to the metadata of the target data of the identifier X, a query can be performed in the created index nodes; if the index node whose field value is the metadata of the target data of the identifier X is queried, the index node whose field value is the metadata of the target data of the identifier X is determined to be the index node 30-n of the target data of the identifier X.

[0069] Furthermore, the target virtual index node corresponding to the target data of any identifier X may be determined according to the data fingerprint of the target data.

[0070] In some embodiments, the data fingerprint of the data and the memory address information of the virtual index node can be stored in the form of a key-value pair (KV), wherein the virtual fingerprint of the data is the key (Key), and the memory address information of the virtual index node is the value (Value) of the key (Key). When the service node 10 determines the target virtual index node corresponding to the target data of any identifier X according to the data fingerprint of the target data, it can determine whether the virtual index node corresponding to the target data already exists based on the stored KV.

[0071] Specifically, the service node 10 can query whether there is a target key-value pair corresponding to the data fingerprint of the target data in the service node 10 according to the data fingerprint of the target data. Among them, the key (Key) of the target key-value pair is the data fingerprint of the target data, and the value (Value) of the target key-value pair is the memory address information of the virtual index node. If there is a key-value pair corresponding to the data fingerprint of the target data in the service node, it means that there is a virtual index node corresponding to the data fingerprint of the target data in the service node 10, and then the virtual index node corresponding to the memory address information stored in the target key-value pair can be determined, that is, the target virtual index node.

[0072] If the target key-value pair corresponding to the data fingerprint of the target data does not exist in the service node, it means that the virtual index node corresponding to the data fingerprint of the target data does not exist in the service node, and you need to determine the virtual index node first. Specifically, if the key-value pair corresponding to the data fingerprint of the target data does not exist in the service node, you can create a target virtual index node based on the metadata of the target data. The obtained target virtual index node is as follows: Figure 2 Specifically, the data structure corresponding to the target virtual index node may be initialized according to the metadata of the target data to obtain the target virtual index node.

[0073] Specifically, metadata corresponding to the fields contained in the data structure corresponding to the target virtual index node is obtained from the metadata of the target data; the metadata corresponding to the fields contained in the data structure corresponding to the target virtual index node can be written into the corresponding fields contained in the data structure corresponding to the target virtual index node to initialize the data structure corresponding to the target virtual index node, and then obtain the target virtual index node.

[0074] Among them, the data structure of the target virtual index node may be the same as or different from the data structure of the index node. Preferably, the data structure of the target virtual index node is different from the index node, such as the target virtual index node does not contain the storage location information of the data, such as the disk pointer of the data. This is mainly because the virtual index node corresponds to multiple index nodes, and the file paths of the target data corresponding to the multiple index nodes may be different. If the target virtual index node stores a certain file path (i.e., the disk pointer of the target data corresponding to a certain index node), if the disk pointed to by the disk pointer is removed, the target virtual index node will be unavailable, thereby causing the operations of reading the target data corresponding to other index nodes to fail to read the data, affecting the stability of the system. Therefore, the data structure of the target virtual index node does not contain the storage location information of the data.

[0075] In other embodiments, if the key-value pair corresponding to the data fingerprint of the target data does not exist in the service node, an index node may be selected from the plurality of index nodes 30-n as the target virtual index node. Figure 3 The index node corresponding to the target data with the identifier 1 shown in the figure. In this embodiment, the data structure of the virtual index node is the same as that of the index node. This implementation does not require additional creation of virtual index nodes, which can simplify the determination logic of virtual index nodes. However, this implementation is different from Figure 2 The implementation of the target virtual index node shown in the figure has the following defects: Figure 3 Taking the implementation method shown as an example, the index node corresponding to identifier 1 is used as the target virtual index node. If a destruction event occurs for the target data file of identifier 1, since the index node is still referenced by other index nodes, the destruction of the target data file of identifier 1 will fail. For example, assuming that the target data is a target file in a container image, if the container corresponding to identifier 1 is to be deleted, but the index node of the target file (the index node corresponding to identifier 1) is still referenced by the index node corresponding to identifier 2, the deletion of the container corresponding to identifier 1 will fail. Therefore, the target virtual index node preferably adopts Figure 2 Provided implementation.

[0076] After determining the target virtual index node, the service node 10 may generate a target key-value pair with the data fingerprint of the target data as the key (Key) and the memory address information of the target virtual index node as the value (Value), and store the target key-value pair. Further, the service node 10 may perform an operation of establishing a mapping relationship between the index node of the target data of the identifier X and the target virtual index node, thereby mapping the index node of the target data of the identifier X to the target virtual index node.

[0077] After determining the target virtual index node corresponding to the target data of identifier X, a mapping relationship between the index node of the target data of identifier X and the target virtual index node 40 may be established, thereby mapping the index node 30 - n of the target data of identifier X to the virtual index node 40 .

[0078] Specifically, the memory address information of the index node of the target data of the identifier X can be added to the linked list corresponding to the target virtual index node 40, so as to establish a mapping relationship between the index node of the target data of the identifier X and the target virtual index node 40, and realize mapping the index node 30-n of the target data of the identifier X to the target virtual index node 40. In the same or similar manner, multiple index nodes 30-n of the target data of multiple identifiers can be mapped to the same target virtual index node 40. These multiple index nodes 30-n can share the target memory space of the target virtual index node 40.

[0079] After establishing the mapping relationship between the index node of the target data of the identifier X and the target virtual index node 40, the number of references of the target virtual index node 40 may be increased. Specifically, the target virtual index node 40 is configured with a counter, which is used to count the number of references of the target virtual index node 40. Accordingly, after establishing the mapping relationship between the index node of the target data of the identifier X and the target virtual index node 40, the current count value of the counter of the target virtual index node 40 may be increased by 1, thereby increasing the number of references of the target virtual index node 40.

[0080] After mapping multiple index nodes corresponding to target data of multiple identifiers to the same target virtual index node, multiple index nodes 30-n can share the target data stored in the target memory space of the target virtual index node. Therefore, when the service node 10 performs a read operation on target data of any identifier X, the target data can be read from the target memory space. Specifically, if the target virtual index node corresponding to the data fingerprint of the target data already exists in the service node 10 when the service node 10 performs a read operation on the target data of any identifier X, the target data can be directly read from the target memory space corresponding to the target virtual index node.

[0081] If the target data does not exist in the target memory space corresponding to the target virtual index node, the storage location information of the target data with identifier X in the storage node 20 can be obtained from the index node corresponding to the target data with identifier X; thereafter, the target data can be read into the target memory space based on the storage location information of the target data with identifier X in the storage node 20.

[0082] If the service node 10 does not have a target virtual index node corresponding to the data fingerprint of the target data when performing a read operation for the target data of any identifier X, the service node 10 may create a target virtual index node in the manner shown in the above embodiment and allocate a target memory space for the target virtual index node. Further, the service node 10 may obtain the storage location information of the target data of the identifier X in the storage node 20 from the index node corresponding to the target data of the identifier X; thereafter, the target data may be read into the target memory space according to the storage location information of the target data of the identifier X in the storage node 20.

[0083] When the read-only file system performs a read operation on target data of multiple identifiers, what is obtained is the logical address of the target data of each identifier. Therefore, for the target data of identifier X, the logical address of the target data can be converted into a physical address based on the storage location information of the target data of identifier X in the storage node 20; thereafter, the target data can be read from the storage node 20 based on the physical address, and the target data can be stored in the target memory space.

[0084] For example, if the index node stores physical address 1234, it means that the starting physical address of the index node on the storage node (such as a physical disk) is 1234. If you want to read the target data of the first 100 bytes of the target file corresponding to the index node (that is, the content of logical addresses 0-100), you can convert the logical address 0-100 to the physical address 1234-1334 according to the physical address 1234 stored in the metadata. After that, you can read the target data from the storage node according to the physical address 1234-1334.

[0085] Among them, after the target data is stored in the target memory space, when the service node 10 performs a read operation on the target data of other identifiers, the target data can be directly read from the target memory space, so that the data stored in the target memory space of the target virtual index node can be shared among multiple index nodes, without caching a copy of the target data in each index node, thereby reducing the occupation of memory resources. The saved memory resources can be used to store other data to improve memory usage efficiency. Accordingly, after the target data is stored in the target memory space, the service node 10 can read the target data from the target memory space.

[0086] In addition to providing a solution for multiple index nodes to share the target memory space of a target virtual index node, the embodiment of the present application also provides a solution for releasing the mapping relationship between an index node and a target virtual index node. The following still takes the index node corresponding to the target data of any identifier X among the multiple index nodes corresponding to the target data of multiple identifiers as an example to exemplify the solution for releasing the mapping relationship between an index node and a target virtual index node.

[0087] Specifically, the service node 10 may monitor the destruction event of the index node corresponding to the target data of the identifier X. In the embodiment of the present application, the specific implementation method of the destruction event of the index node corresponding to the target data of the identifier X is not limited. In some embodiments, the service node 10 may monitor the closing operation of the file corresponding to the target data of the identifier X, and if the closing operation of the file corresponding to the target data of the identifier X is monitored, it is determined that the destruction event of the index node of the target data X of the identifier is generated. In other embodiments, the service node 10 may also monitor the reading progress of the target data of the identifier X, and when it is monitored that the reading of the target data of the identifier X is completed, it is determined that the destruction event of the index node of the target data X of the identifier is generated.

[0088] Further, the service node 10 may respond to the destruction event of the index node of the target data X of the identifier, and determine whether the target virtual index node is mapped with other index nodes according to the number of references of the target virtual index node; other index nodes refer to index nodes other than the index node of the target data of the identifier X. If the number of references of the target virtual index node is 1, it means that the index node mapped by the target virtual index node is only the index node of the target data of the identifier X, that is, no other index nodes are mapped, then the target virtual index node can be deleted, and the resources occupied by the target virtual index node can be released. Among them, the resources occupied by the target virtual index node include the target memory space corresponding to the aforementioned target virtual index node, and of course, can also include the storage resources occupied by the target virtual index node itself.

[0089] Accordingly, if the number of references to the target virtual index node is greater than 1, it means that the target virtual index node is mapped with other index nodes in addition to the index node of the target data X with the identifier, and the mapping relationship between the index node of the target data with the identifier X and the target virtual index node can be released. Specifically, the memory address information of the index node of the target data with the identifier X can be deleted from the linked list corresponding to the target virtual index node, and the number of references to the virtual index node can be reduced, thereby releasing the mapping relationship between the index node of the target data with the identifier X and the target virtual index node. Furthermore, the index node of the target data with the identifier X can be deleted. In this way, other index nodes mapped by the target virtual index node can continue to use and share the target memory space of the target virtual index node.

[0090] The data reading method provided in the embodiment of the present application is applicable to the reading of data stored in any form. The following is an exemplary description of the process of a container reading a container's image file. Figure 4 As shown, in the container scenario, multiple containers are deployed on the service node 10. Multiple refers to more than 2 (including 2). Figure 4 Only two containers, container A and container B, are shown in the figure, but this is not a limitation. The image files of multiple containers have target files with the same content. Figure 4 In the example, image A of container A and image B of container B have the same target file "libc.so.6" with the same content. The target file can be a partial file of the image files of multiple containers, or it can be all the files. Since the mount points of the image files of multiple containers are different, the read-only file system has different file paths when reading the target files contained in the image files of different containers. That is, the file paths of the target files of multiple containers are different, which leads to different identifiers of the target files of multiple containers. Therefore, the read-only file system creates multiple index nodes in memory for target files with the same content but different file paths. Each file path corresponds to an index node. This will also cause a waste of content resources. In this container scenario, the target file is the target data with different identifiers but the same content.

[0091] In an embodiment of the present application, in order to reduce the occupancy of memory resources, multiple index nodes of target files with different file paths but the same content are mapped to the same target virtual index node. These multiple index nodes share the cache corresponding to the target virtual index node, so that target files with the same content retain a cache in the memory, which can save memory resources.

[0092] In order to achieve mapping of multiple index nodes 30-n of target files with the same content to the same target virtual index node, and files with different contents will not be mapped to the same target virtual index node, in the embodiment of the present application, the data fingerprint of the content of the container's mirror file can be determined by data fingerprint technology, and the data fingerprint of the content of the mirror file can be stored. Among them, the data fingerprint of the content of the mirror file includes: the data fingerprint of the content of each file contained in the mirror file, and of course, the data fingerprint of the content of the target file. For the description of the data fingerprint, please refer to the relevant content of the aforementioned embodiment, which will not be repeated here.

[0093] Based on the data fingerprint of the file content, in this embodiment, when the read-only file system performs a read operation on target files of multiple file paths, the service node 10 can read the metadata of the target files of multiple file paths. The metadata of the target files includes the data fingerprint of the target files.

[0094] Further, the service node 10 may create multiple index nodes 30-n based on the metadata of the target files of the multiple file paths. Each index node 30-n corresponds to a file path. Further, the multiple index nodes 30-n may be mapped to the same target virtual index node 40 based on the data fingerprint of the target file. The multiple index nodes 30-n share the memory space (defined as the target memory space) corresponding to the target virtual index node 40. In this way, the service node 10 may read the target file from the target memory space. Among them, regarding the specific implementation method of mapping multiple index nodes to the same target virtual index node and reading the target file from the target memory space, please refer to the relevant content of the aforementioned embodiment, which will not be repeated here.

[0095] In this embodiment, data fingerprint technology is used to make target files with the same content have the same data fingerprint, and multiple index nodes corresponding to multiple target files are mapped to the same target virtual index node, so that multiple index nodes can share the target memory space corresponding to the target virtual index node, that is, multiple index nodes can share the data stored in the target memory space of the target virtual index node, and there is no need to cache a copy of the target file in each index node, thereby reducing the occupancy of memory resources.

[0096] In addition to the data processing system provided in the aforementioned embodiment, the embodiment of the present application also provides a data reading method. The data reading method provided in the embodiment of the present application is exemplarily described below from the perspective of a service node.

[0097] Figure 5 Schematic diagram of the data reading method provided in the embodiment of the present application. Figure 5 As shown, the method mainly includes the following steps:

[0098] 501. When a read-only file system performs a read operation on target data of multiple identifiers, metadata of the target data of the multiple identifiers is obtained; the metadata of the target data includes a data fingerprint of the target data.

[0099] 502. Acquire multiple index nodes according to metadata of target data of multiple identifiers; the multiple identifiers correspond to the multiple index nodes one by one.

[0100] 503. Map multiple index nodes to the same target virtual index node according to the data fingerprint of the target data, so that the multiple index nodes share the target memory space of the target virtual index node.

[0101] 504. Read target data from the target memory space.

[0102] The read-only file system can read data based on the index node of the data. For target data with the same content but different identifiers, multiple index nodes are created in the memory. Each index node is independent of each other and has an independent cache space in the memory, resulting in a waste of memory resources.

[0103] In an embodiment of the present application, in order to reduce the occupancy of memory resources, multiple index nodes of target data with different identifiers but the same content are mapped to the same target virtual index node. These multiple index nodes share the cache corresponding to the target virtual index node, so that the target data with the same content retains a cache in the memory, which can save memory resources.

[0104] The target virtual index node may be an index node created in addition to the multiple index nodes. Alternatively, an index node determined from the multiple index nodes may be used as the target virtual index node. In an embodiment where the target virtual index node is an index node from the multiple index nodes, any index node from the multiple index nodes may be selected as the target virtual index node. Alternatively, the first or last created index node may be selected as the target virtual index node, and so on.

[0105] In order to map multiple index nodes of target data with the same content to the same target virtual index node, and to prevent data with different contents from being mapped to the same target virtual index node, the data fingerprint of the data can be determined and stored by using data fingerprint technology, where the data fingerprint is also a kind of metadata of the data.

[0106] Since data with the same content has a unique data fingerprint and different data has different data fingerprints, it is possible to distinguish whether the content of the data is the same based on the data fingerprint of the data. Based on this, when the read-only file system performs a read operation on target data with multiple identifiers, in step 501, metadata of the target data with multiple identifiers can be read. The multiple identifiers refer to multiple different identifiers. The metadata of the target data includes the data fingerprint of the target data.

[0107] Further, in step 502, multiple index nodes may be obtained according to metadata of target data of multiple identifiers. Each index node corresponds to an identifier. For a specific implementation of obtaining multiple index nodes according to metadata of target data of multiple identifiers, please refer to the relevant content of the aforementioned system embodiment.

[0108] Further, in step 503, multiple index nodes may be mapped to the same target virtual index node according to the data fingerprint of the target data. Multiple index nodes share the memory space corresponding to the target virtual index node (defined as the target memory space). Thus, in step 504, the target data may be read from the target memory space.

[0109] In this embodiment, data with the same content has the same data fingerprint through data fingerprint technology, and multiple index nodes corresponding to multiple identifiers with the same data fingerprint are mapped to the same target virtual index node, so that multiple index nodes can share the target memory space corresponding to the target virtual index node, that is, multiple index nodes can share the data stored in the target memory space of the target virtual index node, without caching a copy of the target data in each index node, thereby reducing the occupation of memory resources. The saved memory resources can be used to store other data to improve memory usage efficiency.

[0110] The process of mapping the index node corresponding to each identifier to the same target virtual index node is the same or similar. The following takes any identifier X among the multiple identifiers as an example to exemplify the specific process of mapping multiple index nodes corresponding to the multiple identifiers to the same target virtual index node.

[0111] Specifically, for any target data of identifier X, when a read operation for the target data of identifier X is performed in the read-only file system, metadata of the target data of identifier X can be obtained; further, the index node of the target data of identifier X can be obtained according to the metadata of the target data of identifier X. In the same way, multiple index nodes of target data of multiple identifiers can be obtained.

[0112] Further, the target virtual index node corresponding to the target data of any identifier X may be determined based on the data fingerprint of the target data. In some embodiments, the data fingerprint of the data and the memory address information of the virtual index node are stored in the form of KV, wherein the virtual fingerprint of the data is the key (Key), and the memory address information of the virtual index node is the value (Value) of the key (Key). When the target virtual index node corresponding to the target data of any identifier X may be determined based on the data fingerprint of the target data, it may be determined whether the virtual index node corresponding to the target data already exists based on the stored KV.

[0113] Specifically, based on the data fingerprint of the target data, it is possible to query whether there is a target key-value pair corresponding to the data fingerprint of the target data in the service node. Among them, the key (Key) of the target key-value pair is the data fingerprint of the target data, and the value (Value) of the target key-value pair is the memory address information of the virtual index node. If there is a key-value pair corresponding to the data fingerprint of the target data, it means that there is a virtual index node corresponding to the data fingerprint of the target data in the service node, and then the virtual index node corresponding to the memory address information stored in the target key-value pair can be determined, that is, the target virtual index node.

[0114] If the target key-value pair corresponding to the data fingerprint of the target data does not exist in the service node, it means that the virtual index node corresponding to the data fingerprint of the target data does not exist in the service node, and the virtual index node needs to be determined first. Specifically, if the key-value pair corresponding to the data fingerprint of the target data does not exist in the service node, the target virtual index node can be created according to the metadata of the target data. Specifically, the data structure corresponding to the target virtual index node can be initialized according to the metadata of the target data to obtain the target virtual index node.

[0115] In other embodiments, if there is no key-value pair corresponding to the data fingerprint of the target data, an index node may be selected from multiple index nodes as the target virtual index node to obtain the target virtual index node. In this embodiment, the data structure of the virtual index node is the same as that of the index node.

[0116] After determining the target virtual index node, a target key-value pair may be generated with the data fingerprint of the target data as the key (Key) and the memory address information of the target virtual index node as the value (Value), and the target key-value pair may be stored. Further, an operation of establishing a mapping relationship between the index node of the target data with identifier X and the target virtual index node may be performed, thereby mapping the index node of the target data with identifier X to the target virtual index node.

[0117] After determining the target virtual index node corresponding to the target data of identifier X, a mapping relationship between the index node of the target data of identifier X and the target virtual index node may be established, thereby mapping the index node of the target data of identifier X to the virtual index node.

[0118] Specifically, the memory address information of the index node of the target data of the identifier X can be added to the linked list corresponding to the target virtual index node, so as to establish a mapping relationship between the index node of the target data of the identifier X and the target virtual index node, and realize mapping the index node of the target data of the identifier X to the target virtual index node. In the same or similar manner, multiple index nodes of the target data of multiple identifiers can be mapped to the same target virtual index node. These multiple index nodes can share the target memory space of the target virtual index node. After establishing the mapping relationship between the index node of the target data of the identifier X and the target virtual index node, the number of references of the target virtual index node can also be increased.

[0119] After mapping multiple index nodes corresponding to target data of multiple identifiers to the same target virtual index node, multiple index nodes can share the target data stored in the target memory space of the target virtual index node. Therefore, in the process of performing a read operation on target data of any identifier X, the target data can be read from the target memory space. Specifically, if, when performing a read operation on target data of any identifier X, a target virtual index node corresponding to the data fingerprint of the target data already exists in the service node, the target data can be directly read from the target memory space corresponding to the target virtual index node.

[0120] If the target data does not exist in the target memory space corresponding to the target virtual index node, the storage location information of the target data with identifier X in the storage node can be obtained from the index node corresponding to the target data with identifier X; then, the target data can be read into the target memory space according to the storage location information of the target data with identifier X in the storage node.

[0121] If, when performing a read operation on target data of any identifier X, the target virtual index node corresponding to the data fingerprint of the target data does not exist in the service node, the target virtual index node can be created in the manner shown in the above embodiment, and the target memory space can be allocated to the target virtual index node. Further, the storage location information of the target data of identifier X in the storage node can be obtained from the index node corresponding to the target data of identifier X; thereafter, the target data can be read into the target memory space according to the storage location information of the target data of identifier X in the storage node.

[0122] Furthermore, the target data can be read from the target memory space. Afterwards, when performing a read operation on the target data of other identifiers, the target data can be directly read from the target memory space, so that the data stored in the target memory space of the target virtual index node can be shared among multiple index nodes, without having to cache a copy of the target data in each index node, thereby reducing the occupation of memory resources. The saved memory resources can be used to store other data to improve memory usage efficiency.

[0123] In addition to providing a solution for multiple index nodes to share the target memory space of the target virtual index node, the embodiment of the present application also provides a solution for releasing the mapping relationship between the index node and the target virtual index node. Specifically, the destruction event of the index node corresponding to the target data of identifier X can be monitored. In the embodiment of the present application, the specific implementation method of the destruction event of the index node corresponding to the target data of identifier X is not limited. In some embodiments, the closing operation of the file corresponding to the target data of identifier X can be monitored, and if the closing operation of the file corresponding to the target data of identifier X is monitored, it is determined that the destruction event of the index node of the target data X of the identifier is generated. In other embodiments, the reading progress of the target data of identifier X can also be monitored, and when the reading of the target data of identifier X is monitored, it is determined that the destruction event of the index node of the target data X of the identifier is generated.

[0124] Furthermore, in response to the destruction event of the index node of the target data X of the identifier, it can be determined whether the target virtual index node is mapped with other index nodes according to the number of references of the target virtual index node; other index nodes refer to index nodes other than the index node of the target data of the identifier X. If the number of references of the target virtual index node is 1, it means that the index node mapped by the target virtual index node is only the index node of the target data X of the identifier, that is, no other index nodes are mapped, then the target virtual index node can be deleted, and the resources occupied by the target virtual index node can be released. Among them, the resources occupied by the target virtual index node include the target memory space corresponding to the aforementioned target virtual index node, and of course, can also include the storage resources occupied by the target virtual index node itself.

[0125] Accordingly, if the number of references to the target virtual index node is greater than 1, it means that the target virtual index node is mapped with other index nodes in addition to the index node of the target data X with the identifier, and the mapping relationship between the index node of the target data with the identifier X and the target virtual index node can be released. Specifically, the memory address information of the index node of the target data with the identifier X can be deleted from the linked list corresponding to the target virtual index node, and the number of references to the virtual required node can be reduced, thereby releasing the mapping relationship between the index node of the target data with the identifier X and the target virtual index node. Furthermore, the index node of the target data with the identifier X can be deleted. In this way, other index nodes mapped by the target virtual index node can continue to use and share the target memory space of the target virtual index node.

[0126] The data reading method provided in the embodiment of the present application is applicable to reading data stored in any form, and the following is an exemplary description taking the process of a container reading an image file of a container as an example. Figure 6The flowchart of the container file reading method provided in the embodiment of the present application is shown in FIG. The method is applicable to a service node. The service node is deployed with multiple containers, and the image files of the multiple containers have target files with the same content, and the file paths of the target files of the multiple containers are different. Figure 6 As shown, the container file reading method mainly includes the following steps:

[0127] 601. When a read-only file system performs a read operation on target files of multiple containers, metadata of the target files of the multiple containers are obtained; the target files of the multiple containers include data fingerprints of contents of the target files.

[0128] 602. Acquire multiple index nodes according to metadata of target files of multiple containers; each file path corresponds to an index node.

[0129] 603. Map multiple index nodes to the same target virtual index node according to the data fingerprint of the content of the target file, so that the multiple index nodes share the target memory space of the target virtual index node.

[0130] 604. Read the target file from the target memory space.

[0131] In the container scenario, in order to map multiple index nodes of target files with the same content to the same target virtual index node, and files with different contents will not be mapped to the same target virtual index node, in an embodiment of the present application, the data fingerprint of the content of the container's mirror file can be determined by data fingerprint technology, and the data fingerprint of the content of the mirror file can be stored. Among them, the data fingerprint of the content of the mirror file includes: the data fingerprint of the content of each file contained in the mirror file, and of course, the data fingerprint of the content of the target file. For the description of the data fingerprint, please refer to the relevant content of the aforementioned embodiment, which will not be repeated here.

[0132] Based on the data fingerprint of the file content, in this embodiment, in step 601, when the read-only file system performs a read operation on target files of multiple file paths, metadata of the target files of multiple file paths can be read. The metadata of the target file includes the data fingerprint of the target file.

[0133] Further, in step 602, multiple index nodes can be obtained based on the metadata of the target files of the multiple file paths. Each index node corresponds to a file path. Further, in step 603, multiple index nodes can be mapped to the same target virtual index node based on the data fingerprint of the target file. Multiple index nodes share the memory space corresponding to the target virtual index node (defined as the target memory space). In this way, in step 604, the target file can be read from the target memory space. Among them, regarding the specific implementation method of mapping multiple index nodes to the same target virtual index node and reading the target file from the target memory space, please refer to the relevant content of the aforementioned embodiment, which will not be repeated here.

[0134] In this embodiment, data fingerprint technology is used to make target files with the same content have the same data fingerprint, and multiple index nodes corresponding to multiple target files are mapped to the same target virtual index node, so that multiple index nodes can share the target memory space corresponding to the target virtual index node, that is, multiple index nodes can share the data stored in the target memory space of the target virtual index node, and there is no need to cache a copy of the target file in each index node, thereby reducing the occupancy of memory resources.

[0135] The above embodiment exemplifies the data reading method provided by the embodiment of the present application from the perspective of the service node. Generally, the read-only file system can be divided into: file layer and index node layer. The following exemplifies the data reading method provided by the embodiment of the present application from the perspective of the file layer and index node layer.

[0136] Figure 7 The file life cycle when reading data provided by the embodiment of the present application. Figure 7 As shown, it mainly includes:

[0137] Step 1: The virtual file system in the kernel of the operating system (such as the Linux system) performs an open operation on the file to be read through the file opening interface provided by the read-only file system (such as the EROFS file system), and then jumps to step 2.

[0138] Step 2: Get the index node corresponding to the file to be read, and get the data fingerprint of the file to be read, and then go to step 3.

[0139] Step 3: According to the data fingerprint of the file to be read, and the KV list with the data fingerprint as the key and the memory address information of the target virtual index node as the value, determine the target virtual index node corresponding to the data fingerprint of the file to be read, and increase the reference count of the target virtual index node.

[0140] Step 4: Allocate a corresponding virtual file to the file to be read through the target virtual index node, and proceed to step 5.

[0141] Among them, the virtual file is an object used by the kernel to track file read and write operations. It does not correspond to a specific physical file, but serves as an abstraction of file operations.

[0142] Step 5: Use the virtual file corresponding to the file to be read to perform related read operations of the virtual file system layer in the kernel, and then go to step 6.

[0143] Step 6: If a read termination event occurs for the file to be read, proceed to step 7; if no read termination event occurs for the file to be read, proceed to step 5 to continue reading until the reading of the file to be read is completed, that is, a read termination event occurs for the file to be read.

[0144] The read termination event refers to an event of stopping reading the file to be read, such as completion of reading the file to be read, or closing the file to be read.

[0145] Step 7: Disassociate the file to be read from the virtual file, reduce the number of references to the virtual file, and release the resources occupied by the virtual file;

[0146] Step 8: Close and release the opened file to be read.

[0147] In this embodiment, the reading process of the file to be read is described from the perspective of the file layer by taking the file to be read as an example, wherein the file to be read is the target file or target data.

[0148] Figure 8 The life cycle of the index node when reading data provided by the embodiment of the present application. Figure 8 As shown, it mainly includes:

[0149] Step 1: Initialize the relevant fields of the index node according to the metadata of the file to be read, obtain the index node, and go to step 2.

[0150] Step 2: Get the data fingerprint corresponding to the current index node and jump to step 3.

[0151] Step 3: Determine whether the target virtual index node corresponding to the data fingerprint exists. If not, proceed to step 4; otherwise, it means that a mapping relationship has been established between the index node and the target virtual index node, and proceed to step 5.

[0152] Step 4: Create a target virtual index node according to the metadata of the file to be read, wherein the file system type corresponding to the target virtual index node is a virtual file type, and return to step 3.

[0153] Step 5: Establish a mapping relationship between the current index node and the target virtual index node, and jump to step 6.

[0154] Step 6: Read the file to be read from the target memory space corresponding to the target virtual index node, and continue to step 7.

[0155] Step 7: If the destruction event of the current index node occurs, execute step 8; otherwise, jump to step 6 to continue the read operation until the file to be read is completed, and then jump to step 8.

[0156] The destruction event of the current index node may be that the data to be read is completed, or the file to be read is closed.

[0157] Step 8: Determine whether the target virtual index node is mapped to other index nodes; if no other index nodes are mapped, execute step 9; otherwise, it means that there are still other index nodes using the target virtual index node to read data, then execute step 10.

[0158] Step 9: Destroy the target virtual index node, release the resources occupied by the target virtual index node, and proceed to step 10.

[0159] Step 10: Release the mapping relationship between the current index node and the target virtual index node, and delete the current index node.

[0160] In order to verify the performance of the data reading method provided in the embodiment of the present application, the inventors of the present application respectively used the data reading solution provided in the embodiment of the present application and the traditional container image reading solution to read the full amount of some container images, and compared the results shown in the following Table 1:

[0161] Table 1 Memory reduction for full reading of container images

[0162]

[0163] In Table 1, memory usage refers to the memory occupied by fully reading the container images listed in Table 1; the percentage of memory usage reduction refers to the percentage of memory occupied by fully reading the container images listed in Table 1 using the data reading method provided in the embodiment of the present application, compared with the percentage of memory occupied by fully reading the container images listed in Table 1 using the traditional container image reading scheme. In Table 1, Redis 7.2.4 & 7.2.5 refers to Redis images with version numbers 7.2.4 and 7.2.5. Redis is an open source key-value storage system, commonly used for caching and message queues. Postgres 16.1 & 16.2 refers to Postgres images with version numbers 16.1 and 16.2. Postgres is PostgreSQL, a powerful open source object-relational database system. Tensorflow 1.11.0 and 2.11.1 refer to Tensorflow images with version numbers 1.11.0 and 2.11.1. Tensorflow is a machine learning framework. Mysql 8.0.11&8.0.12 refers to the Mysql images with version numbers 8.0.11 and 8.0.12. Mysql is an open source relational database system. Nginx 7.2.4&7.2.5 refers to the Nginx images with version numbers 7.2.4 and 7.2.5. Nginx is a high-performance Hypertext Transfer Protocol (HTTP) and reverse proxy server, as well as a mail proxy server. Tomcat 10.1.25&10.1.26 refers to the Tomcat images with version numbers 10.1.25 and 10.1.26. Tomcat refers to an open source Servlet container, mainly used to deploy Java World Wide Web (Web) applications. Among them, Servlet is a component of the Java platform, which provides a standard way to create dynamic Web content.

[0164] The test results shown in Table 1 indicate that the data reading method provided in the embodiment of the present application can save 16%-49% of memory overhead compared to the traditional container image reading method.

[0165] The inventors of the present application also tested the memory usage of the data reading method provided by the embodiment of the present application when the container runs a typical load, and the memory usage of the traditional container image reading method when running a typical load, by running a typical load on the container corresponding to the container image, such as creating a data table, reading and writing data, and obtained the test results shown in Table 2:

[0166] Table 2 Memory reduction when running typical workloads

[0167]

[0168] In Table 2, memory usage refers to the memory occupied when a container built using the container image listed in Table 2 is used to run a typical load; the memory usage reduction percentage refers to the percentage of memory occupied when a typical load is run using a container built using the container image listed in Table 2 using the data reading method provided by an embodiment of the present application, compared to the percentage of memory occupied by a traditional container image reading solution when a container built using the container image listed in Table 2 is used to run a typical load.

[0169] The test results shown in Table 2 indicate that the data reading method provided in the embodiment of the present application can save 4%-47% of memory overhead when running a typical load using a container built using the container image listed in Table 2, compared to the traditional container image reading method.

[0170] It should be noted that the execution subject of each step of the method provided in the above embodiment can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 501 and 502 can be device A; for another example, the execution subject of step 501 can be device A, and the execution subject of step 502 can be device B; and so on.

[0171] In addition, in some of the processes described in the above embodiments and the accompanying drawings, multiple operations appearing in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this document or may be executed in parallel, and the sequence numbers of the operations, such as 501, 502, etc., are only used to distinguish between different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel.

[0172] Accordingly, an embodiment of the present application also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed by one or more processors, the one or more processors are caused to execute the steps in the aforementioned data reading method and / or container file reading method.

[0173] The embodiment of the present application also provides a computer program product, including a computer program. When the computer program is executed by one or more processors, the one or more processors are caused to execute the steps in the aforementioned data reading method and / or container file reading method. In the embodiment of the present application, the specific implementation form of the computer program product is not limited. In some embodiments, the computer program product can be implemented as an application (Application, APP), a mini-program, a computer-side client, a program module, a plug-in, an installation package, a software development kit (Software Development Kit, SDK), an image file of a CD (such as an ISO file), a plug-in or software in the form of software as a service (Software as a Service, SaaS), etc., but is not limited to this.

[0174] Fig. 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Fig. 9 As shown, the electronic device includes: a memory 90a and a processor 90b. The memory 90a is used to store computer programs.

[0175] The processor 90b is coupled to the memory 90a and is used to execute a computer program to execute the steps in the data reading method and / or container file reading method provided in the above embodiments. The specific implementation of each step can be found in the relevant description of the above embodiments, which will not be repeated here.

[0176] In some optional embodiments, such as Fig. 9 As shown, the electronic device may also include optional components such as a communication component 90c, a power component 90d, a display component 90e and an audio component 90f. Fig. 9 The components are shown schematically only, and do not mean that the electronic device must include Fig. 9 The components shown do not necessarily mean that the electronic device can only include Fig. 9 Components shown.

[0177] in addition, Fig. 9 The components in the dashed box are optional components, not mandatory components, and may depend on the product form of the electronic device. The electronic device of this embodiment may be implemented as a terminal device such as a desktop computer, a laptop computer, a mobile phone, or an IoT device; or it may be various server devices such as a traditional server, a cloud server, or a server cluster.

[0178] In an embodiment of the present application, the memory is used to store a computer program and can be configured to store various other data to support operations on the device where it is located. Among them, the processor can execute the computer program stored in the memory to implement the corresponding control logic. The memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random-Access Memory, SRAM), electrically erasable programmable read only memory (Electrically Erasable Programmable Read Only Memory, EEPROM), erasable programmable read only memory (Electrical Programmable Read Only Memory, EPROM), programmable read only memory (Programmable Read Only Memory, PROM), read only memory (Read Only Memory, ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0179] In the embodiment of the present application, the processor may be any hardware processing device that can execute the logic of the above method. Optionally, the processor may be a central processing unit (CPU), a graphics processing unit (GPU) or a microcontroller unit (MCU); it may also be a field programmable gate array (FPGA), a programmable array logic device (PAL), a general array logic device (GAL), a complex programmable logic device (CPLD) or other programmable devices; or it may be an advanced reduced instruction set (RISC) processor (Advanced RISC Machines, ARM) or a system on chip (System on Chip, SoC), etc., but not limited thereto.

[0180] In an embodiment of the present application, the communication component is configured to facilitate wired or wireless communication between the device in which it is located and other devices. The device in which the communication component is located can access a wireless network based on a communication standard, such as Wireless Fidelity (WiFi), 2G or 3G, 4G, 5G or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component can also be implemented based on Near Field Communication (NFC) technology, Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (BT) technology or other technologies.

[0181] In an embodiment of the present application, the display component may include a liquid crystal display (LCD) and a touch panel (TP). If the display component includes a touch panel, the display component may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0182] In an embodiment of the present application, a power supply component is configured to provide power to various components of the device in which it is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which the power supply component is located.

[0183] In an embodiment of the present application, the audio component may be configured to output and / or input audio signals. For example, the audio component includes a microphone (Microphone, MIC), and when the device where the audio component is located is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal may be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker for outputting an audio signal. For example, for a device with a language interaction function, voice interaction with a user may be achieved through an audio component.

[0184] It should be noted that the descriptions such as “first” and “second” in this article are used to distinguish different messages, devices, modules, etc., and do not represent the order of precedence, nor do they limit “first” and “second” to different types.

[0185] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, compact disc read-only memory (CD-ROM), optical storage, etc.) containing computer-usable program codes.

[0186] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0187] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0188] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0189] In a typical configuration, a computing device includes one or more processors (CPU, etc.), input / output interfaces, network interfaces, and memory.

[0190] Memory may include non-permanent storage in a computer-readable medium, in the form of random-access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0191] Computer storage media is readable storage media, also known as readable media. Readable storage media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0192] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of further restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the above elements.

[0193] The above contents are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A data reading method, characterized in that: include: In the case where the read-only file system performs a read operation on target data of a plurality of identifiers, obtaining metadata of the target data of the plurality of identifiers; The metadata of the target data includes a data fingerprint of the target data; the identifier includes: a file path and a file name of the target data; different identifiers correspond to different file paths of the target data and / or different identifiers correspond to different file names of the target data; the data fingerprint is a string of fixed length calculated based on the content of the target data; data with the same content has the same data fingerprint; Acquire a plurality of index nodes according to metadata of target data of the plurality of identifiers; the plurality of identifiers correspond one to one to the plurality of index nodes; According to the data fingerprint of the target data, mapping the multiple index nodes to the same target virtual index node, so that the multiple index nodes share the target memory space of the target virtual index node; The target data is read from the target memory space.

2. The method according to claim 1, characterized in that The mapping of the multiple index nodes to the same target virtual index node according to the data fingerprint of the target data includes: For target data of any identifier, determining a target virtual index node corresponding to the target data of any identifier according to a data fingerprint of the target data; A mapping relationship between the index node of the target data of any identifier and the target virtual index node is established to map the index node of the target data of any identifier to the target virtual index node.

3. The method according to claim 2, characterized in that The establishing of a mapping relationship between the index node of the target data of any identifier and the target virtual index node includes: The memory address information of the index node of the target data of any identifier is added to the linked list corresponding to the target virtual index node to establish a mapping relationship between the index node of the target data of any identifier and the target virtual index node.

4. The method according to claim 2, characterized in that: Determining the target virtual index node corresponding to the target data of any identifier according to the data fingerprint of the target data includes: According to the data fingerprint of the target data, query whether there is a target key-value pair corresponding to the data fingerprint of the target data; the key of the target key-value pair is the data fingerprint of the target data, and the value of the target key-value pair is the memory address information of the target virtual index node; If the target key-value pair exists, the virtual index node corresponding to the memory address information stored in the target key-value pair is determined to be the target virtual index node corresponding to the target data of any identifier.

5. The method according to claim 4, characterized in that The method further comprises: If the target key-value pair does not exist, creating the target virtual index node according to the metadata of the target data, or selecting an index node from the multiple index nodes as the target virtual index node; Generate the target key-value pair, and execute the operation of establishing a mapping relationship between the index node of the target data of any identifier and the target virtual index node.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: If the target data does not exist in the target memory space, obtain the storage location information of the target data of any identifier in the storage node from the index node of the target data of any identifier; read the target data into the target memory space according to the storage location information of the target data of any identifier in the storage node; read the target data from the target memory space.

7. The method according to claim 3, characterized in that The method further comprises: After establishing a mapping relationship between the index node of the target data of any one of the identifiers and the target virtual index node, the reference quantity of the target virtual index node is increased.

8. The method according to claim 7, characterized in that The method further comprises: In response to a destruction event of an index node of target data of any of the identifiers, judging whether the target virtual index node is mapped with other index nodes according to the number of references of the target virtual index node; the other index nodes refer to index nodes other than the index node of the target data of any of the identifiers; If the target virtual index node is mapped to other index nodes, the mapping relationship between the index node of the target data of any identifier and the target virtual index node is released, the reference data of the target virtual index node is reduced, and the index node of the target data of any identifier is deleted.

9. The method according to claim 8, characterized in that The method further comprises: If the target virtual index node is not mapped to other index nodes, the target virtual index node is deleted, and resources occupied by the target virtual index node are released; the resources occupied by the target virtual index node include the target memory space.

10. The method according to claim 8, characterized in that The method further comprises: If a closing operation of a file corresponding to the target data of any one of the identifiers is monitored, determining to generate a destruction event of an index node of the target data of any one of the identifiers; or, If it is monitored that the reading of the target data of any one of the identifiers is completed, it is determined to generate a destruction event of the index node of the target data of any one of the identifiers.

11. A container file reading method, applicable to a service node, characterized in that: The service node is deployed with multiple containers, the image files of the multiple containers have target files with the same content, and the file paths of the target files of the multiple containers are different; the method includes: In the case where the read-only file system performs a read operation on the target files of the multiple containers, metadata of the target files of the multiple containers are obtained; the target files of the multiple containers include data fingerprints of contents of the target files; the data fingerprint is a string of fixed length calculated according to the contents of the target files; target files with the same contents have the same data fingerprint; According to the metadata of the target files of the multiple containers, a plurality of index nodes are obtained, wherein each file path corresponds to an index node; According to the data fingerprint of the content of the target file, the multiple index nodes are mapped to the same target virtual index node, so that the multiple index nodes share the target memory space of the target virtual index node; The target file is read from the target memory space.

12. A data processing system, characterized in that: include: A service node and a storage node, wherein the storage node stores data and metadata of the data; the metadata includes a data fingerprint of the data; The service node reads the data through a read-only file system and creates an independent index node for target data with the same content but different identifiers; The service node is used to execute the steps in the method according to any one of claims 1 to 11.

13. An electronic device, characterized in that: include: A memory and a processor; wherein the memory is used to store a computer program; The processor is coupled to the memory and configured to execute the computer program to perform the steps of the method according to any one of claims 1 to 11.

14. A computer-readable storage medium storing computer instructions, characterized in that: When the computer instructions are executed by one or more processors, the one or more processors are caused to execute the steps in the method according to any one of claims 1 to 11.

15. A computer program product, characterized in that The method comprises a computer program, which, when executed by one or more processors, causes the one or more processors to execute the steps of the method according to any one of claims 1 to 11.

Images