File reading method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a file reading method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the steps of obtaining an initial file reading request for a target file, and searching the target file in a local cache based on the initial file reading request; when the target file does not exist in the local cache, establishing connection with an object storage layer in the distributed storage system, and extracting a storage pool position of the target file in the distributed storage system from the initial file reading request; determining a disk position of the target file in an object storage layer of the distributed storage system based on the storage pool position, and generating a target file reading request based on the disk position; calling an asynchronous reading interface corresponding to the object storage layer, and forwarding the target file reading request to the object storage layer; and obtaining a target file returned by the object storage layer, and sending the target file to a request end corresponding to the file reading request. By adopting the method, the file reading efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of distributed storage systems, and in particular to a file reading method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] Traditional video storage solutions generally use centralized storage management, but the scalability of centralized storage systems is limited. Increasing storage capacity often requires the purchase of additional hardware, which is not only costly but also has a long deployment cycle. In cloud business scenarios, distributed storage systems can be used for data access. However, data access is usually performed using several access interfaces such as the object gateway (RADOSGW), file system (CephFS), and block device (RBD) provided by the distributed storage system. When using one or more of these access interfaces to read files, the corresponding interface usage method is required to call the access interface, resulting in low file reading efficiency. Summary of the invention

[0003] Based on this, it is necessary to provide a file reading method, device, computer equipment, computer-readable storage medium and computer program product that can improve file reading efficiency in response to the above technical problems.

[0004] In a first aspect, the present application provides a file reading method, comprising:

[0005] Obtaining an initial file read request for a target file, and searching for the target file in a local cache based on the initial file read request;

[0006] When the target file does not exist in the local cache, a connection is established with the object storage layer in the distributed storage system, and a storage pool location of the target file in the distributed storage system is extracted from the initial file read request;

[0007] Determine the disk location of the target file in the object storage layer of the distributed storage system based on the storage pool location, and generate a target file read request based on the disk location;

[0008] Call the asynchronous read interface corresponding to the object storage layer and forward the target file read request to the object storage layer;

[0009] Get the target file returned by the object storage layer, and send the target file to the request end corresponding to the file read request.

[0010] In a second aspect, the present application also provides a file reading device, comprising:

[0011] A file search module, used to obtain an initial file read request for a target file, and search for the target file in a local cache based on the initial file read request;

[0012] A connection module, used for establishing a connection with the object storage layer in the distributed storage system when the target file does not exist in the local cache, and extracting the storage pool location of the target file in the distributed storage system from the initial file read request;

[0013] A request generation module, used to determine the disk location of the target file in the object storage layer of the distributed storage system based on the storage pool location, and generate a target file read request based on the disk location;

[0014] The interface calling module is used to call the asynchronous reading interface corresponding to the object storage layer and forward the target file reading request to the object storage layer;

[0015] The file sending module is used to obtain the target file returned by the object storage layer and send the target file to the request end corresponding to the file reading request.

[0016] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0017] Obtaining an initial file read request for a target file, and searching for the target file in a local cache based on the initial file read request;

[0018] When the target file does not exist in the local cache, a connection is established with the object storage layer in the distributed storage system, and a storage pool location of the target file in the distributed storage system is extracted from the initial file read request;

[0019] Determine the disk location of the target file in the object storage layer of the distributed storage system based on the storage pool location, and generate a target file read request based on the disk location;

[0020] Call the asynchronous read interface corresponding to the object storage layer and forward the target file read request to the object storage layer;

[0021] Get the target file returned by the object storage layer, and send the target file to the request end corresponding to the file read request.

[0022] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0023] Obtaining an initial file read request for a target file, and searching for the target file in a local cache based on the initial file read request;

[0024] When the target file does not exist in the local cache, a connection is established with the object storage layer in the distributed storage system, and a storage pool location of the target file in the distributed storage system is extracted from the initial file read request;

[0025] Determine the disk location of the target file in the object storage layer of the distributed storage system based on the storage pool location, and generate a target file read request based on the disk location;

[0026] Call the asynchronous read interface corresponding to the object storage layer and forward the target file read request to the object storage layer;

[0027] Get the target file returned by the object storage layer, and send the target file to the request end corresponding to the file read request.

[0028] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0029] Obtaining an initial file read request for a target file, and searching for the target file in a local cache based on the initial file read request;

[0030] When the target file does not exist in the local cache, a connection is established with the object storage layer in the distributed storage system, and a storage pool location of the target file in the distributed storage system is extracted from the initial file read request;

[0031] Determine the disk location of the target file in the object storage layer of the distributed storage system based on the storage pool location, and generate a target file read request based on the disk location;

[0032] Call the asynchronous read interface corresponding to the object storage layer and forward the target file read request to the object storage layer;

[0033] Get the target file returned by the object storage layer, and send the target file to the request end corresponding to the file read request.

[0034] The above-mentioned file reading method, apparatus, computer equipment, computer-readable storage medium and computer program product establish a connection with the object storage layer in the distributed storage system when the target file does not exist in the local cache, determine the disk position of the target file in the object storage layer of the distributed storage system according to the storage pool position of the target file in the distributed storage system in the initial file reading request, generate a target file reading request according to the disk position, forward the target file reading request to the object storage layer through the abnormal reading interface corresponding to the object storage layer, and obtain the target file returned by the object storage layer. It can realize direct interaction between the service gateway and the object storage layer in the distributed storage system without forwarding requests to the object storage layer through intermediate layers such as object gateways, thereby improving the file reading efficiency of the object storage layer in the distributed storage system, reducing the resource consumption of the request forwarding link of the intermediate layer, and reducing the maintenance of the intermediate layer, thereby simplifying the resource maintenance of the distributed storage system; further, by using the asynchronous reading interface of the object storage layer to read the target file, interface blocking can be avoided, thereby improving the file reading efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 An application environment diagram of a file reading method in an embodiment;

[0037] Figure 2 A schematic diagram of a flow chart of a file reading method in an embodiment;

[0038] Figure 3 A block diagram of the video content center architecture in one embodiment;

[0039] Figure 4 is a timing diagram of a file read request process in one embodiment;

[0040] Figure 5 is a structural block diagram of a file reading device in an embodiment;

[0041] Figure 6 A schematic diagram of using librados in an application in an embodiment;

[0042] Figure 7 is an internal structure diagram of a computer device in one embodiment;

[0043] Figure 8 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0045] The file reading method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the distributed storage system 104 communicates with the terminal 102 externally through the service gateway. The distributed storage system 104 is integrated with a data storage system for storing data that the terminal 102 needs to process. In an exemplary embodiment, the distributed storage system 104 obtains the initial file read request of the terminal 102 for the target file through the service gateway, and searches for the target file in the local cache based on the initial file read request; when the target file does not exist in the local cache, the service gateway extracts the storage pool position of the target file in the distributed storage system from the initial file read request, and establishes a connection with the object storage layer in the distributed storage system; the service gateway determines the disk position of the target file in the object storage layer of the distributed storage system based on the storage pool position, and generates a target file read request based on the disk position; the service gateway calls the asynchronous read interface corresponding to the object storage layer, and forwards the target file read request to the object storage layer; the service gateway obtains the target file returned by the object storage layer, and sends the target file to the request terminal 102 corresponding to the file read request. The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart car devices, projection devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. Head-mounted devices may be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The distributed storage system 104 may be a server cluster or a distributed system consisting of multiple physical servers.

[0046] In an exemplary embodiment, Figure 2 As shown, a file reading method is provided, which is applied to Figure 1 The following is an example of the service gateway provided by the distributed storage system:

[0047] Step 202: Obtain an initial file read request for a target file, and search for the target file in a local cache based on the initial file read request.

[0048] Among them, the target file refers to the file that the requesting end currently requests from the distributed storage system. A distributed storage system refers to a system with a data distributed storage architecture that can store data on multiple computers or servers. The distributed storage system can refer to ceph (distributed file system). The distributed storage system interacts with external devices through the service gateway provided to the outside. The service gateway refers to a single unified access entry provided by the distributed storage system, which is used for external devices to send requests to the service gateway. The service gateway provides request forwarding to the distributed storage system and returns the requested data to the external device. The initial file read request refers to a file read request sent by an external device received by the service gateway. The local cache refers to the storage space for cacheable files set inside the service gateway.

[0049] Exemplarily, the service gateway provided by the distributed storage system receives the initial file read request sent by the terminal, extracts the file information corresponding to the target file from the initial file read request, such as the file name, file type, time information, etc., and searches the local cache for the target file based on the file information. If it exists, the target file is obtained from the local cache and returned to the request end corresponding to the initial file read request; if it does not exist, the target file is requested from the distributed storage system, and the target file is returned to the request end corresponding to the initial file read request.

[0050] Step 204: When the target file does not exist in the local cache, a connection is established with the object storage layer in the distributed storage system, and the storage pool location of the target file in the distributed storage system is extracted from the initial file read request.

[0051] The storage pool refers to a specific storage space in a distributed storage system for storing data. The storage pool is deployed in the object storage layer. The storage pool location refers to the storage location of the target file in the storage pool of the distributed storage system. The object storage layer refers to the core component for implementing data storage in the distributed storage system. It can be a RADOS cluster (Reliable Autonomic Distributed Object Store) in ceph (distributed file system), which consists of nodes of multiple storage devices. Data in the RADOS cluster is stored in the form of objects.

[0052] Exemplarily, when the service gateway fails to find the target file in the local cache, it obtains the cluster address of the object storage layer in the distributed storage system, such as the cluster name of the RADOS cluster. The service gateway connects to the object storage layer based on the cluster address, and then extracts the storage pool location of the target sub-file in the distributed storage system from the initial file read request, such as the storage pool name corresponding to the target file.

[0053] In an exemplary embodiment, the corresponding terminal access rules can be determined based on the storage pool locations of each file in the distributed storage system, and the initial file read request sent by the terminal carries the storage pool location corresponding to the target file generated in the format of the terminal access rule. The service gateway restores the storage pool location in the format corresponding to the terminal access rule in the initial file read request to obtain the restored storage pool location of the target file in the distributed storage system.

[0054] Step 206: determine the disk location of the target file in the object storage layer of the distributed storage system based on the storage pool location, and generate a target file read request based on the disk location.

[0055] The disk location refers to the location of the disk that actually stores the target file in the storage pool corresponding to the storage pool location. The disk location can refer to an OSD (Object storage device, storage node) in a RADOS cluster, including hardware such as processors, memory, and hard disks. A target file read request refers to a read request for a target file that is sent by the service gateway to the object storage layer and is recognizable by the object storage layer.

[0056] Exemplarily, the service gateway also obtains the object name of the target file when it is stored in the object storage layer, and reproduces the disk location of the target file in the object storage layer of the distributed storage system according to the storage pool location and the object name in accordance with the data location storage rule. Then, the initial file read request is converted into a target file read request according to the disk location, which can be to combine the disk location with the initial file read request and convert it into a target file read request in a format corresponding to the object storage layer.

[0057] Step 208: Call the asynchronous read interface corresponding to the object storage layer to forward the target file read request to the object storage layer.

[0058] Step 210: Obtain the target file returned by the object storage layer, and send the target file to the request end corresponding to the file read request.

[0059] The asynchronous read interface refers to an interface provided by the object storage layer that can perform asynchronous read operations.

[0060] Exemplarily, after receiving the target file read request, the service gateway calls the asynchronous read interface provided by the object storage layer, sends the target file read request to the asynchronous read interface, responds to the target file read request through the asynchronous read interface, and obtains the target file from the object storage layer according to the disk location. The service gateway obtains the target file returned by the asynchronous read interface and returns the target file to the request end corresponding to the initial file read request.

[0061] In the above-mentioned file reading method, when the target file does not exist in the local cache, a connection with the object storage layer in the distributed storage system is established according to the storage pool position of the target file in the distributed storage system in the initial file reading request, and a target file reading request is generated according to the disk position of the target file in the object storage layer of the distributed storage system, and the target file reading request is forwarded to the object storage layer through the abnormal reading interface corresponding to the object storage layer, and the target file returned by the object storage layer is obtained, so that direct interaction between the service gateway and the object storage layer in the distributed storage system can be realized without forwarding requests to the object storage layer through intermediate layers such as object gateways, thereby improving the file reading efficiency of the object storage layer in the distributed storage system, reducing the resource consumption of the request forwarding link of the intermediate layer, reducing the maintenance of the intermediate layer, and simplifying the resource maintenance of the distributed storage system; further, by using the asynchronous reading interface of the object storage layer to read the target file, interface blocking can be avoided, thereby improving the file reading efficiency.

[0062] In an exemplary embodiment, step 204, establishing a connection with an object storage layer in a distributed storage system, includes:

[0063] Get the cluster address corresponding to the distributed storage system, and create a connection handle corresponding to the object storage layer based on the cluster address;

[0064] Obtaining preset configuration parameters, and establishing a connection with the object storage layer in the distributed storage system based on the preset configuration parameters and the connection handle;

[0065] The file reading method also includes:

[0066] After the connection is established, the file status corresponding to the target file is obtained based on the storage pool location and the object identifier corresponding to the target file;

[0067] Allocate target storage space corresponding to the target file in the local cache based on the file status.

[0068] The connection handle refers to the connection tool used by the service gateway for communication connection. The preset configuration parameters refer to the information related to the object storage layer configured for the connection handle, which is used to establish a connection with the object storage layer. The file status refers to the information describing the file, including the file size, file format, etc.

[0069] Exemplarily, the service gateway needs to pass identity authentication before it can communicate with the object storage layer. The service gateway uses the key ring file for identity authentication. After the identity authentication is passed, it obtains the cluster address corresponding to the object storage layer, such as the cluster name of the RADOS cluster, initializes the connection with the object storage layer according to the cluster address, and creates a connection handle corresponding to the object storage layer according to the initialized connection. Then, it obtains the preset configuration parameters, configures the connection handle with the preset configuration parameters, and uses the configured connection handle to establish a connection with the object storage layer in the distributed storage system. The service gateway can call the asynchronous read interface corresponding to the object storage layer through the connection with the object storage layer.

[0070] After establishing the connection (or before calling the asynchronous read interface), the service gateway creates an IO context based on the storage pool location (i.e., storage pool name) and object identifier (i.e., object name) corresponding to the target file. The IO context refers to the environment and state of the CPU, process, thread, or task when performing IO (input / output) operations, which is used to indicate the context information that needs to be obtained. The service gateway obtains the file status corresponding to the target file based on the IO context, including file size, file format, etc., and allocates the target storage space corresponding to the target file in the local cache based on the file status, such as allocating storage space corresponding to the file size, or allocating storage space corresponding to the file format. After obtaining the target file through the asynchronous read interface, the service gateway can temporarily store the target file in the target storage space, then obtain the target file from the target storage space, and send it to the request end corresponding to the initial file read request. After the target file is successfully read, the service gateway releases the resources of the target storage space and disconnects from the object storage layer.

[0071] In this embodiment, the service gateway can reduce unnecessary resources of request forwarding by the object gateway and other intermediate layers, as well as maintenance of the intermediate layers, by directly interacting with the object storage layer, thereby simplifying the operation and maintenance of the distributed storage system.

[0072] In an exemplary embodiment, step 206, determining the disk location of the target file in the object storage layer of the distributed storage system based on the storage pool location, includes:

[0073] Perform hash calculation on the object identifier corresponding to the target file to obtain a hash object identifier;

[0074] Determine a target data group identifier corresponding to the target file based on the hash object identifier and the storage pool information corresponding to the storage pool location;

[0075] The data access rule parameters corresponding to the object storage layer are obtained, and based on the target data group identifier and the data access rule parameters, the disk location of the target file in the object storage layer of the distributed storage system is determined.

[0076] The target group identifier refers to the identifier of the data group where the target file is located in each data group obtained by grouping and storing data in the storage pool. The data access rule parameter refers to the operation logic for calculating the corresponding storage position of data in the storage pool.

[0077] Exemplarily, in a distributed storage system, files are stored in a storage pool of an object storage layer, and each storage pool includes multiple data groups. The service gateway performs a hash calculation on the object identifier corresponding to the target file, and the obtained hash value is used as the hash object identifier, and then obtains the storage pool information corresponding to the storage pool location, which is generally the number of groups of each data group in the storage pool corresponding to the storage pool location. The hash object identifier is modulo-operated with the number of groups in the storage pool information to obtain the target data group identifier corresponding to the target data group in the storage pool where the target file is located. Then, the data access rule parameters corresponding to the object storage layer are obtained. The data access rule parameters can be a tree structure describing physical hardware resources and data storage rules in the distributed storage system, which includes the topological structure of the RADOS cluster corresponding to the object storage layer, the weight of the OSD, and the data location storage rules. The service gateway determines the OSD (storage node) mapped to the target data group identifier corresponding to the target file according to the data access rule parameters, and determines the disk location of the target file in the object storage layer of the distributed storage system according to the OSD.

[0078] In this embodiment, the target data grouping identifier corresponding to the target file is determined, and the disk location corresponding to the target file is calculated according to the target data grouping identifier and the data access rule parameters, thereby ensuring the accuracy of the disk location.

[0079] In an exemplary embodiment, before obtaining an initial file reading request for a target file in step 202, the file reading method further includes:

[0080] Create an initial event processing framework corresponding to the service gateway, configure the initial event processing framework based on the cluster information corresponding to the distributed storage system, and obtain the target event processing framework;

[0081] Edit the processing function corresponding to the asynchronous read interface of the target event processing framework to obtain an event-driven model;

[0082] A service gateway is obtained based on an event-driven model.

[0083] The initial event processing framework is an event-driven mechanism for processing input / output (I / O) operations. Cluster information refers to the configuration information related to the object storage layer, which is used to configure the initial event processing framework. The target event processing framework refers to the event processing mechanism dedicated to the object storage layer. The processing function refers to the callback logic function defined for the asynchronous read interface. The event-driven model refers to the event processing mechanism that can be used for the configured object storage layer.

[0084] Exemplarily, the service gateway obtains the cluster information corresponding to the distributed storage system, including the cluster ID of the distributed storage system, the monitor (core node, used to maintain cluster status information, manage cluster configuration, handle cluster failures, etc.) address, port, and cephx protocol key ring, etc. The service gateway creates an initial event processing framework corresponding to the service gateway, and uses the cluster information to configure the parameters of the initial event processing framework, which may be a global configuration of the processing method of the initial event processing framework, to obtain the target event processing framework.

[0085] Edit the processing function corresponding to the asynchronous read interface of the target event processing framework, including calling the asynchronous I / O interface of the distributed storage system to read files, the callback function and event loop of the asynchronous I / O interface to ensure that the event-driven model is not blocked, obtain the event-driven model, and then create a service gateway based on the event-driven model.

[0086] In an exemplary embodiment, the event-driven model uses non-blocking I / O to process network connections and uses multiplexing technology (epoll) to monitor multiple file descriptors. When a file descriptor is ready, the event-driven model immediately responds and processes it. This can be done through an event loop mechanism. When different events are detected, the corresponding processing function is called to respond and process.

[0087] In an exemplary embodiment, the target event processing framework also includes a method for processing error scenarios that occur during the editing of file reading, such as connection failure or timeout, read timeout or failure, etc., and returns an error code to the client to identify the error type of the current request. The target event processing framework also includes a method for managing the connection of the RADOS cluster in the distributed storage system and the life cycle of the asynchronous read object to prevent memory leaks and resource waste.

[0088] In this embodiment, a target event processing framework is created to achieve direct interaction between the service gateway and the object storage layer, and the target file can be quickly read through the asynchronous reading interface without blocking the event model, thereby improving file reading efficiency.

[0089] In an exemplary embodiment, the event-driven model registers a read event corresponding to a target file read request; step 210, obtaining the target file returned by the object storage layer, and sending the target file to the request end corresponding to the file read request, includes:

[0090] After receiving the file receiving instruction returned by the object storage layer, the read event in the event-driven model is triggered to read the target file;

[0091] The target file is stored in the target storage space corresponding to the target file, and the file content corresponding to the target file is returned from the target storage space to the request end corresponding to the file reading request.

[0092] Exemplarily, after forwarding the target file read request to the object storage layer of the distributed storage system, the service gateway registers the read event corresponding to the target file read request in the event-driven model, and is used to trigger the read event to read the content of the target file after receiving the target file returned by the object storage layer.

[0093] After receiving the file receiving instruction returned by the object storage layer, the service gateway triggers the read event in the event-driven model, obtains the target file through the asynchronous read interface, and reads the file content corresponding to the target file. The target file and the corresponding file content are stored in the target storage space, and the file content corresponding to the target file is returned from the target storage space to the request end corresponding to the file reading request. After the target file is received, the local cache is updated according to the target file, and the next file reading request from the request end is waited for.

[0094] In this embodiment, the asynchronous read interface is called by triggering a read event to read the target file, thereby ensuring that the event-driven model is not blocked and ensuring the file reading efficiency.

[0095] In an exemplary embodiment, the local cache includes a first cache and a second cache; and the file reading method further includes:

[0096] Identify a file tag corresponding to a target file, and when the file tag is a hot tag, search for the target file in a first cache;

[0097] When the target file does not exist in the first cache, searching for the target file in the second cache based on time information corresponding to the target file;

[0098] When the target file does not exist in the second cache, the step of extracting the storage pool location of the target file in the distributed storage system from the initial file read request is performed until the target file is acquired.

[0099] Among them, the first cache refers to the storage space in the service gateway for storing hot files. Hot files refer to files with a reading frequency greater than a preset threshold. The second cache is the storage space in the service gateway for storing files for a certain period of time. The file tag refers to a tag used to indicate whether it is a hot file.

[0100] Exemplarily, the first cache may be shared content in the service gateway with a fast data reading speed but a small capacity; the second cache may be a local disk in the service gateway with a slow data reading speed but a large capacity.

[0101] The service gateway obtains the file tag corresponding to the target file from the initial file read request, and identifies whether the file tag corresponding to the target file is a hot tag. If it is a hot tag, the target file is searched in the first cache. If the target file exists in the first cache, the target file is obtained from the first cache and sent to the request end corresponding to the initial file read request; if it is not a hot tag or the target file does not exist in the first cache, the file information corresponding to the target file is obtained, such as the write time of the target file to the distributed storage system, and the target file is searched in the second cache according to the file information. If the target file exists in the second cache, the target file is obtained from the second cache and sent to the request end corresponding to the initial file read request; if the target file does not exist in the second cache, the step of extracting the storage pool location of the target file in the distributed storage system from the initial file read request is executed until the target file is obtained.

[0102] In an exemplary embodiment, a file expiration time is set for the first cache and the second cache. When the storage time of an expired file among the files stored in the first cache and the second cache exceeds the file expiration time, the expired file is deleted.

[0103] In this embodiment, by setting the first cache and the second cache, the data storage capacity of the cache can be increased, and existing files can be quickly read from the first cache and the second cache without requesting files from the distributed storage system, thereby improving file reading efficiency.

[0104] In an exemplary embodiment, Figure 3As shown, a video content center architecture block diagram is provided. The target file may be a target video file, and the distributed storage system serves as a video content center for storing video files. The service gateway may refer to an nginx (enginex, a high-performance HTTP and reverse proxy web server) cluster. LVS (Linux Virtual Server, virtual server technology) is used in front of the nginx cluster to achieve load balancing for received user requests. The nginx cluster communicates with the video content center (ceph distributed storage system) to forward user requests to the video content center, read the video file corresponding to the user request from the video content center, and return it to the client.

[0105] In an exemplary embodiment, Figure 4 As shown, a timing diagram of a file reading request process is provided. Among them, the service gateway may refer to an nginx (enginex, a high-performance HTTP and reverse proxy web server) server. The process of a terminal reading a file from a distributed storage system includes the following steps:

[0106] After the user request (initial file read request) is forwarded to the nginx cluster, the user request is authenticated and other routine verifications are performed. The user request can be an HTTP request.

[0107] After the user request is verified by the nginx server, the nginx server converts the http request into a file read request, forwards the request to the backend (i.e., the object storage layer) according to the configured Ceph (distributed storage system) cluster address, and uses the librados library interface to read the target file from the Ceph cluster.

[0108] Use the asynchronous interface provided by librados (the interface layer for accessing the object storage layer) to read the target file. When the target file content is returned, trigger the read event of the asynchronous event model (event-driven model) to notify nginx to read the target file. Then, nginx returns the file content of the target file to the client based on the write event generated by the asynchronous event model.

[0109] In an exemplary embodiment, taking access to a video content center as an example, the forwarding and conversion process of a request is described:

[0110] Use the asynchronous write interface of the librados library to write files to the distributed storage system. Determine the corresponding client access rules based on the storage location when the file is written. For example, the storage location of the file in the Ceph cluster is: storage pool name test_pool, object name test_object, and the client access URL (initial file read request) can be set to / test_pool / test_object?auth_key=xxx; the storage pool can be created using erasure codes to improve the utilization of the storage server disk;

[0111] Nginx authenticates the authentication parameters carried in the URL. If the authentication fails, it directly returns the error code 403. If the authentication passes, it continues to execute.

[0112] Determine whether the file exists in the memory (first cache) and has not expired. If it does, return the file content directly. Otherwise, search in the local disk (second cache).

[0113] Determine whether the file exists in the local disk and has not expired. If it does, return the file content directly. Otherwise, request the file directly from the Ceph cluster.

[0114] According to the definition rules of URI, the storage pool name and object name used by the restored file are restored. According to the monitor address and client key ring provided by the Ceph cluster, the request is forwarded to the Ceph cluster through the upstream module of nginx, and the read event is registered, waiting for the Ceph cluster to return the file content;

[0115] When the Ceph cluster returns the file, it triggers the read event of the nginx epoll event model (event-driven model), and nginx starts reading the file content and responds to the client through streaming.

[0116] After the file is received, the cache in the local disk is updated, and the next access by the client is waited for, and the above steps are repeated.

[0117] In an exemplary embodiment, in order to further improve the access rate of the video content center, Ceph's object gateway (object storage interface, RADOSGW), block storage interface, file system interface, etc. are not used for forwarding again, but the underlying interface of librados is directly used to interact with the Ceph storage cluster, as shown in the following example. Figure 5 An application shown uses the librados schematic, and the application can directly access the RADOS cluster to achieve direct interaction with the Ceph storage cluster.

[0118] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0119] Based on the same inventive concept, the embodiment of the present application also provides a file reading device for implementing the file reading method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more file reading device embodiments provided below can refer to the limitations on the file reading method above, and will not be repeated here.

[0120] In an exemplary embodiment, Figure 6 As shown, a file reading device 600 is provided, comprising: a file search module 602, a connection module 604, a request generation module 606, an interface calling module 608 and a file sending module 610, wherein:

[0121] A file search module 602, configured to obtain an initial file read request for a target file, and search for the target file in a local cache based on the initial file read request;

[0122] A connection module 604 is used to establish a connection with the object storage layer in the distributed storage system when the target file does not exist in the local cache, and extract the storage pool location of the target file in the distributed storage system from the initial file read request;

[0123] A request generation module 606, configured to determine a disk location of a target file in an object storage layer of a distributed storage system based on a storage pool location, and to generate a target file read request based on the disk location;

[0124] The interface calling module 608 is used to call the asynchronous reading interface corresponding to the object storage layer and forward the target file reading request to the object storage layer;

[0125] The file sending module 610 is used to obtain the target file returned by the object storage layer, and send the target file to the request end corresponding to the file reading request.

[0126] In an exemplary embodiment, the connection module 604 is also used to obtain a cluster address corresponding to the distributed storage system, create a connection handle corresponding to the object storage layer based on the cluster address; obtain preset configuration parameters, and establish a connection with the object storage layer in the distributed storage system based on the preset configuration parameters and the connection handle; after the connection is established, obtain the file status corresponding to the target file based on the storage pool location and the object identifier corresponding to the target file; and allocate the target storage space corresponding to the target file in the local cache based on the file status.

[0127] In an exemplary embodiment, the request generation module 606 is also used to perform a hash calculation on the object identifier corresponding to the target file to obtain a hash object identifier; determine the target data group identifier corresponding to the target file based on the hash object identifier and the storage pool information corresponding to the storage pool location; obtain the data access rule parameters corresponding to the object storage layer, and determine the disk location of the target file in the object storage layer of the distributed storage system based on the target data group identifier and the data access rule parameters.

[0128] In an exemplary embodiment, the file reading device 600 is also used to create an initial event processing framework corresponding to the service gateway, configure the initial event processing framework based on the cluster information corresponding to the distributed storage system, and obtain the target event processing framework; edit the processing function corresponding to the asynchronous read interface of the target event processing framework to obtain an event-driven model; and obtain the service gateway based on the event-driven model.

[0129] In an exemplary embodiment, the event-driven model registers a read event corresponding to a target file read request; the file sending module 610 is also used to trigger the read event in the event-driven model and read the target file after receiving a file receiving instruction returned by the object storage layer; the target file is stored in the target storage space corresponding to the target file, and the file content corresponding to the target file is returned from the target storage space to the request end corresponding to the file read request.

[0130] In an exemplary embodiment, the local cache includes a first cache and a second cache; the file reading device 600 is also used to identify a file tag corresponding to a target file, and when the file tag is a hot tag, search for the target file in the first cache; when the target file does not exist in the first cache, search for the target file in the second cache based on the time information corresponding to the target file; when the target file does not exist in the second cache, execute the step of extracting the storage pool location of the target file in the distributed storage system from the initial file reading request until the target file is obtained.

[0131] Each module in the above-mentioned file reading device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.

[0132] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 7 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store file data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a file reading method is implemented.

[0133] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 8As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a file reading method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0134] Those skilled in the art will understand that Figure 7-Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0135] In an exemplary embodiment, in one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0136] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0137] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0138] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0139] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0140] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0141] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A file reading method, characterized in that: Applied to a service gateway provided externally by a distributed storage system, the method comprises: Obtaining an initial file read request for a target file, and searching for the target file in a local cache based on the initial file read request; When the target file does not exist in the local cache, establishing a connection with the object storage layer in the distributed storage system, and extracting the storage pool location of the target file in the distributed storage system from the initial file read request; Determine the disk location of the target file in the object storage layer of the distributed storage system based on the storage pool location, and generate a target file read request based on the disk location; Calling the asynchronous read interface corresponding to the object storage layer to forward the target file read request to the object storage layer; The target file returned by the object storage layer is obtained, and the target file is sent to the request end corresponding to the file reading request.

2. The method according to claim 1, characterized in that The establishing of a connection with the object storage layer in the distributed storage system includes: Obtaining a cluster address corresponding to the distributed storage system, and creating a connection handle corresponding to the object storage layer based on the cluster address; Acquire preset configuration parameters, and establish a connection with the object storage layer in the distributed storage system based on the preset configuration parameters and the connection handle; The method further comprises: After the connection is established, based on the storage pool location and the object identifier corresponding to the target file, a file status corresponding to the target file is obtained; A target storage space corresponding to the target file is allocated in the local cache based on the file status.

3. The method according to claim 1, characterized in that The determining the disk location of the target file in the object storage layer of the distributed storage system based on the storage pool location includes: Performing hash calculation on the object identifier corresponding to the target file to obtain a hash object identifier; Determine a target data group identifier corresponding to the target file based on the hash object identifier and the storage pool information corresponding to the storage pool location; The data access rule parameters corresponding to the object storage layer are obtained, and based on the target data group identifier and the data access rule parameters, the disk location of the target file in the object storage layer of the distributed storage system is determined.

4. The method according to claim 1, characterized in that: Before obtaining the initial file read request for the target file, the method further includes: Creating an initial event processing framework corresponding to the service gateway, configuring the initial event processing framework based on cluster information corresponding to the distributed storage system, and obtaining a target event processing framework; Editing the processing function corresponding to the asynchronous read interface in the target event processing framework to obtain an event-driven model; The service gateway is obtained based on the event-driven model.

5. The method according to claim 4, characterized in that The event-driven model registers a read event corresponding to the target file read request; The obtaining the target file returned by the object storage layer and sending the target file to the request end corresponding to the file reading request includes: After receiving the file receiving instruction returned by the object storage layer, triggering the read event in the event-driven model to read the target file; The target file is stored in a target storage space corresponding to the target file, and the file content corresponding to the target file is returned from the target storage space to a request end corresponding to the file reading request.

6. The method according to claim 1, characterized in that The local cache includes a first cache and a second cache; the method further includes: Identify a file tag corresponding to the target file, and when the file tag is a hotspot tag, search for the target file in the first cache; When the target file does not exist in the first cache, searching the second cache for the target file based on time information corresponding to the target file; When the target file does not exist in the second cache, the step of extracting the storage pool location of the target file in the distributed storage system from the initial file read request is performed until the target file is acquired.

7. A file reading device, characterized in that: The device comprises: A file search module, used for obtaining an initial file read request for a target file, and searching for the target file in a local cache based on the initial file read request; A connection module, configured to establish a connection with the object storage layer in the distributed storage system when the target file does not exist in the local cache, and extract the storage pool location of the target file in the distributed storage system from the initial file read request; A request generation module, configured to determine a disk location of the target file in the object storage layer of the distributed storage system based on the storage pool location, and generate a target file read request based on the disk location; An interface calling module, used to call the asynchronous reading interface corresponding to the object storage layer, and forward the target file reading request to the object storage layer; The file sending module is used to obtain the target file returned by the object storage layer, and send the target file to the request end corresponding to the file reading request.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images