File Caching Method and Device Based on Content Delivery Network Caching File System

Abstract

This application relates to a file caching method and device based on a content delivery network cache file system. The method is based on the content delivery network cache file system, which includes a logic layer and a storage layer. The storage layer includes multiple key-value pair storage systems. The file caching method includes: obtaining a source file to be cached; splitting the source file to be cached into multiple shard storage data based on the logic layer in the content delivery network cache file system, and storing each shard storage data in multiple key-value pair storage systems in the storage layer. The present invention enables the cache to be written and read quickly, speeds up the timeliness in the data transmission process, reduces the disk load, improves the network transmission speed, reduces the disk overhead cost, and alleviates the pressure on system maintenance by operation and maintenance.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a file caching method and device based on a content distribution network cache file system. Background Art

[0002] The file system is an indispensable part of every computer device. For an ordinary file system, such as the ext2, ext3, and ext4 file systems that come with Linux, an entire file is usually saved. For a cache system, we often encounter the need to save part of the file, read part of the file, and determine whether a certain section of the file is cached, which are all things that file systems do not have.

[0003] Moreover, the file system is characterized by its ability to quickly locate the requested content and its location, which greatly improves the file query speed. Since it is a self-developed file system, every sector of the disk is fully utilized, avoiding space waste and saving operation and maintenance costs. The efficiency of writing large files is too slow because data is written to a single disk, and single file and multi-point storage cannot be achieved. When reading at high concurrency, the disk IO pressure is huge, resulting in slower disk read and write speeds.

[0004] The existing technology uses existing file systems, such as ext2, ext3 and other Linux file systems. However, such file systems cannot save partial files. However, for a content distribution network, it is often necessary to obtain partial files. Therefore, the traditional file system must obtain all files and then intercept the contents of partial files. This method will cause the back-to-source to be amplified. Back-to-source amplification means that the ratio of the flow of the source station responding to the content distribution network and the flow of the content distribution network corresponding to the client is amplified.

[0005] Originally, the content distribution network was to help the source station bear most of the traffic, but this type of file system will greatly increase the traffic pressure on the source station. It can be seen that this type of traditional file system is not suitable for the cache system of the content distribution network. In addition, for large files, the traditional file system will be very slow to write and read, and there are problems such as slow cache writing and reading speed, slow data transmission and low disk load. Summary of the invention

[0006] Based on this, it is necessary to provide a file caching method and device based on a content distribution network cache file system, which can realize fast writing and reading of cache, speed up the timeliness of data transmission, reduce disk load, improve network transmission speed, reduce disk overhead cost, and reduce the pressure of operation and maintenance on system maintenance.

[0007] The technical solution of the present invention is as follows:

[0008] A file caching method based on a content delivery network cache file system, the method being based on a content delivery network cache file system, the content delivery network cache file system including a logic layer and a storage layer, the storage layer including a plurality of key-value pair storage systems; the file caching method includes:

[0009] Step S100: Obtain the source file to be cached;

[0010] Step S200: Based on the logic layer in the content delivery network cache file system, split the source file to be cached into multiple shard storage data, and store each shard storage data in a plurality of key-value pair storage systems in the storage layer.

[0011] Specifically, the source file to be cached includes the complete source file to be cached;

[0012] Step S200: Based on the logic layer in the content delivery network cache file system, split the source file to be cached into multiple shard storage data, and store each shard storage data in a plurality of key-value pair storage systems in the storage layer; specifically including:

[0013] Step S211: Based on the logic layer in the content delivery network cache file system, generate a plurality of complete source file shard objects according to the complete source file to be cached;

[0014] Step S212: Encode and convert each complete source file shard object to generate a complete source file byte code, wherein one complete source file shard object corresponds to one complete source file byte code;

[0015] Step S213: Use the complete source file shard object as the key and the complete source file byte code as the value, and store each complete source file shard object in each key-value pair storage system in the storage layer;

[0016] Step S214: Based on the logic layer in the content delivery network cache file system, generate a complete source file metadata object according to the complete source file to be cached, wherein the key of the complete source file metadata object is the requested URL;

[0017] Step S215: Convert the complete source file metadata object into a complete source file metadata byte code according to the complete source file metadata object;

[0018] Step S216: Using each of the complete source file metadata objects as the key and the complete source file metadata bytecode as the value, store each of the complete source file metadata objects in each of the key-value pair storage systems in the storage layer.

[0019] Specifically, the source file to be cached includes an incomplete source file to be cached; after being segmented, the incomplete source file to be cached forms multiple target incomplete source file shard objects;

[0020] Step S200: Based on the logic layer in the content delivery network cache file system, segment the source file to be cached into multiple shards of stored data, and store each shard of stored data in multiple key-value pair storage systems in the storage layer; specifically including:

[0021] Step S221: Based on the logic layer in the content delivery network cache file system, generate an incomplete source file metadata object according to the incomplete source file to be cached, where the key of the incomplete source file metadata object is the requested URL;

[0022] Step S222: Convert the incomplete source file metadata object into incomplete source file metadata bytecode according to the incomplete source file metadata object;

[0023] Step S223: Using each of the incomplete source file metadata objects as the key and the incomplete source file metadata bytecode as the value, store each of the incomplete source file metadata objects in each of the key-value pair storage systems in the storage layer;

[0024] Step S224: Based on the logic layer in the content delivery network cache file system, generate each target incomplete source file shard object according to the incomplete source file to be cached;

[0025] Step S225: Encode and transform each of the target incomplete source file shard objects to generate incomplete source file bytecode, where one target incomplete source file shard object corresponds to one incomplete source file bytecode;

[0026] Step S216: Using each of the target incomplete source file shard objects as the key and the incomplete source file bytecode as the value, store each of the target incomplete source file shard objects in each of the key-value pair storage systems in the storage layer.

[0027] Specifically, step S216: Store each of the complete source file metadata objects in each of the key-value pair storage systems in the storage layer, with the complete source file metadata object as the key and the complete source file metadata bytecode as the value; afterwards, it further includes:

[0028] Step S311: Generate a key for the complete source file metadata object to be read based on the URL of the request when reading the complete file;

[0029] Step S312: Obtain the complete source file metadata object to be read according to the key of the complete source file metadata object to be read;

[0030] Step S313: Calculate the current data cache time based on the complete source file metadata object to be read, and determine whether the cache has expired according to the current data cache time. If the determination is yes, delete the cache;

[0031] Step S314: Obtain the key of the complete source file shard object to be read according to the complete source file metadata object to be read;

[0032] Step S315: Obtain the shard data to be read according to the complete source file shard object to be read;

[0033] Step S316: Merge each of the shard data to be read and generate a complete source file to be cached after merging, and output the complete source file to be cached after merging.

[0034] Specifically, step S216: Store each of the target incomplete source file shard objects in each of the key-value pair storage systems in the storage layer, with each of the target incomplete source file shard objects as the key and the incomplete source file bytecode as the value; afterwards, it further includes:

[0035] Step S321: Generate a key for the incomplete source file metadata object to be read based on the URL of the request when reading the incomplete file;

[0036] Step S322: Obtain the incomplete source file metadata object to be read according to the key of the incomplete source file metadata object to be read;

[0037] Step S323: Calculate the current data cache time based on the incomplete source file metadata object to be read, and determine whether the cache has expired according to the current data cache time. If the determination is yes, delete the cache;

[0038] Step S324: Obtain the key of the complete source file shard object to be read according to the complete source file metadata object to be read;

[0039] Step S325: Obtain the incomplete shard data to be read according to the complete source file shard object to be read.

[0040] Step S326: Merge the incomplete shard data to be read and generate an incomplete source file to be cached after merging, and output the incomplete source file to be cached after merging.

[0041] Specifically, a file caching device based on a content delivery network cache file system, the device includes:

[0042] A source file to be cached acquisition module, configured to acquire a source file to be cached;

[0043] A storage shard storage data module, configured to split the source file to be cached into multiple shard storage data based on the logical layer in the content delivery network cache file system, and store each shard storage data in multiple key-value pair storage systems in the storage layer.

[0044] Specifically, the storage shard storage data module includes:

[0045] A shard object file module, configured to generate multiple complete source file shard objects based on the logical layer in the content delivery network cache file system according to the complete source file to be cached;

[0046] A source file bytecode conversion module, configured to encode and convert each complete source file shard object to generate a complete source file bytecode, where one complete source file shard object corresponds to one complete source file bytecode;

[0047] A storage system module, configured to store each complete source file shard object in each key-value pair storage system in the storage layer with the complete source file shard object as the key and the complete source file bytecode as the value;

[0048] A file metadata object generation module, configured to generate a complete source file metadata object based on the logical layer in the content delivery network cache file system according to the complete source file to be cached, where the key of the complete source file metadata object is the requested URL;

[0049] A metadata bytecode conversion module, configured to convert the complete source file metadata object into a complete source file metadata bytecode according to the complete source file metadata object;

[0050] A storage module, configured to store each complete source file metadata object in each key-value pair storage system in the storage layer with the complete source file metadata object as the key and the complete source file metadata bytecode as the value.

[0051] Specifically, the storage sharding storage data module includes:

[0052] An incomplete source file module, configured to generate an incomplete source file metadata object based on the logical layer in the content delivery network cache file system according to the incomplete source file to be cached, wherein the key of the incomplete source file metadata object is the requested URL;

[0053] An incomplete source file metadata bytecode conversion module, configured to convert the incomplete source file metadata object into an incomplete source file metadata bytecode according to the incomplete source file metadata object;

[0054] A key-value pair storage system module, configured to store each of the incomplete source file metadata objects in each of the key-value pair storage systems in the storage layer with the incomplete source file metadata object as the key and the incomplete source file metadata bytecode as the value;

[0055] A target incomplete source file generation module, configured to generate each target incomplete source file shard object based on the logical layer in the content delivery network cache file system according to the incomplete source file to be cached;

[0056] A source file bytecode generation module, configured to encode and convert each of the target incomplete source file shard objects into an incomplete source file bytecode, wherein one target incomplete source file shard object corresponds to one incomplete source file bytecode;

[0057] An object key storage module, configured to store each of the target incomplete source file shard objects in each of the key-value pair storage systems in the storage layer with each of the target incomplete source file shard objects as the key and the incomplete source file bytecode as the value.

[0058] A computer device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the steps of the above-mentioned file caching method based on the content delivery network cache file system are implemented.

[0059] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned file caching method based on the content delivery network cache file system are implemented.

[0060] The technical effects achieved by the present invention are as follows:

[0061] The above file caching method and device based on the content delivery network cache file system set up a content delivery network cache file system, which includes a logical layer and a storage layer. The storage layer includes multiple key-value pair storage systems. The file caching method includes: first, obtaining the source file to be cached; then, based on the logical layer in the content delivery network cache file system, splitting the source file to be cached into multiple shard storage data, and storing each shard storage data in multiple key-value pair storage systems in the storage layer, achieving fast writing and reading of the cache, accelerating the timeliness in the data transmission process, reducing the disk load, improving the network transmission speed, reducing the disk overhead cost, and alleviating the operation and maintenance pressure on system maintenance;

[0062] In addition, by making the storage layer include multiple key-value pair storage systems, the storage and reading efficiency of large files can be greatly improved, because a large file can be split into multiple shard storage data and stored in different key-value pair storage systems, and when reading, different shard data can also be read from multiple key-value pair storage systems simultaneously;

[0063] Of course, for a key-value pair storage system, a storage system that can achieve circular writing can be implemented to avoid the problem that writing cannot be performed after the disk is full. Description of the Drawings

[0064] Figure 1 It is a schematic flowchart of the file caching method based on the content delivery network cache file system in an embodiment;

[0065] Figure 2 It is a structural block diagram of the content delivery network cache file system;

[0066] Figure 3 It is a schematic diagram of the data structure of the metadata object;

[0067] Figure 4 It is an architecture diagram of the key-value pair storage system;

[0068] Figure 5 It is a structural block diagram of the file caching device based on the content delivery network cache file system in an embodiment;

[0069] Figure 6 It is an internal structure diagram of a computer device in an embodiment. Detailed Embodiments

[0070] In order to make the objectives, technical solutions and advantages of the present application clearer and more understandable, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0071] In one embodiment, as Figure 1 shown, a file caching method based on a content delivery network cache file system is provided. The method is based on a content delivery network cache file system, and the content delivery network cache file system includes a logic layer and a storage layer. The storage layer includes multiple key-value pair storage systems. Specifically, by making the storage layer include multiple key-value pair storage systems, the storage and reading efficiency of large files can be greatly improved, because a large file can be split into multiple shards and stored in different key-value pair storage systems, and when reading, different shard data can also be read from multiple key-value pair storage systems simultaneously.

[0072] Further, as Figure 2 shown, it is a structural block diagram of a content delivery network cache file system. Figure 2 Each K-V memory in it belongs to a key-value pair storage system, and each key-value pair storage system is used to store data.

[0073] For a key-value pair storage system, a storage system that can achieve circular writing can be realized, avoiding the problem that writing cannot be performed after the disk is full.

[0074] Specifically, the file caching method includes:

[0075] Step S100: Obtain the source file to be cached;

[0076] Step S200: Based on the logic layer in the content delivery network cache file system, split the source file to be cached into multiple shards of stored data, and store each shard of stored data in multiple key-value pair storage systems in the storage layer.

[0077] The present invention sets up a content delivery network cache file system, which includes a logic layer and a storage layer. The storage layer includes multiple key-value pair storage systems. The file caching method includes: in the method, first obtain the source file to be cached; then, based on the logic layer in the content delivery network cache file system, split the source file to be cached into multiple shard storage data, and store each shard storage data in multiple key-value pair storage systems in the storage layer, so as to realize that the cache can be written and read quickly, speed up the timeliness in the data transmission process, reduce the disk load, improve the network transmission speed, reduce the disk overhead cost, and relieve the pressure of operation and maintenance on system maintenance.

[0078] In one embodiment, as Figure 4 shown, it is the architecture diagram of the key-value pair storage system. The key-value pair storage system includes two layers. The upper layer is the cache system storage header, and the lower layer is the cache system storage body. Specifically, the cache system storage header contains an index table, and the index table stores the offset position and storage size corresponding to the cache system storage body where the value corresponding to the key Key is located. And in order to reduce the io overhead caused by the reading and writing of the key-value pair storage system, a memory Buffer is allocated in the memory. During the reading or writing process of the key-value pair storage system, this memory Buffer will be preferentially read or written. When the block memory Buffer is full, this memory Buffer will be written into the key-value pair storage system at the same time.

[0079] In one embodiment, the source file to be cached includes the complete source file to be cached;

[0080] Step S200: Based on the logic layer in the content delivery network cache file system, split the source file to be cached into multiple shard storage data, and store each shard storage data in multiple key-value pair storage systems in the storage layer; specifically including:

[0081] Step S211: Based on the logic layer in the content delivery network cache file system, generate multiple complete source file shard objects according to the complete source file to be cached;

[0082] Specifically, after generating multiple complete source file shard objects according to the complete source file to be cached, continue to fill the shard object content and calculate the check code.

[0083] Step S212: Encode and transform each complete source file shard object to generate a complete source file byte code, where one complete source file shard object corresponds to one complete source file byte code;

[0084] Specifically, in this step, the bytecode is msgp. Of course, it can also be other encodings, such as protobuf.

[0085] Step S213: Using the complete source file shard object as the key and the complete source file bytecode as the value, store each of the complete source file shard objects in each of the key-value pair storage systems in the storage layer;

[0086] Step S214: Based on the logic layer in the content delivery network cache file system, generate a complete source file metadata object according to the complete source file to be cached, where the key of the complete source file metadata object is the requested URL;

[0087] Specifically, all shard keys will also be saved in the complete source file metadata object.

[0088] Further, the complete source file metadata object is a metadata object, as Figure 3 shown. The complete source file metadata object includes a cache key-value Key generated through the URL. The metadata object also contains the key values of all shard data objects. In addition, the metadata object also contains some other information about this cache.

[0089] Step S215: Convert the complete source file metadata object into complete source file metadata bytecode according to the complete source file metadata object;

[0090] Step S216: Using the complete source file metadata object as the key and the complete source file metadata bytecode as the value, store each of the complete source file metadata objects in each of the key-value pair storage systems in the storage layer.

[0091] Specifically, through Step S215 and Step S216, store each of the complete source file metadata objects in each of the key-value pair storage systems in the storage layer.

[0092] In one embodiment, the source file to be cached includes an incomplete source file to be cached; the incomplete source file to be cached is divided to form multiple target incomplete source file shard objects;

[0093] Specifically, for example, if the file size is 10M, then the file from the 3M to the 5M needs to be obtained.

[0094] Step S200: Based on the logic layer in the content delivery network cache file system, the source file to be cached is split into multiple shards for data storage, and each shard of data is stored in multiple key-value pair storage systems in the storage layer; specifically including:

[0095] Step S221: Based on the logic layer in the content delivery network cache file system, an incomplete source file metadata object is generated according to the source file to be cached that is incomplete, where the key of the incomplete source file metadata object is the requested URL;

[0096] Specifically, all shard keys will also be saved in the incomplete source file metadata object.

[0097] Step S222: Convert the incomplete source file metadata object into incomplete source file metadata bytecode according to the incomplete source file metadata object;

[0098] Step S223: Using the incomplete source file metadata object as the key and the incomplete source file metadata bytecode as the value, store each incomplete source file metadata object in each key-value pair storage system in the storage layer;

[0099] Specifically, by using the incomplete source file metadata bytecode as the value, each incomplete source file metadata object is stored in each key-value pair storage system in the storage layer.

[0100] Step S224: Based on the logic layer in the content delivery network cache file system, generate each target incomplete source file shard object according to the source file to be cached that is incomplete;

[0101] Step S225: Encode and convert each target incomplete source file shard object to generate incomplete source file bytecode, where one target incomplete source file shard object corresponds to one incomplete source file bytecode.

[0102] Specifically, the generated incomplete source file bytecode realizes that one target incomplete source file shard object corresponds to one incomplete source file bytecode.

[0103] Step S216: Using each target incomplete source file shard object as the key and the incomplete source file bytecode as the value, store each target incomplete source file shard object in each key-value pair storage system in the storage layer.

[0104] Furthermore, through the above steps, one or part of the files are stored in the cache file system in a fragmented form.

[0105] In one embodiment, step S216: Using the complete source file metadata object as the key and the complete source file metadata bytecode as the value, store each of the complete source file metadata objects in each key-value pair storage system in the storage layer; after that, file reading is also included, and the specific reading steps are as follows:

[0106] Step S311: Generate the key of the complete source file metadata object to be read according to the URL of the request when reading the complete file;

[0107] Specifically, when reading the complete file, a request will be triggered, and the URL of the request will be generated.

[0108] Step S312: Obtain the complete source file metadata object to be read according to the key of the complete source file metadata object to be read;

[0109] In this step, after obtaining the key of the complete source file metadata object to be read, the complete source file metadata object to be read can be obtained.

[0110] Step S313: Calculate the current data cache time according to the complete source file metadata object to be read, and determine whether the cache has expired according to the current data cache time. If the determination is yes, delete the cache;

[0111] In this step, if the determination is yes, that is, it is determined that the cache has expired according to the current data cache time, so the cache is deleted at this time.

[0112] Step S314: Obtain the key of the complete source file fragment object to be read according to the complete source file metadata object to be read;

[0113] Step S315: Obtain the data of the fragment to be read according to the complete source file fragment object to be read;

[0114] In this step, obtaining the data of the fragment to be read facilitates obtaining the complete source file to be cached after merging.

[0115] Step S316: Merge the data of each fragment to be read and generate a complete source file to be cached after merging, and output the complete source file to be cached after merging.

[0116] Specifically, by merging the data of each fragment to be read and generating a complete source file to be cached after merging, the reading of the already stored file is realized.

[0117] In one embodiment, step S216: Using each of the target incomplete source file shard objects as keys and the incomplete source file bytecode as values, store each of the target incomplete source file shard objects in each of the key-value pair storage systems in the storage layer; and afterwards, further include:

[0118] Step S321: Generate a key for the metadata object of the incomplete source file to be read according to the URL of the request when reading the incomplete file;

[0119] In this step, when an incomplete file needs to be read, the URL of the request can be obtained.

[0120] Step S322: Obtain the metadata object of the incomplete source file to be read according to the key of the metadata object of the incomplete source file to be read;

[0121] Specifically, obtaining the metadata object of the incomplete source file to be read facilitates obtaining the complete source file subsequently.

[0122] Step S323: Calculate the current data cache time according to the metadata object of the incomplete source file to be read, and determine whether the cache has expired according to the current data cache time. If the determination is yes, delete the cache;

[0123] In this step, if the determination is yes, it means determining that the cache has expired according to the current data cache time.

[0124] Step S324: Obtain the key of the shard object of the complete source file to be read according to the metadata object of the complete source file to be read;

[0125] Step S325: Obtain the shard data of the incomplete source file to be read according to the shard object of the complete source file to be read;

[0126] In this step, the shard data of the incomplete source file to be read is used for merging and generating a complete file.

[0127] Step S326: Merge each of the shard data of the incomplete source file to be read and generate a merged incomplete source file to be cached, and output the merged incomplete source file to be cached.

[0128] In one embodiment, as Figure 5 shown, a file caching device for a content delivery network caching file system, the device includes:

[0129] An incomplete source file to be cached acquisition module, configured to acquire an incomplete source file to be cached;

[0130] A storage shard storage data module is used to split the to-be-cached source file into multiple shard storage data based on the logic layer in the content delivery network cache file system, and store each shard storage data in multiple key-value pair storage systems in the storage layer.

[0131] In one embodiment, the storage shard storage data module includes:

[0132] A shard object file module is used to generate multiple complete source file shard objects based on the logic layer in the content delivery network cache file system according to the to-be-cached complete source file;

[0133] A source file bytecode conversion module is used to encode and convert each complete source file shard object to generate a complete source file bytecode, where one complete source file shard object corresponds to one complete source file bytecode;

[0134] A storage system module is used to store each complete source file shard object in each key-value pair storage system in the storage layer with the complete source file shard object as the key and the complete source file bytecode as the value;

[0135] A file metadata object generation module is used to generate a complete source file metadata object based on the logic layer in the content delivery network cache file system according to the to-be-cached complete source file, where the key of the complete source file metadata object is the requested URL;

[0136] A metadata bytecode conversion module is used to convert the complete source file metadata object into a complete source file metadata bytecode according to the complete source file metadata object;

[0137] A storage module is used to store each complete source file metadata object in each key-value pair storage system in the storage layer with the complete source file metadata object as the key and the complete source file metadata bytecode as the value.

[0138] In one embodiment, it is characterized in that the storage shard storage data module includes:

[0139] An incomplete source file module is used to generate an incomplete source file metadata object based on the logic layer in the content delivery network cache file system according to the to-be-cached incomplete source file, where the key of the incomplete source file metadata object is the requested URL;

[0140] An incomplete source file metadata bytecode conversion module, configured to convert the incomplete source file metadata object into incomplete source file metadata bytecode according to the incomplete source file metadata object;

[0141] A key-value pair storage system module, configured to store each of the incomplete source file metadata objects in each of the key-value pair storage systems in the storage layer, with the incomplete source file metadata object as the key and the incomplete source file metadata bytecode as the value;

[0142] A target incomplete source file generation module, configured to generate each target incomplete source file shard object based on the logic layer in the content delivery network cache file system according to the incomplete source file to be cached;

[0143] A source file bytecode generation module, configured to encode and convert each of the target incomplete source file shard objects into incomplete source file bytecode, where one target incomplete source file shard object corresponds to one incomplete source file bytecode

[0144] An object as key storage module, configured to store each of the target incomplete source file shard objects in each of the key-value pair storage systems in the storage layer, with each of the target incomplete source file shard objects as the key and the incomplete source file bytecode as the value.

[0145] In one embodiment, the storage shard storage data module is further configured to: generate a key of the complete source file metadata object to be read according to the URL of the request when reading the complete file; obtain the complete source file metadata object to be read according to the key of the complete source file metadata object to be read; calculate the current data cache time according to the complete source file metadata object to be read, and determine whether the cache has expired according to the current data cache time. If the determination is yes, delete the cache; obtain the key of the complete source file shard object to be read according to the complete source file metadata object to be read; obtain the shard data to be read according to the complete source file shard object to be read; merge each of the shard data to be read and generate a merged complete source file to be cached, and output the merged complete source file to be cached.

[0146] In one embodiment, the stored shard storage data module is further configured to: generate a key of the metadata object of the incomplete source file to be read according to the URL of the request when reading an incomplete file; obtain the metadata object of the incomplete source file to be read according to the key of the metadata object of the incomplete source file to be read; calculate the current data cache time according to the metadata object of the incomplete source file to be read, and determine whether the cache has expired according to the current data cache time. If the determination is yes, delete the cache; obtain the key of the shard object of the complete source file to be read according to the metadata object of the incomplete source file to be read; obtain the incomplete shard data to be read according to the shard object of the complete source file to be read; merge the incomplete shard data to be read and generate a merged incomplete source file to be cached, and output the merged incomplete source file to be cached.

[0147] In one embodiment, as Figure 6 shown, a computer device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the steps of the file caching method based on the content delivery network cache file system described above are implemented.

[0148] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the file caching method based on the content delivery network cache file system described above are implemented.

[0149] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the various embodiments provided in the present application can include non-volatile and / or volatile memories. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0150] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this specification.

[0151] The above-described embodiments merely represent several implementation manners of the present application. The description is relatively specific and detailed, but it should not be construed as a limitation on the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the appended claims.

Claims

1. A file caching method based on a content delivery network cache file system, characterized in that, the method is based on a content delivery network cache file system, the content delivery network cache file system includes a logic layer and a storage layer, and the storage layer includes multiple key-value pair storage systems; the file caching method includes: Step S100: Obtain the source file to be cached; Step S200: Based on the logic layer in the content delivery network cache file system, split the source file to be cached into multiple shard storage data, the source file to be cached includes a complete source file to be cached, and store each shard storage data in multiple key-value pair storage systems in the storage layer, specifically including: Step S211: Based on the logic layer in the content delivery network cache file system, generate multiple complete source file shard objects according to the complete source file to be cached; Step S212: Encode and convert each complete source file shard object to generate a complete source file byte code, where one complete source file shard object corresponds to one complete source file byte code; Step S213: Use the complete source file shard object as the key and the complete source file byte code as the value, and store each complete source file shard object in each key-value pair storage system in the storage layer; Step S214: Based on the logic layer in the content delivery network cache file system, generate a complete source file metadata object according to the complete source file to be cached, where the key of the complete source file metadata object is the requested URL; Step S215: Convert the complete source file metadata object into a complete source file metadata byte code according to the complete source file metadata object; Step S216: Use the complete source file metadata object as the key and the complete source file metadata byte code as the value, and store each complete source file metadata object in each key-value pair storage system in the storage layer.

2. The file caching method based on a content delivery network cache file system according to claim 1, characterized in that, the source file to be cached includes an incomplete source file to be cached; the incomplete source file to be cached is split to form multiple target incomplete source file shard objects; Step S200: Based on the logic layer in the content delivery network cache file system, split the source file to be cached into multiple shard storage data, and store each shard storage data in multiple key-value pair storage systems in the storage layer; specifically including: Step S221: Based on the logic layer in the content delivery network cache file system, generate an incomplete source file metadata object according to the incomplete source file to be cached, where the key of the incomplete source file metadata object is the requested URL; Step S222: Convert the incomplete source file metadata object into an incomplete source file metadata byte code according to the incomplete source file metadata object; Step S223: Store each of the incomplete source file metadata objects in each of the key-value pair storage systems in the storage layer, with the incomplete source file metadata object as the key and the incomplete source file metadata bytecode as the value; Step S224: Generate each target incomplete source file shard object based on the incomplete source file to be cached by the logic layer in the content delivery network cache file system; Step S225: Encode and transform each of the target incomplete source file shard objects to generate incomplete source file bytecodes, where one target incomplete source file shard object corresponds to one incomplete source file bytecode; Step S216: Store each of the target incomplete source file shard objects in each of the key-value pair storage systems in the storage layer, with each of the target incomplete source file shard objects as the key and the incomplete source file bytecode as the value.

3. The file caching method based on a content delivery network cache file system according to claim 1, wherein, Step S216: Store each of the complete source file metadata objects in each of the key-value pair storage systems in the storage layer, with the complete source file metadata object as the key and the complete source file metadata bytecode as the value; After that, it further includes: Step S311: Generate a key for the complete source file metadata object to be read according to the URL of the request when reading the complete file; Step S312: Obtain the complete source file metadata object to be read according to the key of the complete source file metadata object to be read; Step S313: Calculate the current data cache time according to the complete source file metadata object to be read, and determine whether the cache has expired according to the current data cache time. If the determination is yes, delete the cache; Step S314: Obtain the key of the complete source file shard object to be read according to the complete source file metadata object to be read; Step S315: Obtain the shard data to be read according to the complete source file shard object to be read; Step S316: Merge each of the shard data to be read and generate a merged complete source file to be cached, and output the merged complete source file to be cached.

4. The file caching method based on a content delivery network cache file system according to claim 2, wherein, Step S216: Store each of the target incomplete source file shard objects in each of the key-value pair storage systems in the storage layer, with each of the target incomplete source file shard objects as the key and the incomplete source file bytecode as the value; After that, it further includes: Step S321: Generate a key for the incomplete source file metadata object to be read according to the URL of the request when reading the incomplete file; Step S322: Obtain the complete source file metadata object to be unread according to the key of the incomplete source file metadata object to be read; Step S323: Calculate the current data cache time according to the incomplete source file metadata object to be read, and determine whether the cache has expired based on the current data cache time. If the determination is yes, delete the cache; Step S324: Obtain the key of the complete source file shard object to be read according to the complete source file metadata object to be read; Step S325: Obtain the incomplete shard data to be read according to the complete source file shard object to be read; Step S326: Merge the incomplete shard data to be read and generate a merged incomplete source file to be cached, and output the merged incomplete source file to be cached.

5. A file caching device for a content delivery network cache file system, which is applied to the file caching method of the content delivery network cache file system according to any one of claims 1-4. Characterized in that: The device includes: A source file to be cached acquisition module, which is used to acquire the source file to be cached; A storage shard storage data module, which is used to split the source file to be cached into multiple shard storage data based on the logic layer in the content delivery network cache file system, and store each shard storage data in multiple key-value pair storage systems in the storage layer.

6. The file caching device for a content delivery network cache file system according to claim 5. Characterized in that: The storage shard storage data module includes: A shard object file module, which is used to generate multiple complete source file shard objects based on the logic layer in the content delivery network cache file system according to the complete source file to be cached; A source file bytecode conversion module, which is used to encode and convert each complete source file shard object to generate a complete source file bytecode, where one complete source file shard object corresponds to one complete source file bytecode; A storage system module, which is used to store each complete source file shard object in each key-value pair storage system in the storage layer with the complete source file shard object as the key and the complete source file bytecode as the value; A file metadata object generation module, which is used to generate a complete source file metadata object based on the logic layer in the content delivery network cache file system according to the complete source file to be cached, where the key of the complete source file metadata object is the requested URL; A metadata bytecode conversion module, which is used to convert the complete source file metadata object into a complete source file metadata bytecode according to the complete source file metadata object; A storage module, which is used to store each complete source file metadata object in each key-value pair storage system in the storage layer with the complete source file metadata object as the key and the complete source file metadata bytecode as the value.

7. The file caching device for a content delivery network cache file system according to claim 5. Characterized in that: The storage shard storage data module includes: An incomplete source file module, which is used to generate an incomplete source file metadata object based on the logic layer in the content delivery network cache file system according to the incomplete source file to be cached, wherein the key of the incomplete source file metadata object is the requested URL; An incomplete source file metadata bytecode conversion module, which is used to convert the incomplete source file metadata object into incomplete source file metadata bytecode according to the incomplete source file metadata object; A key-value pair storage system module, which is used to store each of the incomplete source file metadata objects in each of the key-value pair storage systems in the storage layer, with the incomplete source file metadata object as the key and the incomplete source file metadata bytecode as the value; A target incomplete source file generation module, which is used to generate each target incomplete source file shard object based on the logic layer in the content delivery network cache file system according to the incomplete source file to be cached; A source file bytecode generation module, which is used to encode and convert each of the target incomplete source file shard objects to generate incomplete source file bytecodes, wherein one target incomplete source file shard object corresponds to one incomplete source file bytecode; An object-as-key storage module, which is used to store each of the target incomplete source file shard objects in each of the key-value pair storage systems in the storage layer, with each of the target incomplete source file shard objects as the key and the incomplete source file bytecode as the value.

8. A computer device, including a memory and a processor, the memory stores a computer program, characterized in that, when the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

9. A computer-readable storage medium, on which a computer program is stored, characterized in that, when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images