File distribution method and device based on CDN and object storage

Abstract

The invention provides a file distribution method and device based on a CDN and object storage, and the method comprises the steps: receiving a multi-scene file pre-distribution task request sent by a client terminal, and the task request at least comprises injection, deletion, preheating or refreshing; analyzing the task request, and determining task data corresponding to the task request; obtaining a resource file corresponding to the task data according to the task data; allocating the resource file to object storage, and then allocating the resource file to a father node and an edge node of the CDN node to obtain an allocation result of the resource file; according to the file pre-distribution processing method and device, related file pre-distribution requirements of clients accessing the CDN and the object storage are met, and the problems that in a current pre-distribution processing mode, the manual participation degree is high, the fusion degree of the CDN and the object storage is low, and the expansibility is poor are solved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a file distribution method and device based on CDN and object storage. Background Art

[0002] At present, the application of content delivery network (CDN) is becoming more and more widespread. Under the CDN architecture, when the nearest CDN node has no resources, CDN will go back to the customer's resource source station to pull resources back to the user and cache them at the service node. Object storage is a large-capacity, scalable storage service suitable for storing large amounts of unstructured data. It has the advantages of high cost-effectiveness, strong scalability, strong data persistence, and strong disaster recovery capabilities.

[0003] CDN can be used to distribute data in object storage, and object storage can be used as a backup and disaster recovery method for CDN. When CDN encounters problems, object storage can quickly take over to ensure service continuity. CDN and object storage can complement each other and bring greater advantages. Therefore, more and more customers have the need to access object storage while accessing CDN. Object storage can be used as the main source station or backup source station of CDN. On the basis of opening object storage and creating storage buckets, customers need to pre-inject a large number of historical files and newly produced files every day into object storage. Furthermore, files need to be distributed to the corresponding parent nodes and edge nodes of CDN.

[0004] At present, if CDN customers who have access to object storage for supplementation have the need for file pre-distribution, they usually provide a list of resource files first, and the technical staff of the object storage party downloads the files locally and then uploads them to the storage in batches; after confirming that the files have been uploaded to the object storage, the customer or CDN technical staff calls the CDN preheating refresh interface to distribute the files to the parent node and edge node of the CDN, thereby distributing the files to the object storage and CDN nodes. It can be seen that the existing processing methods for file pre-distribution have the problems of high manual participation and low integration of CDN and object storage. Summary of the invention

[0005] The present application shows a file distribution method and device based on CDN and object storage.

[0006] In a first aspect, the present application shows a file distribution method based on CDN and object storage, the method comprising:

[0007] Receiving a task request for pre-distribution of files in multiple scenarios sent by a client terminal, wherein the task request at least includes injection, deletion, preheating or refreshing;

[0008] Parsing the task request to determine task data corresponding to the task request;

[0009] According to the task data, obtaining a resource file corresponding to the task data;

[0010] Allocate the resource file to the object storage, and then to the parent node and edge node of the CDN node to obtain the allocation result of the resource file;

[0011] The allocation result is sent to the client terminal.

[0012] Optionally, parsing the task request to determine the task data corresponding to the task request includes:

[0013] The task request is deserialized to obtain task data, which at least includes task identification information, the name of the customer who issues the task, the injected object key, the first MD5 of the injected file, the task operation type, the object storage address, the object storage account, the source address information, the resource server address, the injection mode, the cloud storage bucket, the cloud storage node, and the object storage encryption password.

[0014] Optionally, the method further comprises:

[0015] Performing parameter verification on the task data to obtain task data that passes the verification;

[0016] A pre-distribution condition check is performed on the task data to obtain a condition check result.

[0017] Optionally, the method further comprises:

[0018] Verify the task data, and if passed, verify the pre-distribution conditions;

[0019] If the pre-distribution condition check is passed, the pre-injection task is packaged into a request of the http protocol according to the task data;

[0020] Start a new thread to asynchronously process file injection or deletion tasks;

[0021] The task processing result is called back to the client terminal.

[0022] Optionally, acquiring a resource file corresponding to the task data according to the task data includes:

[0023] According to the resource server address and the back-to-source address information, an http request is initiated to the back-to-source service layer to obtain the resource file.

[0024] Optionally, the method further comprises:

[0025] Generate an authentication header according to the object storage address, the object storage account, the injected object key and the object storage encryption password;

[0026] According to the preset transmission method, the corresponding interface of the object storage is called to synchronize the resource file to the object storage space. The preset transmission method includes at least one of the streaming atomic upload method, the streaming segment upload method or the file disk concurrent segment upload method.

[0027] Optionally, the method further includes: obtaining a second MD5 of the resource file;

[0028] Compare the first MD5 and the second MD5 to obtain a comparison result;

[0029] According to the comparison result, it is determined whether to delete the resource file.

[0030] In a second aspect, the present application shows a file distribution device based on CDN and object storage, the device comprising:

[0031] A receiving module, used for receiving a task request for pre-distribution of files in multiple scenarios sent by a client terminal, wherein the task request at least includes injection, deletion, preheating or refreshing;

[0032] A parsing module, used to parse the task request and determine the task data corresponding to the task request;

[0033] An acquisition module, used for acquiring a resource file corresponding to the task data according to the task data;

[0034] An allocation module is used to allocate the resource file to the object storage, and then to the parent node and edge node of the CDN node to obtain the allocation result of the resource file;

[0035] A sending module is used to send the allocation result to the client terminal.

[0036] Optionally, the parsing module is used to:

[0037] The task request is deserialized to obtain task data, which at least includes task identification information, the name of the customer who issues the task, the injected object key, the first MD5 of the injected file, the task operation type, the object storage address, the object storage account, the source address information, the resource server address, the injection mode, the cloud storage bucket, the cloud storage node, and the object storage encryption password.

[0038] Optionally, the parsing module is used to:

[0039] Performing parameter verification on the task data to obtain task data that passes the verification;

[0040] A pre-distribution condition check is performed on the task data to obtain a condition check result.

[0041] Optionally, the parsing module is used to:

[0042] Verify the task data, and if passed, verify the pre-distribution conditions;

[0043] If the pre-distribution condition check is passed, the pre-injection task is packaged into a request of the http protocol according to the task data;

[0044] Start a new thread to asynchronously process file injection or deletion tasks;

[0045] The task processing result is called back to the client terminal.

[0046] Optionally, the acquisition module is used to:

[0047] According to the resource server address and the back-to-source address information, an http request is initiated to the back-to-source service layer to obtain the resource file.

[0048] Optionally, the parsing module is used to:

[0049] Generate an authentication header according to the object storage address, the object storage account, the injected object key and the object storage encryption password;

[0050] According to the preset transmission method, the corresponding interface of the object storage is called to synchronize the resource file to the object storage space. The preset transmission method includes at least one of the streaming atomic upload method, the streaming segment upload method or the file disk concurrent segment upload method.

[0051] Optionally, the parsing module is used to:

[0052] Get the second MD5 of the resource file;

[0053] Compare the first MD5 and the second MD5 to obtain a comparison result;

[0054] According to the comparison result, it is determined whether to delete the resource file. In a third aspect, the present application shows an electronic device, the electronic device comprising: a processor; a memory for storing processor executable instructions; wherein the processor is configured to execute the method as described in any of the above aspects.

[0055] In a fourth aspect, the present application illustrates a non-temporary computer-readable storage medium, which, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to execute a method as described in any of the above aspects.

[0056] In a fifth aspect, the present application illustrates a computer program product. When instructions in the computer program product are executed by a processor of an electronic device, the electronic device is enabled to execute the method described in any of the above aspects.

[0057] The technical solution provided by the present application may include the following beneficial effects: a file distribution method and device based on CDN and object storage provided in an embodiment of the present application include: receiving a task request for file pre-distribution in multiple scenarios sent by a client terminal, wherein the task request at least includes injection, deletion, preheating or refresh; parsing the task request to determine the task data corresponding to the task request; obtaining a resource file corresponding to the task data according to the task data; allocating the resource file to the object storage, and then to the parent node and edge node of the CDN node to obtain the allocation result of the resource file; sending the allocation result to the client terminal. The embodiment of the present application meets the file pre-distribution related needs of customers who access CDN and object storage, meets a variety of file pre-distribution application scenarios, and aims to solve the problems of high manual participation, low integration of CDN and object storage, and poor scalability in the current pre-distribution processing method. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 This is a flowchart of the steps of a file distribution method based on CDN and object storage in the present application.

[0059] Figure 2 It is a system internal layer diagram of the present application;

[0060] Figure 3 It is a pre-distribution interface layer flow chart of this application;

[0061] Figure 4 It is a file injection timing diagram of this application;

[0062] Figure 5 This is a mirror source station automatic synchronization timing diagram of this application;

[0063] Figure 6 This is a structural block diagram of a file distribution device based on CDN and object storage in the present application;

[0064] Figure 7 is a block diagram of an electronic device of the present application;

[0065] Figure 8 It is a block diagram of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0066] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0067] Glossary:

[0068] CDN: Content Delivery Network is a new network architecture added to the existing Internet, consisting of high-performance acceleration nodes spread all over the world. When a user makes a request for a certain content, the service node closest to the user responds quickly. When the nearest service node does not have resources, CDN will go back to the source station to pull resources back to the user and cache them at the service node.

[0069] Object storage: It is built using distributed storage technology to meet the requirements of access, processing and elastic expansion of massive videos, pictures and other unstructured data. It can be accessed nearby to achieve localized storage and supports multiple network access to give full play to the advantages of cloud-network integration.

[0070] Reference Figure 1 , shows a flowchart of a method for distributing files based on CDN and object storage in the present application, which can be applied to electronic devices, wherein the method may specifically include the following steps:

[0071] S101, receiving a task request for pre-distribution of files in multiple scenarios sent by a client terminal, wherein the task request at least includes injection, deletion, preheating or refreshing;

[0072] S102, parsing the task request to determine the task data corresponding to the task request;

[0073] S103, according to the task data, obtaining a resource file corresponding to the task data;

[0074] S104, allocating the resource file to the object storage, and then to the parent node and edge node of the CDN node, to obtain the allocation result of the resource file;

[0075] S105: Send the allocation result to the client terminal.

[0076] like Figure 2As shown, the embodiment of the present application provides a file pre-distribution system applicable to multiple scenarios under the combination of CDN and object storage. It uses a layered design concept and divides the system into a pre-distribution interface layer, a file synchronization layer, and a back-to-source service layer. The pre-distribution interface layer handles the interaction with the client and the CDN node, the file synchronization layer handles the interaction with the object storage, and the back-to-source service layer handles the interaction with the customer resource server, which reduces the responsibility of the system, makes the system have good scalability and reusability, strengthens the connection between the object storage and the CDN node in the file pre-distribution, and provides an architectural foundation for multi-scenario applications. In addition, it provides a variety of pre-distribution options and functions to meet the various file pre-distribution application scenarios required by the industry, and more application scenarios can be expanded in the future.

[0077] Specifically: the system includes a pre-distribution interface layer, a file synchronization layer, and a back-to-source service layer. The pre-distribution interface layer provides a pre-distribution task issuance interface, parses the tasks issued by the customer, identifies the task type, initiates file injection or file deletion requests to the file synchronization layer, initiates preheating or refresh requests to the CDN node, and notifies the client of the task issuance status and task completion status. The file synchronization layer is used to handle interactions with object storage. If a file deletion request is received, the deletion interface of the object storage is called to delete the corresponding object in the storage; if a file injection request is received, the back-to-source service layer is requested to obtain the resource file, and the upload interface of the object storage is called to upload the resource file to the storage. The back-to-source service layer is used to handle interactions with the resource server and pull the required resource files from the resource server.

[0078] Another embodiment of the present application further supplements the file distribution method based on CDN and object storage provided in the above embodiment.

[0079] Optionally, parsing the task request to determine task data corresponding to the task request includes:

[0080] The task request is deserialized to obtain task data, which at least includes task identification information, the name of the customer who issued the task, the injected object key, the first MD5 of the injected file, the task operation type, the object storage address, the object storage account, the back-source address information, the resource server address, the injection mode, the cloud storage bucket, the cloud storage node, and the object storage encryption password.

[0081] like Figure 3 As shown, optionally, the method further includes:

[0082] Perform parameter verification on the task data to obtain the task data that has passed the verification;

[0083] Perform pre-distribution condition check on the task data to obtain the condition check result.

[0084] Optionally, according to the task data, obtaining a resource file corresponding to the task data includes:

[0085] According to the resource server address and the back-to-source address information, an HTTP request is initiated to the back-to-source service layer to obtain the resource file.

[0086] Optionally, the method further comprises:

[0087] Generate an authentication header based on the object storage address, object storage account, injected object key, and object storage encryption password;

[0088] According to the preset transmission method, the corresponding interface of the object storage is called to synchronize the resource file to the object storage space. The preset transmission method includes at least one of the streaming atomic upload method, the streaming segment upload method or the file disk concurrent segment upload method.

[0089] Optionally, the method further comprises:

[0090] Get the second MD5 of the resource file;

[0091] Compare the first MD5 with the second MD5 to obtain a comparison result;

[0092] Based on the comparison result, determine whether to delete the resource file.

[0093] Specifically: 1. The pre-distribution interface layer provides a pre-distribution task issuing interface, a task status query interface, and a heartbeat detection interface.

[0094] 1.1. The pre-distribution task issuing interface receives the HTTP request with the request method of POST from the client. The received request body is a serialized JSON string. The interface processing flow is as follows:

[0095] (1) Deserialize the JSON string to obtain the task information, which includes the task ID, the name of the customer who issued the task, the injected object key, the MD5 of the injected file, the operation type, the injection mode, the cloud storage bucket, the cloud storage endpoint, the object storage encryption ak, the last-modified of the source file, and the task callback address. The detailed description is shown in Table 1.

[0096] (2) Check the task parameters. If they pass, proceed to step (3). If they fail, proceed to step (5).

[0097] (3) Perform pre-distribution condition check. If it passes, proceed to step (4); if it fails, proceed to step (5);

[0098] (4) Package the pre-injected task into a request of the http protocol according to the task parameters;

[0099] (5) Send a synchronous response to the request initiator. If the verification of steps (2) and (3) passes, proceed to step (6); otherwise, end the process;

[0100] (6) Start a new thread to asynchronously process file injection or deletion tasks;

[0101] (7) Call back the task processing result to notify the client.

[0102] Table 1 Pre-distribution task information parameter table

[0103]

[0104]

[0105] 1.1.1. The task parameter verification described in step (2) of 1.1.1 is to verify the type and range legality of the task parameters.

[0106] 1.1.2. The pre-distribution condition check described in step (3) of 1.1, when the operation type is injection, executes the corresponding logic according to the injection condition check type inject_mode parameter value in the task parameters.

[0107] If inject_mode is 0, it is forced injection mode, no conditional check is performed, and forced injection is performed;

[0108] If inject_mode is 1, check whether the storage file is the latest. If it is the latest, do not perform injection;

[0109] If inject_mode is 2, check whether the resource server file is updated. If not, do not perform the injection.

[0110] 1.1.2.1 and 1.1.2 verify whether the object file is the latest by carrying the "If-Modified-Since" header, with the header value last_modified in the task parameter, and initiating an http request with the request method of HEAD to the object storage. If a 304 status code is obtained, it means that the file is the latest.

[0111] 1.1.2.2. Check whether the resource server file is updated in 1.1.2 by carrying the "If-Modified-Since" header, with the header value last_modified in the task parameter, and initiating an http request with the request method of HEAD to the resource server. If a 304 status code is obtained, it means that the file has not been updated.

[0112] 1.1.3. The synchronous response described in step (5) of 1.1 is a pre-distribution request response sent by the client. The response status code 200 indicates that the task is successfully distributed, and the response status code 403 indicates that the task is failed to be distributed. The reason for the failure is responded to the client in the form of a body.

[0113] 1.1.4 The file injection task described in step (6) of 1.1 is that the pre-distribution interface layer sends a file synchronization request to the file synchronization layer. If a 200 status code is received in response from the file synchronization layer, it means that the file upload object is stored successfully. Optionally, a preheating request is sent to the CDN edge node.

[0114] The file synchronization request described in 1.1.4.1 and 1.1.4 is an http request that carries task information and a request header "Oper-Type:sync". The request method is GET, and the task information is carried in the form of a request line or a request header.

[0115] The purpose of the preheating request described in 1.1.4.2 and 1.1.4 is to trigger the CDN to automatically return to the object storage to obtain the resource files that have just been successfully uploaded, and cache the files in the CDN edge nodes and parent nodes, thereby completing the pre-distribution of the files on the CDN.

[0116] 1.1.5. The file deletion task described in step (6) of 1.1 is that the pre-distribution interface layer sends a file deletion request to the file synchronization layer. If a 200 status code is received in response from the file synchronization layer, it means that the file is successfully deleted in the object storage. Optionally, a refresh request is sent to the CDN edge node.

[0117] The file deletion request described in 1.1.5.1 and 1.1.5 is an http request that carries task information and a request header "Oper-Type: delete", the request method is DELETE, and the task information is carried in the form of a request line or a request header.

[0118] The refresh request process described in 1.1.5.2 and 1.1.5 is to obtain the cache key corresponding to the deleted file in the object storage in the CDN, and delete the resource identified by the cache key from the CDN edge node and the parent node.

[0119] 1.1.6. In the notification client described in step (7) of 1.1, the server address where the client receives the notification is the callback parameter value in the task parameters.

[0120] 1.2. The task status query interface described in 1 uses the unique identifier of the task, taskid, as the query key to query the real-time information of the corresponding task, including the injected or deleted URL, task start time, task completion time, task time spent, task status, and task failure reason.

[0121] 1.3 The heartbeat detection interface described in 1 is used by external components to detect whether the pre-distribution system is running normally.

[0122] 2. The file synchronization layer parses the http request initiated by the pre-distribution interface layer, identifies the task operation type and performs one of the following operations: synchronizing files from the resource server to the object storage; deleting the object storage file. In addition, it provides flow limiting protection for the traffic uploaded to the object storage and the back-to-source traffic.

[0123] 2.1.2 Parsing the request information initiated by the pre-distribution interface layer refers to obtaining the task information including the task operation type, object storage address, object storage account, object key, back-to-source URL, and resource server address from the request line and internal request glue header of the request information. See Table 2 for the description of the internal request glue header.

[0124] Table 2 Internal Request Adhesive Header Description Table

[0125]

[0126]

[0127] 2.2. The identification operation type described in 2 is obtained from the Oper-Type request header in the request message. The meaning of the Oper-Type value refers to the above table. SYNC and REMOVE correspond to the above two operations.

[0128] 2.3. The detailed process of synchronizing to object storage as described in 2 is as follows:

[0129] (1) Based on the parsed resource server address and back-to-source URL, an HTTP request is initiated to the back-to-source service layer to retrieve the resource file;

[0130] (2) According to the object storage address, object storage account, and object key obtained by parsing, the account's ak and sk are obtained internally, an authentication header is generated, the required upload method is selected, the corresponding interface provided by the object storage is called, and the file is synchronized to the object storage;

[0131] (3) If the edge pre-distribution client carries the File-Md5 request header, after the file is successfully uploaded to the object storage, the md5 of the file in the storage is obtained to verify whether it is consistent with the value of the File-Md5 request header. If it is confirmed that the file md5 is inconsistent, the deletion interface provided by the object storage is called to delete the file, and the file synchronization fails;

[0132] (4) If the synchronization is successful, the client will be responded with a 200 status code; if the synchronization fails, the client will be responded with a 403 status code.

[0133] 2.3.1. The required upload methods described in step (2) of 2.3 include streaming atomic upload, streaming multi-segment upload, and concurrent multi-segment upload with file storage.

[0134] 2.3.2. The md5 verification described in step (3) of 2.3 is inconsistent. A retry function is provided. If the verification fails within the limited number of retries, the file md5 is confirmed to be inconsistent.

[0135] 2.4. The object storage file is deleted as described in 2. According to the object storage address, object storage account, and object key obtained by parsing, the ak and sk of the account are obtained internally, an authentication header is generated, and the deletion interface provided by the object storage is called to delete the object from the object storage.

[0136] 2.5. The current limiting protection described in 2 supports current limiting operations on requests based on the leaky bucket algorithm and flow control based on a second-level sliding time window to protect object storage and resource servers.

[0137] 3. The back-to-source service layer provides back-to-source information settings, 302 follow-up, and source station response rewriting services, which customers can configure by themselves according to actual needs. Furthermore, customer-defined programming services are provided as a supplement.

[0138] The back-to-source information settings described in 3.1 and 3 include back-to-source parameter settings, back-to-source request header settings, and back-to-source encryption string generation, to ensure that the back-to-source request can successfully obtain the resource file through the source station's anti-hotlink and other restricted access measures.

[0139] 3.2. The 302 follow mentioned in 3 means that if the back-to-source service layer receives a 302 redirect response from the source site, it will jump to the 302 corresponding address to obtain the real resource file and then respond.

[0140] 3.3. The source station response rewriting described in 3 refers to the back-to-source service layer editing the small files obtained from the source station, modifying the file content and then responding to the file synchronization layer.

[0141] The size of small files mentioned in 3.3.1 and 3.3 cannot exceed 10M.

[0142] 3.4. The customer-defined programming described in 3 creates a sandbox environment to execute the business code written by the customer based on the programmable CDN function. Customers can use the provided interface functions and supported scripting languages ​​to write business codes and independently implement business logic including setting back-to-source request information, customized anti-hotlinking, authentication, rewriting, redirection, speed limiting, etc. Specific embodiment one:

[0144] This embodiment is based on a file pre-distribution system that is applicable to multiple scenarios under the combination of CDN and object storage, and provides the following file injection method for customers who access both CDN and object storage. Figure 4 As shown, the method comprises the following steps:

[0145] The S1 client initiates a pre-distribution task with the pre-distribution task issuing interface provided by the file pre-distribution system, whose operation type is injection and injection mode is forced injection, and which needs to be preheated to the CDN node, that is, the task parameters operation is 1, inject_mode is 0, and sync_cdn is 1.

[0146] The S2 file pre-distribution system processes the pre-distribution tasks sent by the client. The internal processing flow is as follows:

[0147] The S21 pre-distribution interface layer receives the http request with the request method of POST from the client, deserializes the json string of the request body to obtain the task parameters, performs parameter verification on each field of the pre-distribution task, and skips the conditional verification of the injection file in the forced injection mode. After the verification passes, the pre-distribution task is packaged into a request of the http protocol, the asynchronous thread request file synchronization layer is started, and the pre-distribution task delivery status is synchronously responded to the client.

[0148] The S22 file synchronization layer parses the http request initiated by the pre-distribution interface layer, obtains the task information including the task operation type, object storage address, object storage account, object key, back-to-source URL, and resource server address from the request line and internal request bonding header of the request information, and identifies the task type as synchronizing files from the resource server to the object storage. The file synchronization layer uses the object storage address as the key, and if the current limiting condition has been met, the request is current limited based on the leaky bucket algorithm.

[0149] The S23 file synchronization layer sends a request to the back-source service layer to fetch the resources to be uploaded.

[0150] The S24 back-to-source service layer receives the request from the synchronization layer, matches the request information to the corresponding sub-channel configuration, executes the relevant back-to-source information settings, and 302 follows the business logic. If the customer uses the custom programming function, the business code written by the customer is executed in the sandbox environment, and the back-to-source request information is generated, and a request is sent to the resource server to obtain the resource file.

[0151] S25 If the resource file does not need to be edited, the back-to-source service layer will transparently transmit the response to the file synchronization layer; if the resource file needs to be edited, the back-to-source service layer will execute the relevant text rewriting logic. If the customer uses the custom programming function, the business code written by the customer will be executed in the sandbox environment. After the text editing is completed, the edited file will be responded to the file synchronization layer.

[0152] The S26 file synchronization layer obtains the customer account's ak and sk internally based on the parsed object storage address, object storage account, and object key, generates an authentication header, and selects one of the upload methods, streaming atomic upload, streaming multi-slice upload, or file storage concurrent multi-slice upload, to upload the file responded by the back-to-source service layer to the object storage.

[0153] S27 If the edge pre-distribution client carries the File-Md5 request header, the file synchronization layer obtains the md5 of the file in the storage after the file is successfully uploaded to the object storage, and verifies whether it is consistent with the value of the File-Md5 request header. If it is confirmed that the file md5 is inconsistent, the deletion interface provided by the object storage is called to delete the file, and the file synchronization fails.

[0154] The S28 file synchronization layer responds to the edge pre-distribution client injection result. If the synchronization is successful, the response is a 200 status code, and if the synchronization fails, the response is a 403 status code.

[0155] The S29 pre-distribution interface layer receives the 200 status code from the file synchronization layer, indicating that the file has been successfully uploaded to the object storage. It sends a preheating request to the CDN edge, triggering the CDN to automatically return to the object storage to obtain the resource file that has just been successfully uploaded, cache the file in the CDN edge node and parent node, and complete the pre-distribution of the file on the CDN node.

[0156] The S3 file pre-distribution system sends an http request with the request method of POST to the address in the task parameter callback, and notifies the client of the task processing result. Specific embodiment 2:

[0158] This embodiment is based on a file pre-distribution system applicable to multiple scenarios under the combination of CDN and object storage, and provides the following file refresh method for customers who access both CDN and object storage. The method includes the following steps:

[0159] The S1 client initiates a pre-distribution task with an operation type of deletion and a CDN node refresh requirement to the pre-distribution task delivery interface provided by the file pre-distribution system, that is, the task parameters operation is 0 and sync_cdn is 1.

[0160] The S2 file pre-distribution system processes the pre-distribution tasks sent by the client. The internal processing flow is as follows:

[0161] The S21 pre-distribution interface layer receives the http request with the request method of POST from the client, deserializes the json string of the request body to obtain the task parameters, performs parameter verification on each field of the pre-distribution task, and after the verification, packages the pre-distribution task into a request of the http protocol, starts the asynchronous thread request file synchronization layer, and synchronously responds the pre-distribution task distribution status to the client.

[0162] The S22 file synchronization layer parses the http request initiated by the pre-distribution interface layer, obtains the task information including the task operation type, object storage address, object storage account, object key, back-to-source URL, and resource server address from the request line and internal request header of the request information, and identifies the task type as deleting the specified object from the object storage.

[0163] The S23 file synchronization layer obtains the customer account's ak and sk internally based on the parsed object storage address, object storage account, and object key, generates an authentication header, and calls the deletion interface provided by the object storage to delete the object.

[0164] The S24 file synchronization layer responds to the edge pre-distribution client injection results. Delete the 200 status code for successful responses and the 403 status code for failed responses.

[0165] The S25 pre-distribution interface layer receives the 200 status code from the file synchronization layer, deletes the object successfully, sends a refresh request to the CDN edge, deletes the corresponding cache in the CDN edge node and the parent node, and completes the refresh of the file on the CDN node.

[0166] The S3 file pre-distribution system sends an http request with the request method of POST to the address in the task parameter callback, and notifies the client of the task processing result. Specific embodiment three:

[0168] This embodiment is based on a file pre-distribution system applicable to multiple scenarios under the combination of CDN and object storage, and provides the following method for automatically synchronizing customer object storage and resource server files for customers who access both CDN and object storage to achieve mirroring source stations. The timing diagram of this embodiment is shown in the attached figure. Figure 5 As shown, the method comprises the following steps:

[0169] The cache server of the S1 CDN parent node determines whether the requested file meets the pre-distribution conditions. If so, it initiates an injection-type pre-distribution task to the file pre-distribution system.

[0170] S11 satisfies the pre-distribution condition if any of the following conditions is met:

[0171] (1) When the object storage is the primary source and the client resource server is the backup source, the cache server returns to the resource server to fetch the file after returning a 404 response to the object storage for the first time. The file is now completely cached.

[0172] (2) When the client resource server is the primary source and the object storage is the backup source, the cache server returns a 200 response when it first obtains a complete file from the primary source or when the subsequent cache expires.

[0173] The S12 cache server determines that the requested file meets the pre-distribution conditions, and initiates a pre-distribution task to the file pre-distribution system with an operation type of injection, an injection mode of verifying whether the storage verification file is the latest, and a resource server address of the cache server address, that is, the task parameters operation is 1, inject_mode is 1, and origin_addr is the cache server address.

[0174] The S2 file pre-distribution system processes the pre-distribution tasks sent by the client. The internal processing flow is as follows:

[0175] The S21 pre-distribution interface layer receives the http request with the request method of POST from the client, deserializes the json string of the request body to obtain the task parameters, performs parameter verification on each field of the pre-distribution task, and verifies whether the file in the storage is the latest. After the verification, the pre-distribution task is packaged into a request of the http protocol, the asynchronous thread request file synchronization layer is started, and the pre-distribution task delivery status is synchronously responded to the client.

[0176] The S22 file synchronization layer parses the http request initiated by the pre-distribution interface layer, obtains the task information including the task operation type, object storage address, object storage account, object key, back-to-source URL, and CDN cache server address from the request line and internal request glue header of the request information, and identifies the task type as synchronizing files from the CDN cache server to the object storage. The file synchronization layer uses the object storage address as the key, and if the current limiting condition has been met, the request is current limited based on the leaky bucket algorithm.

[0177] The S23 file synchronization layer initiates a request to the back-to-source service layer to fetch the resources to be uploaded. The back-to-source service layer receives the request from the synchronization layer, sends a request to the CDN cache server to obtain the resource file, and transmits the response to the file synchronization layer.

[0178] The S24 file synchronization layer obtains the customer account's ak and sk internally based on the parsed object storage address, object storage account, and object key, generates an authentication header, and selects one of the upload methods, streaming atomic upload, streaming multi-slice upload, or concurrent multi-slice upload to upload the file responded by the back-to-source service layer to the object storage.

[0179] The S25 file synchronization layer responds to the edge pre-distribution client injection result. If the synchronization is successful, the response is a 200 status code, and if the synchronization fails, the response is a 403 status code.

[0180] The S26 pre-distribution interface layer receives a 200 status code from the file synchronization layer, and the file upload object is successfully stored.

[0181] The S3 file pre-distribution system sends an http request with the request method of POST to the address in the task parameter callback, and notifies the client of the task processing result.

[0182] The embodiment of the present application provides a file pre-distribution system applicable to multiple scenarios under the combination of CDN and object storage. It uses a layered design concept and is divided into a pre-distribution interface layer, a file synchronization layer, and a back-to-source service layer. The pre-distribution interface layer handles the interaction with the client and the CDN node, the file synchronization layer handles the interaction with the object storage, and the back-to-source service layer handles the interaction with the customer resource server. The complexity of the system is reduced, the system has good scalability and reusability, the connection between the object storage and the CDN node in the file pre-distribution is strengthened, and the architectural foundation is provided for multi-scenario applications.

[0183] It provides a variety of pre-distribution options, including operation type, injection mode, whether to verify file MD5, and whether to synchronize CDN nodes when injecting or deleting files. The system identifies the corresponding options through the task information parameters of the client pre-distribution task, and completes operations such as file injection condition verification, file consistency verification, file pre-injection into object storage and distribution to CDN nodes, file deletion from object storage and refresh in CDN nodes. It provides a functional basis for file pre-distribution in multiple scenarios under the combination of CDN and object storage, and can provide integrated file pre-distribution and file refresh services, as well as mirror source services for customers who are connected to CDN and object storage at the same time. More application scenarios can be expanded in the future.

[0184] The system back-to-source service layer provides self-configurable back-to-source information settings, 302 follow-up, and source station response rewriting general services. Furthermore, it provides customer-defined programming services based on programmable CDN as a supplement. Customers can self-configure or implement customized anti-hotlinking, customized back-to-source, m3u8 rewriting and other requirements, reducing the workload of demand development and improving demand delivery efficiency.

[0185] The system can solve the problems of high manual participation, low integration of CDN and object storage, and poor scalability in traditional pre-distribution processing methods, and can adapt to various application scenarios of the combination of CDN and object storage.

[0186] Another embodiment of the present application provides a file distribution device based on CDN and object storage, which is used to execute the file distribution method based on CDN and object storage provided by the above embodiment.

[0187] like Figure 6 , which is a schematic diagram of the structure of a file distribution device based on CDN and object storage provided in an embodiment of the present application. The file distribution device based on CDN and object storage includes a receiving module 601, a parsing module 602, an acquisition module 603, an allocation module 604 and a sending module 605, wherein:

[0188] The receiving module 601 is used to receive a task request for pre-distribution of files in multiple scenarios sent by a client terminal, wherein the task request at least includes injection, deletion, preheating or refreshing;

[0189] The parsing module 602 is used to parse the task request and determine the task data corresponding to the task request;

[0190] The acquisition module 603 is used to acquire the resource file corresponding to the task data according to the task data;

[0191] The allocation module 604 is used to allocate the resource file to the object storage, and then to the parent node and edge node of the CDN node to obtain the allocation result of the resource file;

[0192] The sending module 605 is used to send the allocation result to the client terminal.

[0193] Regarding the device in this embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0194] The embodiment of the present application provides a file distribution device based on CDN and object storage, including: receiving task requests for file pre-distribution in multiple scenarios sent by a client terminal, wherein the task request at least includes injection, deletion, preheating or refresh; parsing the task request to determine the task data corresponding to the task request; obtaining the resource file corresponding to the task data according to the task data; allocating the resource file to the object storage, and then to the parent node and edge node of the CDN node to obtain the allocation result of the resource file; sending the allocation result to the client terminal. The embodiment of the present application connects to the file pre-distribution related needs of customers with CDN and object storage, meets a variety of file pre-distribution application scenarios, and aims to solve the problems of high manual participation, low integration of CDN and object storage, and poor scalability in the current pre-distribution processing method.

[0195] It should be noted that, for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions involved are not necessarily required by the present application.

[0196] Optionally, an embodiment of the present application further provides an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the various processes of the above-mentioned method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0197] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, each process of the above method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0198] FIG7 is a block diagram of an electronic device 800 shown in the present application. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0199] Reference Figure 7 , the electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0200] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0201] The memory 804 is configured to store various types of data to support operations on the device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, images, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0202] The power supply component 806 provides power to the various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 800.

[0203] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0204] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0205] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0206] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of components, such as the display and keypad of the electronic device 800, and the sensor assembly 814 can also detect the position change of the electronic device 800 or a component of the electronic device 800, the presence or absence of contact between the user and the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0207] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast operation information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0208] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0209] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the instructions can be executed by a processor 820 of an electronic device 800 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0210] Figure 819 is a block diagram of a computer-readable storage medium 1900 shown in the present application. For example, the computer-readable storage medium 1900 may be provided as a server.

[0211] Reference Figure 8 , the computer-readable storage medium 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions, such as an application, that can be executed by the processing component 1922. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0212] The computer readable storage medium 1900 may also include a power supply component 1926 configured to perform power management of the computer readable storage medium 1900, a wired or wireless network interface 1950 configured to connect the computer readable storage medium 1900 to a network, and an input / output (I / O) interface 1958. The computer readable storage medium 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™ or the like.

[0213] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0214] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the present application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0215] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A file distribution method based on CDN and object storage, characterized in that: The method comprises: Receiving a task request for pre-distribution of files in multiple scenarios sent by a client terminal, wherein the task request at least includes injection, deletion, preheating or refreshing; Parsing the task request to determine task data corresponding to the task request; According to the task data, obtaining a resource file corresponding to the task data; Allocate the resource file to the object storage, and then to the parent node and edge node of the CDN node to obtain the allocation result of the resource file; The allocation result is sent to the client terminal.

2. The file distribution method based on CDN and object storage according to claim 1, characterized in that: The step of parsing the task request to determine the task data corresponding to the task request includes: The task request is deserialized to obtain task data, which at least includes task identification information, the name of the customer who issues the task, the injected object key, the first MD5 of the injected file, the task operation type, the object storage address, the object storage account, the source address information, the resource server address, the injection mode, the cloud storage bucket, the cloud storage node, and the object storage encryption password.

3. The file distribution method based on CDN and object storage according to claim 2, characterized in that: The method further comprises: Performing parameter verification on the task data to obtain task data that passes the verification; The task data that has passed the verification is subjected to a pre-distribution condition verification to obtain a condition verification result.

4. The file distribution method based on CDN and object storage according to claim 3 is characterized in that: The method further comprises: Verify the task data, and if passed, verify the pre-distribution conditions; If the pre-distribution condition check is passed, the pre-injection task is packaged into a request of the http protocol according to the task data; Start a new thread to asynchronously process file injection or deletion tasks; The task processing result is called back to the client terminal.

5. The file distribution method based on CDN and object storage according to claim 2, characterized in that: The step of acquiring a resource file corresponding to the task data according to the task data includes: According to the resource server address and the back-to-source address information, an http request is initiated to the back-to-source service layer to obtain the resource file.

6. The file distribution method based on CDN and object storage according to claim 2, characterized in that: The method further comprises: Generate an authentication header according to the object storage address, the object storage account, the injected object key and the object storage encryption password; According to the preset transmission method, the corresponding interface of the object storage is called to synchronize the resource file to the object storage space. The preset transmission method includes at least one of the streaming atomic upload method, the streaming segment upload method or the file disk concurrent segment upload method.

7. The file distribution method based on CDN and object storage according to claim 6, characterized in that: The method further comprises: obtaining a second MD5 of the resource file; Compare the first MD5 and the second MD5 to obtain a comparison result; According to the comparison result, it is determined whether to delete the resource file.

8. A file distribution device based on CDN and object storage, characterized in that: The device comprises: A receiving module, used for receiving a task request for pre-distribution of files in multiple scenarios sent by a client terminal, wherein the task request at least includes injection, deletion, preheating or refreshing; A parsing module, used to parse the task request and determine the task data corresponding to the task request; An acquisition module, used for acquiring a resource file corresponding to the task data according to the task data; An allocation module is used to allocate the resource file to the object storage, and then to the parent node and edge node of the CDN node to obtain the allocation result of the resource file; A sending module is used to send the allocation result to the client terminal.

9. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the method according to any one of claims 1 to 7 when executed by the processor.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images