A file distribution method and device based on CDN and object storage

Abstract

The application provides a file distribution method and device based on CDN and object storage, comprising: receiving a multi-scene file pre-distribution task request sent by a client terminal, wherein the task request at least includes injection, deletion, preheating or refreshing; analyzing the task request to determine task data corresponding to the task request; obtaining resource files corresponding to the task data according to the task data; distributing the resource files to object storage, and then to parent nodes and edge nodes of CDN nodes to obtain a distribution result of the resource files; and sending the distribution result to the client terminal. The application accesses the file pre-distribution related needs of the client of CDN and object storage, meets various file pre-distribution application scenarios, and aims to solve the problems of high manual participation, low integration of CDN and object storage, and poor scalability of the current pre-distribution processing mode.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, and in particular to a file distribution method and device based on CDN and object storage. BACKGROUND

[0002] At present, the application of content delivery network (CDN) is more and more extensive. Under the architecture of CDN, when there is no resource in the nearby CDN node, the CDN will pull the resource from the customer resource source station and return it to the user and cache it in the service node. Object storage is a large-capacity and scalable storage service, which is suitable for storing a large amount of unstructured data, and has the advantages of high cost-effectiveness, strong scalability, strong data persistence and strong disaster recovery capability.

[0003] CDN can be used to distribute data in object storage, and object storage can be used as a backup and disaster recovery means for CDN. When CDN has a problem, object storage can quickly take over to ensure the continuity of the service. CDN and object storage can complement each other and exert greater advantages. Therefore, more and more customers have the demand to access object storage while accessing CDN. Object storage can be used as the main source station or backup source station of CDN. On the basis of opening object storage and creating a storage bucket, the customer needs to pre-inject a large number of historical files and daily newly produced files into the object storage, and further needs to distribute the files to the corresponding parent node and edge node of CDN.

[0004] At present, if a CDN customer who has accessed object storage for supplementation has the demand for file pre-distribution, the customer generally provides a list of resource files first, and then the technical personnel of the object storage party downloads the files to the local and then uploads them to the storage in batches. After confirming that the files are uploaded to the object storage, the technical personnel of the customer or CDN party calls the pre-warming refresh interface of CDN to distribute the files to the parent node and edge node of CDN, so as to distribute the files to the object storage and CDN node. It can be seen that the existing processing method for file pre-distribution has the problems of high manual participation and low integration of CDN and object storage. SUMMARY

[0005] The present application shows a file distribution method and device based on CDN and object storage.

[0006] In a first aspect, the present application shows a file distribution method based on CDN and object storage, which comprises:

[0007] receiving a multi-scene file pre-distribution task request sent by a customer terminal, wherein the task request at least includes injection, deletion, pre-warming or refreshing;

[0008] The task request is parsed to determine the task data corresponding to the task request;

[0009] Based on the task data, obtain the resource file corresponding to the task data;

[0010] The resource file is allocated to object storage, and then to the parent node and edge node of the CDN node to obtain the allocation result of the resource file;

[0011] The allocation result is sent to the client terminal.

[0012] Optionally, parsing the task request to determine the task data corresponding to the task request includes:

[0013] The task request is deserialized to obtain task data, which includes at least the task identifier information, the client name that issued the task, the injected object key, the first MD5 hash of the injected file, the task operation type, the object storage address, the object storage account, the origin address information, the resource server address, the injection mode, the cloud storage bucket, the cloud storage node, and the object storage encryption password.

[0014] Optionally, the method further includes:

[0015] The task data is validated to obtain valid task data.

[0016] The task data is pre-distributed and the conditions are verified to obtain the verification results.

[0017] Optionally, the method further includes:

[0018] The task data is verified, and if it passes, the pre-distribution conditions are verified.

[0019] If the pre-distribution condition verification is passed, the pre-injected task is packaged into an HTTP protocol request based on the task data.

[0020] Start a new thread to asynchronously process file injection or deletion tasks;

[0021] The task processing result is returned to the client terminal via callback.

[0022] Optionally, obtaining the resource file corresponding to the task data based on the task data includes:

[0023] Based on the resource server address and the origin address information, an HTTP request is initiated to the origin service layer to obtain the resource file.

[0024] Optionally, the method further includes:

[0025] Generate an authentication header based on the object storage address, object storage account, injected object key, and object storage encryption password;

[0026] According to the preset transmission method, the corresponding interface of the object storage is called to synchronize the resource file to the object storage space. The preset transmission method includes at least one of the following: streaming atomic upload method, streaming chunked upload method, or file disk-based concurrent chunked upload method.

[0027] Optionally, the method further includes: obtaining a second MD5 hash of the resource file;

[0028] The first MD5 and the second MD5 are compared to obtain the comparison result;

[0029] Based on the comparison results, determine whether to delete the resource file.

[0030] Secondly, this application discloses a file distribution apparatus based on CDN and object storage, the apparatus comprising:

[0031] The receiving module is used to receive multi-scenario file pre-distribution task requests sent by the client terminal, wherein the task requests include at least injection, deletion, preheating or refresh;

[0032] The parsing module is used to parse the task request and determine the task data corresponding to the task request;

[0033] The acquisition module is used to acquire the resource file corresponding to the task data based on the task data;

[0034] The allocation module is used to allocate the resource file to object storage, and then to the parent node and edge node of the CDN node to obtain the allocation result of the resource file;

[0035] The sending module is used to send the allocation result to the client terminal.

[0036] Optionally, the parsing module is used for:

[0037] The task request is deserialized to obtain task data, which includes at least the task identifier information, the client name that issued the task, the injected object key, the first MD5 hash of the injected file, the task operation type, the object storage address, the object storage account, the origin address information, the resource server address, the injection mode, the cloud storage bucket, the cloud storage node, and the object storage encryption password.

[0038] Optionally, the parsing module is used for:

[0039] The task data is validated to obtain valid task data.

[0040] The task data is pre-distributed and the conditions are verified to obtain the verification results.

[0041] Optionally, the parsing module is used for:

[0042] The task data is verified, and if it passes, the pre-distribution conditions are verified.

[0043] If the pre-distribution condition verification is passed, the pre-injected task is packaged into an HTTP protocol request based on the task data.

[0044] Start a new thread to asynchronously process file injection or deletion tasks;

[0045] The task processing result is returned to the client terminal via callback.

[0046] Optionally, the acquisition module is used to:

[0047] Based on the resource server address and the origin address information, an HTTP request is initiated to the origin service layer to obtain the resource file.

[0048] Optionally, the parsing module is used for:

[0049] Generate an authentication header based on the object storage address, object storage account, injected object key, and object storage encryption password;

[0050] According to the preset transmission method, the corresponding interface of the object storage is called to synchronize the resource file to the object storage space. The preset transmission method includes at least one of the following: streaming atomic upload method, streaming chunked upload method, or file disk-based concurrent chunked upload method.

[0051] Optionally, the parsing module is used for:

[0052] Obtain the second MD5 hash of the resource file;

[0053] The first MD5 and the second MD5 are compared to obtain the comparison result;

[0054] Based on the comparison result, it is determined whether to delete the resource file. Thirdly, this application discloses an electronic device comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to perform the method as described in any of the preceding aspects.

[0055] Fourthly, this application discloses a non-transitory computer-readable storage medium in which, when the instructions in the storage medium are executed by a processor of an electronic device, enable the electronic device to perform the methods described in any of the preceding aspects.

[0056] Fifthly, this application discloses a computer program product in which, when the instructions in the computer program product are executed by a processor of an electronic device, the electronic device is enabled to perform the method described in any of the preceding aspects.

[0057] The technical solution provided in this application can include the following beneficial effects: An embodiment of this application provides a file distribution method and apparatus based on CDN and object storage, comprising: receiving multi-scenario file pre-distribution task requests sent by a client terminal, wherein the task request includes at least injection, deletion, preheating, or refresh; parsing the task request to determine task data corresponding to the task request; obtaining resource files corresponding to the task data based on the task data; allocating the resource files to object storage, and then to the parent nodes and edge nodes of CDN nodes to obtain the allocation result of the resource files; and sending the allocation result to the client terminal. This embodiment of the application integrates CDN and object storage to meet the file pre-distribution needs of clients, satisfying various file pre-distribution application scenarios, and aims to solve the problems of high manual intervention, low integration of CDN and object storage, and poor scalability in current pre-distribution processing methods. Attached Figure Description

[0058] Figure 1 This is a flowchart illustrating the steps of a file distribution method based on CDN and object storage according to this application.

[0059] Figure 2 This is a system internal layer diagram of this application;

[0060] Figure 3 This is a flowchart of a pre-distribution interface layer in this application;

[0061] Figure 4 This is a file injection sequence diagram of this application;

[0062] Figure 5 This is a sequence diagram of automatic synchronization of a mirror source site according to this application;

[0063] Figure 6 This is a structural block diagram of a file distribution device based on CDN and object storage according to this application;

[0064] Figure 7 This is a block diagram of an electronic device according to this application;

[0065] Figure 8 This is a block diagram of a computer-readable storage medium according to this application. Detailed Implementation

[0066] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0067] Definitions:

[0068] CDN: Content Delivery Network, is a new network architecture added to the existing Internet, consisting of high-performance acceleration nodes located in various locations. When a user requests content, the service node closest to the user responds quickly. If the nearest service node does not have the resource, the CDN will retrieve the resource from the origin server and return it to the user, caching it on the service node.

[0069] Object storage: Built with distributed storage technology, it meets the requirements for accessing, processing, and elastically expanding massive amounts of video, images, and other unstructured data. It can be accessed locally to achieve localized storage and supports multiple network access methods to leverage the advantages of cloud-network convergence.

[0070] Reference Figure 1 The diagram illustrates a flowchart of a file distribution method based on CDN and object storage according to this application. This method can be applied to electronic devices, and specifically includes the following steps:

[0071] S101. Receive a multi-scenario file pre-distribution task request sent by the client terminal, wherein the task request includes at least injection, deletion, preheating or refresh;

[0072] S102. Parse the task request and determine the task data corresponding to the task request;

[0073] S103. Based on the task data, obtain the resource file corresponding to the task data;

[0074] S104. Allocate the resource files to object storage, and then to the parent nodes and edge nodes of the CDN nodes to obtain the allocation results of the resource files;

[0075] S105. Send the allocation result to the client terminal.

[0076] like Figure 2As shown in the embodiments of this application, a file pre-distribution system applicable to multiple scenarios under the combination of CDN and object storage uses a layered design approach. The system is divided into a pre-distribution interface layer, a file synchronization layer, and an origin service layer. The pre-distribution interface layer handles interactions with clients and CDN nodes; the file synchronization layer handles interactions with object storage; and the origin service layer handles interactions with client resource servers. This reduces the system's complexity, gives the system good scalability and reusability, strengthens the connection between object storage and CDN nodes in file pre-distribution, and provides an architectural foundation for multi-scenario applications. Furthermore, it provides diverse pre-distribution options and functions to meet the various file pre-distribution application scenarios required by the industry, and can be further expanded to include more application scenarios in the future.

[0077] Specifically, the system comprises a pre-distribution interface layer, a file synchronization layer, and an origin server service layer. The pre-distribution interface layer provides an interface for pre-distributing tasks, parses the tasks distributed by clients, identifies the task type, initiates file injection or deletion requests to the file synchronization layer, initiates preheating or refresh requests to CDN nodes, and notifies clients of the task distribution status and task completion. The file synchronization layer handles interactions with object storage. If a file deletion request is received, it calls the object storage's deletion interface to delete the corresponding object from storage; if a file injection request is received, it requests the origin server service layer to retrieve resource files and calls the object storage's upload interface to upload the resource files to storage. The origin server service layer handles interactions with resource servers, retrieving the required resource files from the resource servers.

[0078] Another embodiment of this application further supplements the description of the file distribution method based on CDN and object storage provided in the above embodiments.

[0079] Optionally, the task request is parsed to determine the task data corresponding to the task request, including:

[0080] The task request is deserialized to obtain task data. The task data includes at least the task identifier information, the client name that issued the task, the injected object key, the first MD5 hash of the injected file, the task operation type, the object storage address, the object storage account, the origin address information, the resource server address, the injection mode, the cloud storage bucket, the cloud storage node, and the object storage encryption password.

[0081] like Figure 3 As shown, optionally, the method further includes:

[0082] Perform parameter validation on the task data to obtain the validated task data;

[0083] Perform pre-distribution condition verification on the task data to obtain the condition verification results.

[0084] Optionally, based on the task data, obtain the resource files corresponding to the task data, including:

[0085] Based on the resource server address and origin address information, an HTTP request is sent to the origin service layer to retrieve the resource file.

[0086] Optionally, the method further includes:

[0087] Generate an authentication header based on the object storage address, object storage account, injected object key, and object storage encryption password;

[0088] According to the preset transmission method, the corresponding interface of the object storage is called to synchronize the resource files to the object storage space. The preset transmission method includes at least one of the following: streaming atomic upload method, streaming chunked upload method, or file-to-disk concurrent chunked upload method.

[0089] Optionally, the method further includes:

[0090] Obtain the second MD5 hash of the resource file;

[0091] Compare the first MD5 and the second MD5 to obtain the comparison result;

[0092] Based on the comparison results, determine whether to delete the resource file.

[0093] Specifically: 1. Pre-distribution interface layer, providing pre-distribution task issuance interface, task status query interface, and heartbeat detection interface.

[0094] 1.1 The pre-distribution task distribution interface receives HTTP requests from clients using the POST method. The received request body is a serialized JSON string. The interface processing flow is as follows:

[0095] (1) The JSON string is deserialized to obtain the task information, which includes the task ID, the name of the client who issued the task, the key of the injected object, the MD5 of the injected file, the operation type, the injection mode, the cloud storage bucket, the cloud storage endpoint, the object storage encryption AK, the last-modified of the source file, and the task callback address. See Table 1 for a detailed description.

[0096] (2) Verify the task parameters. If they pass, proceed to step (3); otherwise, proceed to step (5).

[0097] (3) Perform pre-distribution condition verification. If it passes, proceed to step (4); otherwise, proceed to step (5).

[0098] (4) Package the pre-injected task into an HTTP protocol request based on the task parameters;

[0099] (5) Send a synchronization response to the request initiator. If the verification in steps (2) and (3) passes, proceed to step (6); otherwise, end the process.

[0100] (6) Start a new thread to asynchronously process file injection or deletion tasks;

[0101] (7) Notify the customer of the task processing result via callback.

[0102] Table 1 Pre-distribution task information parameter table

[0103]

[0104]

[0105] The task parameter verification mentioned in step (2) of 1.1.1 is to verify the type and range of the task parameters.

[0106] 1.1.2 and 1.1 The pre-distribution condition verification described in step (3) is performed according to the injection_mode parameter value in the task parameters when the operation type is injection.

[0107] If inject_mode is 0, it is in forced injection mode, no condition validation is performed, and forced injection is executed;

[0108] If inject_mode is 1, then check if the stored file is up-to-date; if it is, then do not perform the injection.

[0109] If inject_mode is 2, then check whether the resource server file has been updated; if not, the injection will not be performed.

[0110] The method for verifying whether the object file is up-to-date in 1.1.2.1 and 1.1.2 is to send an HTTP request to the object storage with the HEAD method, carrying an "If-Modified-Since" header with the value of last_modified in the task parameters. If a 304 status code is obtained, it means that the file is up-to-date.

[0111] The methods for verifying whether the resource server file has been updated in 1.1.2.2 and 1.1.2 are to send an HTTP request to the resource server with the HEAD method, carrying an "If-Modified-Since" header with the value of last_modified in the task parameters. If a 304 status code is obtained, it means that the file has not been updated.

[0112] The synchronous response mentioned in step (5) of 1.1.3 and 1.1 is the pre-distribution request response sent by the client. The response status code 200 indicates that the task was successfully distributed, and the response status code 403 indicates that the task was not distributed. The reason for failure is responded to the client in the form of the body.

[0113] 1.1.4, 1.1 The file injection task described in step (6) is to send a file synchronization request from the pre-distribution interface layer to the file synchronization layer. If a 200 status code is received from the file synchronization layer, it means that the file upload object has been successfully stored. Optionally, a preheating request is sent to the CDN edge node.

[0114] The file synchronization request described in 1.1.4.1 and 1.1.4 is an HTTP request that carries task information and the request header "Oper-Type:sync". The request method is GET, and the task information is carried in the form of a request line or a request header.

[0115] The preheating request described in 1.1.4.2 and 1.1.4 is to trigger the CDN to autonomously retrieve the resource file that has just been successfully uploaded from the object storage and cache the file in the CDN edge node and parent node, thereby completing the pre-distribution of the file on the CDN.

[0116] 1.1.5, 1.1 The file deletion task described in step (6) is to send a file deletion request from the pre-distribution interface layer to the file synchronization layer. If a 200 status code is received from the file synchronization layer, it means that the file has been successfully deleted in the object storage. Optionally, a refresh request is sent to the CDN edge node.

[0117] The file deletion request described in 1.1.5.1 and 1.1.5 is an HTTP request that carries task information and the request header "Oper-Type: delete", with the request method being DELETE, and the task information being carried in the form of a request line or a request header.

[0118] The refresh request process described in 1.1.5.2 and 1.1.5 is to obtain the cache key corresponding to the deleted file in the CDN in the object storage, and delete the resource identified by the cache key from the CDN edge node and parent node.

[0119] 1.1.6, 1.1 The notification to the customer in step (7) is the server address where the customer receives the notification, which is the callback parameter value in the task parameters.

[0120] The task status query interface described in 1.2.1 uses the unique identifier taskid as the query key to query the real-time information of the corresponding task. The information includes the injected or deleted URL, task start time, task completion time, task time spent, task status, and reason for task failure.

[0121] The heartbeat detection interface described in 1.3.1 is used by external components to detect and determine whether the pre-distribution system is operating normally.

[0122] 2. The file synchronization layer parses the HTTP requests initiated by the pre-distribution interface layer, identifies the task operation type, and performs one of the following operations: synchronizes files from the resource server to object storage; deletes files in object storage. Furthermore, it provides rate limiting protection for upload traffic to object storage and back-to-origin traffic.

[0123] The parsing of request information initiated by the pre-distribution interface layer, as described in 2.1 and 2, refers to obtaining task information, including task operation type, object storage address, object storage account, object key, origin URL, and resource server address, from the request line and internal request glue header of the request information. See Table 2 for an explanation of the internal request glue header.

[0124] Table 2 Internal Request Adhesive Head Instructions

[0125]

[0126]

[0127] The operation type identified in 2.2.2 is obtained from the Opera-Type request header in the request message. The meaning of the Opera-Type value is referred to in the table above. SYNC and REMOVE correspond to the two operations mentioned above.

[0128] The detailed process for synchronizing to object storage as described in 2.3.2 is as follows:

[0129] (1) Based on the parsed resource server address and origin URL, initiate an HTTP request to the origin service layer to retrieve the resource file;

[0130] (2) Based on the parsed object storage address, object storage account, and object key, internally obtain the account's ak and sk, generate the authentication header, select the required upload method, call the corresponding interface provided by the object storage, and synchronize the file to the object storage.

[0131] (3) If the edge pre-distribution client carries a File-Md5 request header, after the file is successfully uploaded to the object storage, the md5 of the file in the storage is obtained and it is verified whether it is consistent with the value of the File-Md5 request header. If it is confirmed that the file md5 is inconsistent, the deletion interface provided by the object storage is called to delete the file and file synchronization fails.

[0132] (4) Successful synchronization will result in a 200 status code response from the client, while failure to synchronize will result in a 403 status code response.

[0133] The required upload methods described in steps (2) of 2.3.1 and 2.3 include streaming atomic upload, streaming chunked upload, and concurrent chunked upload to disk.

[0134] If the MD5 checksums described in steps (3) of 2.3.2 and 2.3 are inconsistent, a retry function is provided. If the checksums fail within the limit number of retries, the file MD5 checksums are confirmed to be inconsistent.

[0135] The deletion of the object storage file described in 2.4.2 involves obtaining the AK and SK of the account internally based on the parsed object storage address, object storage account, and object key, generating an authentication header, and calling the deletion interface provided by the object storage to delete the object from the object storage.

[0136] The rate limiting protection described in 2.5.2 supports rate limiting operations based on the leaky bucket algorithm and traffic control based on a second-level sliding time window, protecting object storage and resource servers.

[0137] 3. The aforementioned origin server service layer provides origin server information settings, 302 follow-up, and origin server response rewriting services, which customers can configure themselves according to their actual needs. Furthermore, a customer-defined programming service is provided as a supplement.

[0138] The origin information settings mentioned in 3.1 and 3 include origin parameter settings, origin request header settings, and origin encryption string generation, ensuring that the origin request can successfully retrieve the resource file through the origin site's anti-leeching and other access restrictions.

[0139] The 302 follow-up mentioned in 3.2 and 3 refers to the process where, if the origin service layer receives a 302 redirect response from the origin server, it will jump to the address corresponding to the 302 redirect to obtain the actual resource file before responding.

[0140] The source server response rewriting mentioned in 3.3 and 3 refers to the process of editing small files obtained from the source server at the source service layer, modifying the file content, and then responding to the file synchronization layer.

[0141] The small files mentioned in 3.3.1 and 3.3 shall not exceed 10M in size.

[0142] 3.4. The customer-defined programming described in section 3.3, based on the programmable CDN function, creates a sandbox environment to execute customer-written business code. Customers can use the provided interface functions and supported scripting languages ​​to write business code and independently implement business logic including setting origin request information, customized anti-hotlinking, authentication, rewriting, redirection, rate limiting, etc. Specific Implementation Example 1:

[0144] This embodiment is based on a file pre-distribution system applicable to multiple scenarios combining CDN and object storage, providing the following file injection method for customers who simultaneously access CDN and object storage. The timing diagram for this embodiment is attached. Figure 4 As shown, the method includes the following steps:

[0145] The S1 client initiates a pre-distribution task with operation type injection, injection mode forced injection, and preheating to CDN nodes through the pre-distribution task distribution interface provided by the file pre-distribution system. Specifically, the task parameters operation is 1, inject_mode is 0, and sync_cdn is 1.

[0146] The S2 file pre-distribution system processes pre-distribution tasks sent by clients. The internal processing flow is as follows:

[0147] The S21 pre-distribution interface layer receives HTTP requests with the POST method from the client, deserializes the JSON string in the request body to obtain the task parameters, performs parameter validation on each field of the pre-distribution task, skips the condition validation of the injection file in the forced injection mode, and after the validation passes, packages the pre-distribution task into an HTTP protocol request, starts an asynchronous thread to request the file synchronization layer, and synchronously responds to the client with the pre-distribution task distribution status.

[0148] The S22 file synchronization layer parses the HTTP request initiated by the pre-distribution interface layer. It extracts task information from the request line and internal request glue header, including the task operation type, object storage address, object storage account, object key, origin URL, and resource server address. It identifies the task type as synchronizing files from the resource server to object storage. Using the object storage address as the key, the file synchronization layer performs rate limiting on requests based on the leaky bucket algorithm if the rate limiting condition has been met.

[0149] The S23 file synchronization layer sends a request to the origin service layer to retrieve the resources to be uploaded.

[0150] When the S24 origin service layer receives a request from the synchronization layer, it matches the corresponding sub-channel configuration with the request information, executes the relevant origin information settings and 302 follow business logic. If the customer has used custom programming functions, it executes the customer-written business code in the sandbox environment, generates origin request information, and sends a request to the resource server to obtain resource files.

[0151] If the resource file does not need to be edited, the S25 will pass the response back to the origin service layer to the file synchronization layer; if the resource file needs to be edited, the S25 will execute the relevant text rewriting logic back to the origin service layer. If the customer has used custom programming functions, the customer's business code will be executed in the sandbox environment. After the text editing is completed, the edited file will be sent back to the file synchronization layer.

[0152] The S26 file synchronization layer retrieves the AK and SK of the client account based on the parsed object storage address, object storage account, and object key, generates an authentication header, and selects one of the following upload methods—streaming atomic upload, streaming chunked upload, or concurrent chunked upload to disk—to upload the file responded to by the origin service layer to the object storage.

[0153] S27 If the edge pre-distribution client carries a File-MD5 request header, after the file is successfully uploaded to object storage, the file synchronization layer obtains the MD5 of the file in storage and verifies whether it is consistent with the value of the File-MD5 request header. If it is confirmed that the file MD5 is inconsistent, the deletion interface provided by object storage is called to delete the file, and file synchronization fails.

[0154] The S28 file synchronization layer responds to the edge pre-distribution client injection result. A successful synchronization responds with a 200 status code, while a failed synchronization responds with a 403 status code.

[0155] The S29 pre-distribution interface layer receives a 200 status code response from the file synchronization layer, indicating that the file upload to object storage was successful. It then sends a preheating request to the CDN edge, triggering the CDN to autonomously retrieve the resource file that was just successfully uploaded from object storage. The file is then cached in the CDN edge node and the parent node, completing the pre-distribution of the file on the CDN node.

[0156] The S3 file pre-distribution system sends an HTTP request with the POST method to the address in the task parameter callback to notify the client of the task processing result. Specific Implementation Example 2:

[0158] This embodiment is based on a file pre-distribution system applicable to multiple scenarios under the combination of CDN and object storage, and provides the following file refresh method for customers who simultaneously access CDN and object storage. The method includes the following steps:

[0159] The S1 client initiates a pre-distribution task with the operation type of deletion and the requirement to refresh CDN nodes to the pre-distribution task distribution interface provided by the file pre-distribution system. That is, the task parameter operation is 0 and sync_cdn is 1.

[0160] The S2 file pre-distribution system processes pre-distribution tasks sent by clients. The internal processing flow is as follows:

[0161] The S21 pre-distribution interface layer receives HTTP requests with the POST method from the client, deserializes the JSON string in the request body to obtain the task parameters, validates the parameters of each field of the pre-distribution task, and after the validation is successful, packages the pre-distribution task into an HTTP protocol request, starts an asynchronous thread to request the file synchronization layer, and synchronously responds to the client with the pre-distribution task distribution status.

[0162] The S22 file synchronization layer parses the HTTP request initiated by the pre-distribution interface layer, and obtains task information including task operation type, object storage address, object storage account, object key, origin URL, and resource server address from the request line and internal request glue header of the request information. It identifies the task type as deleting the specified object from object storage.

[0163] The S23 file synchronization layer retrieves the AK and SK of the client account based on the parsed object storage address, object storage account, and object key, generates an authentication header, and calls the deletion interface provided by the object storage to delete the object.

[0164] The S24 file synchronization layer responds to the edge pre-distribution client injection result. A successful deletion responds with a 200 status code, while a failed deletion responds with a 403 status code.

[0165] The S25 pre-distribution interface layer receives a 200 status code response from the file synchronization layer, indicating successful object deletion. It then sends a refresh request to the CDN edge, deleting the corresponding cache in the CDN edge node and the parent node, thus completing the file refresh on the CDN node.

[0166] The S3 file pre-distribution system sends an HTTP request with the POST method to the address in the task parameter callback to notify the client of the task processing result. Specific Implementation Example 3:

[0168] This embodiment is based on a file pre-distribution system applicable to multiple scenarios combining CDN and object storage. It provides customers simultaneously accessing both CDN and object storage with the following method for automatically synchronizing customer object storage and resource server files, thus achieving mirroring of the origin server. The timing diagram for this embodiment is attached. Figure 5 As shown, the method includes the following steps:

[0169] The S1 CDN parent node's cache server determines whether the requested file meets the pre-distribution conditions. If it does, it initiates an injection-type pre-distribution task to the file pre-distribution system.

[0170] S11 satisfies the pre-distribution condition by satisfying any one of the following conditions:

[0171] (1) When the object storage is the primary source and the client resource server is the secondary source, after the cache server returns to the object storage with a 404 error on the first attempt, it returns to the resource server to retrieve the file, which has already been fully cached.

[0172] (2) When the client resource server is the primary source and the object storage is the secondary source, the cache server returns 200 when it retrieves the complete file from the primary source for the first time or when the cache expires.

[0173] The S12 cache server determines that the requested file meets the pre-distribution conditions and initiates a pre-distribution task to the file pre-distribution system. The operation type is injection, the injection mode is to verify whether the stored verification file is up-to-date, and the resource server address is the address of this cache server. That is, the task parameters operation is 1, injection_mode is 1, and origin_addr is the cache server address.

[0174] The S2 file pre-distribution system processes pre-distribution tasks sent by clients. The internal processing flow is as follows:

[0175] The S21 pre-distribution interface layer receives HTTP requests with the POST method from the client, deserializes the JSON string in the request body to obtain the task parameters, validates the parameters of each field of the pre-distribution task, and verifies whether the file in storage is up-to-date. After the verification is successful, the pre-distribution task is packaged into an HTTP protocol request, an asynchronous thread is started to request the file synchronization layer, and the pre-distribution task distribution status is synchronously responded to the client.

[0176] The S22 file synchronization layer parses the HTTP request initiated by the pre-distribution interface layer. It extracts task information from the request line and internal request glue header, including the task operation type, object storage address, object storage account, object key, origin URL, and CDN cache server address. It identifies the task type as synchronizing files from the CDN cache server to object storage. Using the object storage address as the key, the file synchronization layer performs rate limiting on requests based on the leaky bucket algorithm if the rate limiting condition has been met.

[0177] The S23 file synchronization layer initiates a request to the origin service layer to retrieve the resources to be uploaded. Upon receiving the request from the synchronization layer, the origin service layer sends a request to the CDN cache server to obtain the resource files and then passes the response to the file synchronization layer.

[0178] The S24 file synchronization layer retrieves the AK and SK of the client account based on the parsed object storage address, object storage account, and object key, generates an authentication header, and selects one of the following upload methods—streaming atomic upload, streaming chunked upload, or concurrent chunked upload to disk—to upload the file responded to by the origin service layer to the object storage.

[0179] The S25 file synchronization layer responds to the edge pre-distribution client injection result. A successful synchronization responds with a 200 status code, while a failed synchronization responds with a 403 status code.

[0180] The S26 pre-distribution interface layer received a 200 status code response from the file synchronization layer, indicating that the file upload object was successfully stored.

[0181] The S3 file pre-distribution system sends an HTTP request with the POST method to the address in the task parameter callback to notify the client of the task processing result.

[0182] This application provides a file pre-distribution system applicable to multiple scenarios by combining CDN and object storage. It employs a layered design, consisting of a pre-distribution interface layer, a file synchronization layer, and an origin service layer. The pre-distribution interface layer handles interactions with clients and CDN nodes, the file synchronization layer handles interactions with object storage, and the origin service layer handles interactions with client resource servers. This reduces system complexity, enhances scalability and reusability, strengthens the connection between object storage and CDN nodes in file pre-distribution, and provides an architectural foundation for multi-scenario applications.

[0183] It offers diverse pre-distribution options, including operation type, injection mode, whether file MD5 verification is required, and whether file injection or deletion is synchronized with CDN nodes. The system identifies the corresponding options through the task information parameters of the client's pre-distribution task, completing operations such as file injection condition verification, file consistency verification, file pre-injection into object storage and distribution to CDN nodes, and file deletion from object storage and refresh on CDN nodes. This provides a functional foundation for file pre-distribution in multiple scenarios combining CDN and object storage, offering integrated file pre-distribution and file refresh services, as well as mirror origin server services, to customers simultaneously accessing CDN and object storage. Further expansion to more application scenarios is possible in the future.

[0184] The system's origin server service layer provides self-configurable origin server information settings, 302 redirection, and origin server response rewriting services. Furthermore, it offers customer-defined programming services based on a programmable CDN as a supplement. Customers can self-configure or implement customized anti-leeching, customized origin server requests, m3u8 rewriting, and other requirements, reducing the workload of requirement development and improving requirement delivery efficiency.

[0185] This system can solve the problems of high human intervention, low integration of CDN and object storage, and poor scalability in traditional pre-distribution processing methods, and can be adapted to a variety of application scenarios combining CDN and object storage.

[0186] Another embodiment of this application provides a file distribution apparatus based on CDN and object storage for executing the file distribution method based on CDN and object storage provided in the above embodiments.

[0187] like Figure 6 The diagram shown is a structural schematic of a file distribution device based on CDN and object storage provided in an embodiment of this application. This CDN and object storage-based file distribution device includes a receiving module 601, a parsing module 602, an acquisition module 603, an allocation module 604, and a sending module 605, wherein:

[0188] The receiving module 601 is used to receive multi-scenario file pre-distribution task requests sent by the client terminal, wherein the task requests include at least injection, deletion, preheating or refresh;

[0189] The parsing module 602 is used to parse the task request and determine the task data corresponding to the task request;

[0190] The acquisition module 603 is used to acquire the resource file corresponding to the task data based on the task data;

[0191] The allocation module 604 is used to allocate resource files to object storage, and then to the parent nodes and edge nodes of CDN nodes to obtain the allocation results of resource files;

[0192] The sending module 605 is used to send the allocation result to the client terminal.

[0193] Regarding the apparatus in this embodiment, the specific manner in which each module performs its operations has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0194] This application provides a file distribution device based on CDN and object storage, comprising: receiving multi-scenario file pre-distribution task requests sent by client terminals, wherein the task requests include at least injection, deletion, preheating, or refresh; parsing the task requests to determine the task data corresponding to the task requests; obtaining resource files corresponding to the task data based on the task data; allocating the resource files to object storage, and then to the parent nodes and edge nodes of CDN nodes to obtain the allocation results of the resource files; and sending the allocation results to the client terminals. This application embodiment integrates CDN and object storage to meet the file pre-distribution needs of clients, satisfying various file pre-distribution application scenarios, and aims to solve the problems of high manual intervention, low integration of CDN and object storage, and poor scalability in the current pre-distribution processing methods.

[0195] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions involved are not necessarily required by this application.

[0196] Optionally, this application also provides an electronic device, including: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the various processes of the above method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0197] This application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0198] The figure is a block diagram of an electronic device 800 as shown in the seven applications. For example, the electronic device 800 can be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0199] Reference Figure 7 The electronic device 800 may include one or more of the following components: a processing component 802, a memory 804, a power supply component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.

[0200] Processing component 802 typically controls the overall operation of electronic device 800, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the methods described above. Furthermore, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

[0201] Memory 804 is configured to store various types of data to support the operation of device 800. Examples of this data include instructions for any application or method operating on electronic device 800, contact data, phonebook data, messages, images, videos, etc. Memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0202] Power supply component 806 provides power to various components of electronic device 800. Power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 800.

[0203] Multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 808 includes a front-facing camera and / or a rear-facing camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0204] Audio component 810 is configured to output and / or input audio signals. For example, audio component 810 includes a microphone (MIC) configured to receive external audio signals when electronic device 800 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 804 or transmitted via communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

[0205] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0206] Sensor assembly 814 includes one or more sensors for providing state assessments of various aspects of electronic device 800. For example, sensor assembly 814 may detect the on / off state of device 800, the relative positioning of components such as the display and keypad of electronic device 800, changes in position of electronic device 800 or a component of electronic device 800, the presence or absence of user contact with electronic device 800, orientation or acceleration / deceleration of electronic device 800, and temperature changes of electronic device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 814 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0207] Communication component 816 is configured to facilitate wired or wireless communication between electronic device 800 and other devices. Electronic device 800 can access wireless networks based on communication standards, such as WiFi, carrier networks (such as 2G, 3G, 4G, or 5G), or combinations thereof. In one exemplary embodiment, communication component 816 receives broadcast signals or broadcast operation information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 816 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0208] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0209] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, which can be executed by a processor 820 of an electronic device 800 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0210] Figure 8This is a block diagram illustrating a computer-readable storage medium 1900. For example, the computer-readable storage medium 1900 can be provided as a server.

[0211] Reference Figure 8 The computer-readable storage medium 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by memory 1932 for storing instructions executable by the processing component 1922, such as an application program. The application program stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0212] The computer-readable storage medium 1900 may also include a power supply component 1926 configured to perform power management of the computer-readable storage medium 1900, a wired or wireless network interface 1950 configured to connect the computer-readable storage medium 1900 to a network, and an input / output (I / O) interface 1958. The computer-readable storage medium 1900 can operate on an operating system stored in memory 1932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or similar.

[0213] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

[0214] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0215] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A file distribution method based on CDN and object storage, characterized in that, The method includes: Receive a multi-scenario file pre-distribution task request sent by a client terminal, wherein the task request includes at least injection, deletion, preheating or refresh; The task request is parsed to determine the task data corresponding to the task request; Based on the task data, obtain the resource file corresponding to the task data; The resource file is allocated to object storage, and then to the parent node and edge node of the CDN node to obtain the allocation result of the resource file; The allocation result is sent to the client terminal; The step of parsing the task request to determine the task data corresponding to the task request includes: The task request is deserialized to obtain task data, which includes at least the task identifier information, the client name that issued the task, the injected object key, the first MD5 hash of the injected file, the task operation type, the object storage address, the object storage account, the origin address information, the resource server address, the injection mode, the cloud storage bucket, the cloud storage node, and the object storage encryption password.

2. The file distribution method based on CDN and object storage according to claim 1, characterized in that, The method further includes: The task data is validated to obtain valid task data. The pre-distribution conditions are checked on the verified task data to obtain the condition check results.

3. The file distribution method based on CDN and object storage according to claim 2, characterized in that, The method further includes: The task data is verified, and if it passes, the pre-distribution conditions are verified. If the pre-distribution condition verification is passed, the pre-injected task is packaged into an HTTP protocol request based on the task data. Start a new thread to asynchronously process file injection or deletion tasks; The task processing result is returned to the client terminal via callback.

4. The file distribution method based on CDN and object storage according to claim 1, characterized in that, The step of obtaining the resource file corresponding to the task data includes: Based on the resource server address and the origin address information, an HTTP request is initiated to the origin service layer to obtain the resource file.

5. The file distribution method based on CDN and object storage according to claim 1, characterized in that, The method further includes: Generate an authentication header based on the object storage address, object storage account, injected object key, and object storage encryption password; According to the preset transmission method, the corresponding interface of the object storage is called to synchronize the resource file to the object storage space. The preset transmission method includes at least one of the following: streaming atomic upload method, streaming chunked upload method, or file disk-based concurrent chunked upload method.

6. The file distribution method based on CDN and object storage according to claim 5, characterized in that, The method further includes: obtaining the second MD5 hash of the resource file; The first MD5 and the second MD5 are compared to obtain the comparison result; Based on the comparison results, determine whether to delete the resource file.

7. A file distribution device based on CDN and object storage, characterized in that, The device includes: The receiving module is used to receive multi-scenario file pre-distribution task requests sent by the client terminal, wherein the task requests include at least injection, deletion, preheating or refresh; The parsing module is used to parse the task request and determine the task data corresponding to the task request; The acquisition module is used to acquire the resource file corresponding to the task data based on the task data; The allocation module is used to allocate the resource file to object storage, and then to the parent node and edge node of the CDN node to obtain the allocation result of the resource file; The sending module is used to send the allocation result to the client terminal; The parsing module includes: The task request is deserialized to obtain task data, which includes at least the task identifier information, the client name that issued the task, the injected object key, the first MD5 hash of the injected file, the task operation type, the object storage address, the object storage account, the origin address information, the resource server address, the injection mode, the cloud storage bucket, the cloud storage node, and the object storage encryption password.

8. An electronic device, characterized in that, include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 6.

Images