Data transmission method and device based on content distribution network, equipment and medium
By utilizing data caching graphs within the CDN network to determine the transmission path of target data, the problem of slow CDN data transmission is solved, resulting in more efficient data transmission and a better user experience.
Patent Information
- Application Number
- CN202511180941.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-22
- Publication Date
- 2025-11-11
AI Technical Summary
The existing CDN data transmission suffers from slow data transmission, resulting in a degraded user experience.
By utilizing the data cache graph to identify the CDN nodes for the target data and transmitting the target data from the CDN nodes to the target terminal, data transmission efficiency is improved.
It improves data transmission efficiency, reduces user waiting time, and enhances user experience.
Smart Images
Figure CN120935280A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data transmission technology, and in particular to data transmission methods, apparatus, devices and media based on content delivery networks. Background Technology
[0002] CDN (Content Delivery Network) is an intelligent virtual network built on top of the existing network infrastructure. Relying on edge servers deployed in various locations, it enables users to obtain the content they need from the nearest location through the load balancing, content distribution, and scheduling functions of the central platform, thereby reducing network congestion and improving user access response speed and hit rate.
[0003] Currently, data transmission using CDN typically relies on statically configured edge node caching strategies and fixed routing rules. This data transmission method suffers from slow data transmission, which reduces the user experience. Therefore, improving data transmission efficiency when using CDN has become a technical problem that needs to be solved. Summary of the Invention
[0004] In view of this, the purpose of this invention is to provide a data transmission method, apparatus, device, and medium based on a content delivery network, which can determine a second target CDN node containing the target data by utilizing a data cache graph, and transmit the target data from the second target CDN node to the target terminal, thereby improving the efficiency of data transmission. The specific solution is as follows:
[0005] Firstly, this application provides a data transmission method based on a content delivery network, comprising:
[0006] Based on a preset time interval, real-time data access requests from the target terminal are obtained, target feature information corresponding to each real-time data access request is determined, and the target feature information and target content prediction model are used to predict the data to be accessed within a preset time period with an access probability not less than a preset probability threshold.
[0007] The data to be accessed is obtained from the target data source, cached in the first target CDN node corresponding to the target terminal, and a preset data cache map is dynamically updated based on the physical distance and network topology between CDN nodes and the data access records in each CDN node; wherein, the preset data cache map includes the node ID of each CDN node, the data ID of the network data stored in each CDN node, and the physical distance between each CDN node;
[0008] The current data access request of the target terminal is obtained, and it is determined whether the target data corresponding to the current data access request exists in the data to be accessed. If the target data corresponding to the current data access request exists in the data to be accessed, the target data is directly transmitted from the first target CDN node to the target terminal.
[0009] If the target data corresponding to the current data access request does not exist in the data to be accessed, the second target CDN node corresponding to the target data is determined using the preset data cache map, and the target data is transmitted from the second target CDN node to the target terminal.
[0010] Optionally, before using the target feature information and the target content prediction model to predict the data to be accessed within a preset time period with an access probability not less than a preset probability threshold, the method further includes:
[0011] Obtain historical data access requests corresponding to the target terminal, construct a target training set using each historical data access request, and set a target learning rate corresponding to the preset Adam optimizer;
[0012] The target content prediction model is trained based on the target training set, the target learning rate, and the target loss function to obtain the target content prediction model; wherein the target content prediction model is a two-layer LSTM structure.
[0013] Optionally, caching the data to be accessed to the first target CDN node corresponding to the target terminal includes:
[0014] Determine whether the total data volume of each of the data to be accessed is greater than the target data capacity of the first target CDN node. If the total data volume of each of the data to be accessed is not greater than the target data capacity of the first target CDN node, then directly cache each of the data to be accessed to the first target CDN node.
[0015] If the total data volume of each of the data to be accessed is greater than the target data capacity of the first target CDN node, then the access probability and data volume corresponding to each of the data to be accessed are determined, the target ratio of the access probability to the data volume of each of the data to be accessed is calculated, and data to be cached is selected from each of the data to be accessed based on the target ratio, and each of the data to be cached is cached to the first target CDN node.
[0016] Optionally, determining the second target CDN node corresponding to the target data using the preset data cache map includes:
[0017] The target physical distance between each CDN node and the first target CDN node is determined using the preset data cache map, and the target network hop count between each CDN node and the first target CDN node is also determined.
[0018] The second target CDN node is determined from the CDN nodes based on the physical distance to each target and the number of network hops to each target.
[0019] Optionally, transmitting the target data from the second target CDN node to the target terminal includes:
[0020] Determine whether the data volume of the target data is less than a preset data volume threshold. If the data volume of the target data is not less than the preset data volume threshold, determine the bandwidth of each data transmission path between the target terminal and the second target CDN node, perform fragmentation processing on the target data according to the bandwidth of each data transmission path to obtain corresponding target data fragments, and use each data transmission path to transmit each target data fragment from the second target CDN node to the target terminal.
[0021] Optionally, transmitting the target data from the second target CDN node to the target terminal includes:
[0022] If the data volume of the target data is less than the preset data volume threshold, then based on the target node status and target routing decision model corresponding to each CDN node, the target data transmission path is dynamically determined from each data transmission path, and the target data is transmitted from the second target CDN node to the target terminal using the target data transmission path; wherein, the target node status includes the CPU utilization, memory usage, and bandwidth utilization corresponding to each CDN node.
[0023] Optionally, after transmitting the target data from the second target CDN node to the target terminal, the method further includes:
[0024] The target data fragments received by the target terminal are verified to be complete based on the target hash values corresponding to each target data fragment. If the target data fragments received by the target terminal are incomplete, the missing data fragments in the target data fragments are transmitted to the target terminal using a preset backup data transmission path.
[0025] Secondly, this application provides a data transmission apparatus based on a content delivery network, comprising:
[0026] The data prediction module is used to acquire real-time data access requests from the target terminal based on a preset time interval, determine the target feature information corresponding to each real-time data access request, and use the target feature information and the target content prediction model to predict the data to be accessed within a preset time period with an access probability not less than a preset probability threshold.
[0027] The data cache update module is used to obtain the data to be accessed from the target data source, cache the data to be accessed in the first target CDN node corresponding to the target terminal, and dynamically update the preset data cache map based on the physical distance and network topology between each CDN node and the data access records in each CDN node; wherein, the preset data cache map includes the node ID of each CDN node, the data ID of the network data stored in each CDN node, and the physical distance between each CDN node;
[0028] The first data transmission module is used to obtain the current data access request of the target terminal, determine whether the target data corresponding to the current data access request exists in the data to be accessed, and if the target data corresponding to the current data access request exists in the data to be accessed, then the target data is directly transmitted from the first target CDN node to the target terminal.
[0029] The second data transmission module is used to determine the second target CDN node corresponding to the target data using the preset data cache map if the target data corresponding to the current data access request does not exist in the data to be accessed, and to transmit the target data from the second target CDN node to the target terminal.
[0030] Thirdly, this application provides an electronic device, comprising:
[0031] Memory, used to store computer programs;
[0032] A processor for executing the computer program to implement the aforementioned data transmission method based on a content delivery network.
[0033] Fourthly, this application provides a computer-readable storage medium for storing a computer program, which, when executed by a processor, implements the aforementioned data transmission method based on a content delivery network.
[0034] This application first acquires real-time data access requests from the target terminal based on a preset time interval, determines the target feature information corresponding to each real-time data access request, and uses the target feature information and a target content prediction model to predict data to be accessed with an access probability not less than a preset probability threshold within a future preset time period. Then, it acquires the data to be accessed from the target data source, caches the data to be accessed in the first target CDN node corresponding to the target terminal, and dynamically updates a preset data cache map based on the physical distance and network topology between each CDN node and the data access records in each CDN node. The preset data cache map includes the node ID of each CDN node, and the data access records of each CDN node... The system stores the data ID of the network data and the physical distance between each CDN node. Then, it obtains the current data access request of the target terminal and determines whether the target data corresponding to the current data access request exists in the data to be accessed. If the target data corresponding to the current data access request exists in the data to be accessed, the target data is directly transmitted from the first target CDN node to the target terminal. If the target data corresponding to the current data access request does not exist in the data to be accessed, the second target CDN node corresponding to the target data is determined using the preset data cache map, and the target data is transmitted from the second target CDN node to the target terminal. Therefore, this application improves the speed at which users obtain data by acquiring several real-time data access requests corresponding to the target terminal and using the real-time data access requests and content prediction models to predict the data that users may access in the future. The data to be accessed is then preloaded into the first target CDN node corresponding to the target terminal. If the target data required by the user is not found in the first target CDN node, the data cache graph is used to determine the second target CDN node containing the target data, and the target data in the second target CDN node is transmitted to the target terminal. This improves the coordination effect between CDN nodes, avoids directly obtaining data from the target data source, and thus improves the efficiency of data transmission. Attached Figure Description
[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0036] Figure 1 This is a flowchart of a data transmission method based on a content delivery network disclosed in this application;
[0037] Figure 2This is a schematic diagram of a data transmission method based on a content delivery network disclosed in this application;
[0038] Figure 3 This is a schematic diagram of a dynamic scaling mechanism disclosed in this application;
[0039] Figure 4 This is a schematic diagram of a data transmission device based on a content delivery network disclosed in this application;
[0040] Figure 5 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation
[0041] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0042] Currently, methods using CDNs for data transmission suffer from slow data transfer speeds, negatively impacting the user experience. To address this, this application provides a data transmission method based on a content delivery network. This method utilizes a data cache graph to identify a second target CDN node containing the target data, and then transmits the target data from this second target CDN node to the target terminal, thereby improving data transmission efficiency.
[0043] See Figure 1 As shown, this embodiment of the invention discloses a data transmission method based on a content delivery network, including:
[0044] Step S11: Obtain real-time data access requests from the target terminal based on a preset time interval, determine the target feature information corresponding to each real-time data access request, and use the target feature information and the target content prediction model to predict the data to be accessed within a preset time period with an access probability not less than a preset probability threshold.
[0045] In this embodiment, the process for processing user requests is as follows: Figure 2 As shown, the edge node receives the user's response and checks the local cache. If the cache is hit, it responds directly. If the cache is not hit, it needs to query the data cache graph, select the optimal node, and use intelligent routing technology to transmit the target data in the optimal node to the target terminal.
[0046] This embodiment designs a dynamic collaborative caching mechanism: it utilizes a lightweight machine learning model (i.e., a target content prediction model), such as LSTM (Long Short-Term Memory), to analyze the spatiotemporal characteristics of user requests, predict high-probability access content within the next hour, and pushes it to the edge nodes of the target region in advance. Simultaneously, it designs a cross-node cache heatmap (i.e., a data cache graph), allowing edge nodes to pull content from adjacent nodes on demand, reducing the number of times they need to retrieve data from the origin server.
[0047] In this embodiment, before using target feature information and target content prediction model to predict data to be accessed within a future preset time period with an access probability not less than a preset probability threshold, the method further includes: obtaining historical data access requests corresponding to the target terminal, constructing a target training set using each historical data access request, and setting a target learning rate corresponding to a preset Adam optimizer; training a preset initial content prediction model based on the target training set, target learning rate, and target loss function to obtain a target content prediction model; wherein, the target content prediction model is a two-layer LSTM structure.
[0048] Specifically, the model architecture and training process of the LSTM-based hot topic content prediction model (i.e., the target content prediction model) are as follows:
[0049] Input features: user request timestamp (converted to 24-hour periodic sine code), content ID (IdentityDocument) hash value (128-dimensional embedding vector), geolocation (GeoHash encoded as a 10-character string), device type (mobile / PC classification code).
[0050] Network structure: It adopts a two-layer LSTM (64 units per layer) followed by a fully connected layer (the output dimension is the probability distribution of Top-50 hot content).
[0051] Training data: Use request logs from the past 30 days (i.e., historical data access requests), slice them by hour, and construct time series samples with a sequence length of 6 (e.g., use data from time t-5 to time t-1 to predict hotspots at time t).
[0052] Loss function: Cross-entropy loss + Regularization term (L2 coefficient) =0.01), the optimizer uses Adam (learning rate) =0.001).
[0053] In this embodiment, the process of collecting real-time data access requests and using target feature information and a target content prediction model to predict the data to be accessed within a preset time period with an access probability not less than a preset probability threshold is as follows:
[0054] 1. Real-time feature generation: Edge nodes aggregate requested data (i.e., real-time data access requests) every 5 minutes and generate standardized feature vectors.
[0055] 2. Prediction Execution: Input the features (i.e. target feature information) into the LSTM model and output the access probability of each content ID in the next hour. If the probability is greater than the preset probability threshold, such as 0.8, it is determined as data to be accessed.
[0056] By using content prediction models to predict the data that users may access in the future, the data that users may access can be loaded into the cache in advance in subsequent processes, reducing data loading time and lowering the waiting time for users during data access.
[0057] Step S12: Obtain the data to be accessed from the target data source, cache the data to be accessed in the first target CDN node corresponding to the target terminal, and dynamically update the preset data cache map based on the physical distance and network topology between each CDN node and the data access records in each CDN node; wherein, the preset data cache map includes the node ID of each CDN node, the data ID of the network data stored in each CDN node, and the physical distance between each CDN node.
[0058] In this embodiment, the process of caching the data to be accessed to the first target CDN node corresponding to the target terminal may specifically include: determining whether the total data volume of each data to be accessed is greater than the target data capacity of the first target CDN node; if the total data volume of each data to be accessed is not greater than the target data capacity of the first target CDN node, then directly caching each data to be accessed to the first target CDN node; if the total data volume of each data to be accessed is greater than the target data capacity of the first target CDN node, then determining the access probability and data volume corresponding to each data to be accessed, calculating the target ratio of the access probability to the data volume of each data to be accessed, filtering out the data to be cached from the data to be accessed based on the target ratio, and caching each data to be cached to the first target CDN node.
[0059] The above process is as follows: for content with an access probability greater than or less than a preset probability threshold, such as 0.8, if it is not cached locally and the storage space (i.e., the target data capacity) is sufficient, it is pulled from the upper-level node or the origin server; if the space is insufficient, low-priority content is eliminated according to the "popularity-volume" ratio, i.e., the target ratio (popularity / file size).
[0060] By determining the data to be stored in the target cache space based on the ratio of the popularity to the size of the data to be accessed, the impact of data popularity and size on the data caching process is balanced, avoiding the problem of a small amount of large data occupying cache space, which would reduce the hit rate of the user's current data access request in the local first CDN node.
[0061] In this embodiment, the preset data cache graph includes the node ID of each CDN node, the data ID of the network data stored in each CDN node, and the physical distance between each CDN node. The specific structure of the data cache graph is shown below:
[0062] Node Relationship Table: Records the physical distance (based on IP geography database) and network topology hop count between edge nodes.
[0063] Content distribution matrix: with content ID as the row and node ID as the column, it records whether each node is currently caching the content and the most recent access timestamp.
[0064] Popularity weight calculation: The popularity weight H of a single piece of content is dynamically updated by the following formula:
[0065] H = *(Access frequency / Total number of requests) + *(1 / (current time - last access time + 1)) + *(Related content recommendation weight)
[0066] in =0.6, =0.3, =0.1, used to balance real-time performance with long-term trends.
[0067] In addition, this embodiment also designs a resource elastic scheduling framework. When storing data, containerized virtual resource pools are deployed on edge nodes, and computing / storage resources are automatically expanded according to traffic peaks. A "hot and cold resource layer" is set up to transfer low-frequency content to lower-cost near-edge storage, saving core node resources. Specifically, this embodiment designs containerized resource pool management:
[0068] Horizontal scaling (Scale-Out): When a node's CPU utilization is >80% for 2 minutes, the Kubernetes control center schedules idle container instances from the resource pool (priority: same region > neighboring region > central cloud).
[0069] Scale-Up: If the CPU / memory of the current instance cannot be increased, a new instance is created (upgraded by one level), and tasks are migrated gradually (using blue-green deployment, with traffic to the old instance decreasing by 10%).
[0070] Scaling down conditions: If CPU utilization is less than 30% for 5 consecutive minutes, release 20% of redundant instances (prioritizing the release of stateless service instances).
[0071] The scaling decision process in this embodiment is as follows: Figure 3 As shown, the CPU or memory trend is first analyzed to determine whether it exceeds the preset threshold, and the scaling API (Application Programming Interface) is triggered to perform scaling operations based on the corresponding judgment result.
[0072] In addition, this embodiment also designs a cold and hot tiered storage architecture:
[0073] Hot-layer SSD (Solid State Drive): Stores content accessed within the last 7 days, using RAID (Redundant Array of Independent Disks) with a 10-array configuration to guarantee IOPS (Input / Output Operations Per Second) > 50k.
[0074] Level 3 HDD (Hard-Disk Drive): Stores content accessed within 7-30 days, and saves 40% of space by enabling compression algorithms (Zstandard Level 3).
[0075] Cold layer (near edge storage pool): Stores content that has not been accessed for more than 30 days, and uses erasure coding (Erasure Coding 6+3) to reduce storage costs.
[0076] Migration triggering mechanism:
[0077] Downgrade migration: The migration task starts at 2:00 AM every day, moving eligible content from the hot layer to the warm layer, and then to the cold layer.
[0078] Upgrade migration: When cold layer content is accessed, the backend asynchronously copies it back to the hot layer and updates the metadata to mark it as "active".
[0079] Step S13: Obtain the current data access request of the target terminal, determine whether the target data corresponding to the current data access request exists in the data to be accessed, and if the target data corresponding to the current data access request exists in the data to be accessed, then directly transmit the target data from the first target CDN node to the target terminal.
[0080] In this embodiment, if the target data corresponding to the current data access request exists in the data to be accessed, it proves that the prediction result of the target content prediction model is correct. Since the target data has been pre-loaded from the source database into the first target CDN node corresponding to the target terminal, the target terminal can directly obtain the target data from the first target CDN node closest to the local terminal without having to call the target data from the source database in a timely manner. This greatly reduces the time required for data transmission, thereby reducing the user's waiting time and improving the user experience.
[0081] Step S14: If the target data corresponding to the current data access request does not exist in the data to be accessed, then the second target CDN node corresponding to the target data is determined using the preset data cache map, and the target data is transmitted from the second target CDN node to the target terminal.
[0082] In this embodiment, the process of determining the second target CDN node corresponding to the target data using a preset data cache map may specifically include: determining the target physical distance between each CDN node and the first target CDN node using the preset data cache map, and determining the target network hop count between each CDN node and the first target CDN node; and determining the second target CDN node from each CDN node based on each target physical distance and each target network hop count.
[0083] Specifically, when the local node (i.e., the first target CDN node) cannot hit the request, neighboring nodes are queried according to the following priority:
[0084] 1. One-hop node: A node with a physical distance of less than 50km and a hop count of no more than 2, and a response latency threshold of less than 50ms.
[0085] 2. Two-hop nodes: Nodes with a physical distance of less than 200km or a hop count of no more than 4, and a latency threshold of less than 120ms.
[0086] It uses gRPC (Remote Procedure Call) long-connection to transmit content, supporting breakpoint resumption and parallel chunked download (default chunk size = 4MB).
[0087] That is, in this embodiment, the first step is to query each of the second target CDN nodes whose distance from the first target CDN node is less than a first preset distance threshold, whose network hop count is not greater than a first preset network hop count threshold, and whose response latency is not greater than a first preset latency threshold. If none of the second target CDN nodes contain the target data corresponding to the current data access requirement, then the target data is queried from each of the second target CDN nodes whose distance from the first target CDN node is less than a second preset distance threshold, whose network hop count is not greater than a second preset network hop count threshold, and whose response latency is not greater than a second preset latency threshold.
[0088] It is understood that the first preset distance threshold is less than the second preset distance threshold, the first preset network hop count threshold is less than the second preset network hop count threshold, and the first preset response delay threshold is less than the second preset response delay threshold. In this embodiment, the first preset distance threshold, the second preset distance threshold, the first preset network hop count threshold, the second preset network hop count threshold, the first preset delay threshold, and the second preset delay threshold can all be determined according to actual needs. The specific thresholds mentioned above are only examples.
[0089] Furthermore, this embodiment also designs an intelligent routing algorithm based on reinforcement learning: constructing a real-time state matrix (including load, bandwidth, and latency) between edge nodes, and dynamically selecting the optimal transmission path through a Q-Learning model (a reinforcement learning model). For large files such as video streams, a fragmented multi-path transmission strategy is adopted to improve throughput.
[0090] Correspondingly, the process of transmitting target data from the second target CDN node to the target terminal may specifically include: determining whether the data volume of the target data is less than a preset data volume threshold; if the data volume of the target data is not less than the preset data volume threshold, determining the bandwidth of each data transmission path between the target terminal and the second target CDN node; performing fragmentation processing on the target data according to the bandwidth of each data transmission path to obtain corresponding target data fragments; and using each data transmission path to transmit each target data fragment from the second target CDN node to the target terminal.
[0091] Specifically, for files larger than a preset data size threshold, such as 100MB, multipath transmission is enabled. The number of fragments N = min(number of available paths, ceil(file size / 50MB)). Fragment size is dynamically adjusted: based on the real-time bandwidth B_i of each path, the allocated data block size is: BlockSize_i = (B_i / ... B_j) * file size, ensuring that high-bandwidth paths handle more data transmission.
[0092] In this embodiment, the process of transmitting target data from the second target CDN node to the target terminal may specifically include: if the data volume of the target data is less than a preset data volume threshold, then based on the target node status corresponding to each CDN node and the target routing decision model, the target data transmission path is dynamically determined from each data transmission path, and the target data is transmitted from the second target CDN node to the target terminal using the target data transmission path; wherein, the target node status includes the CPU utilization, memory usage, and bandwidth utilization corresponding to each CDN node.
[0093] The process of selecting the target data transmission path described above is handled by the routing decision model. Specifically, the state space definition of the Q-Learning-based routing decision model is as follows:
[0094] Each edge node (i.e., CDN node) maintains a state vector S = [CPU utilization, memory usage, current bandwidth utilization, RTT (Round-Trip Time) to the target node].
[0095] Discretization: Divide each dimension into 4 levels (e.g., CPU utilization: 0-25% is "low", 25-50% is "medium", 50-75% is "high", and 75-100% is "overload"), and the total number of states = 4 to the power of 4 = 256.
[0096] Action space and reward function:
[0097] Action selection: Select a node from the set of available next-hop nodes (or terminate the action if the target is reached directly).
[0098] Instant Rewards R:
[0099] R = w1*(1 / latency)+w2*(1-packet loss rate)-w3*node load penalty, where w1=0.5, w2=0.3, w3=0.2, and load penalty=0.2*(CPU utilization+memory utilization) / 2.
[0100] Q table update rules:
[0101] Learning rate =0.1, discount factor =0.9, -greedy strategy (exploration rate) The initial value is 0.3, which decreases by 10% every 1000 cycles.
[0102] Updated formula:
[0103] Q(S,A)←Q(S,A) + *[R + *max(Q(S',A')) - Q(S,A)];
[0104] In this embodiment, after transmitting the target data from the second target CDN node to the target terminal, the method further includes: verifying whether the target data fragments received by the target terminal are complete according to the target hash values corresponding to each target data fragment; if the target data fragments received by the target terminal are incomplete, then transmitting the missing data fragments in the target data fragments to the target terminal using a preset backup data transmission path.
[0105] Specifically, hash verification: Each fragment is appended with a SHA-256 hash value, and the receiver verifies its integrity. Timeout retransmission: If a fragment does not arrive within the expected time (=2*RTT + fragment size / bandwidth), a retransmission via an alternative path is triggered. Parallel reassembly: The receiver uses memory-mapped file technology, allowing fragments to be written directly to a specified location on disk even when they arrive out of order.
[0106] In addition, if the LSTM model fails to predict (e.g., due to abnormal input data) in this embodiment, it switches to the backup strategy:
[0107] Sliding window statistics: Sort by the frequency of actual requests in the last hour and cache the top-N content (N = local storage capacity / average file size).
[0108] Based on association rules: Apriori algorithm is used to mine user access sequences and cache frequently co-occurring content groups (e.g., if A is accessed, B may also be accessed).
[0109] It should be noted that this embodiment also includes corresponding countermeasures for node failures:
[0110] Health check: Sends a heartbeat packet every 10 seconds to check the node status. If there is no response for 3 consecutive times, it is marked as "unavailable".
[0111] Traffic switching: The requests from the faulty node are distributed to other available nodes according to the weights of the Q-Learning model, and the unfinished fragment transmission tasks are recorded and reallocated by the central scheduler.
[0112] Therefore, this application improves the speed at which users obtain data by acquiring several real-time data access requests corresponding to the target terminal and using the real-time data access requests and content prediction models to predict the data that users may access in the future. The data to be accessed is then preloaded into the first target CDN node corresponding to the target terminal. If the target data required by the user is not found in the first target CDN node, the data cache graph is used to determine the second target CDN node containing the target data, and the target data in the second target CDN node is transmitted to the target terminal. This improves the coordination effect between CDN nodes, avoids directly obtaining data from the target data source, and thus improves the efficiency of data transmission.
[0113] See Figure 4 As shown, this embodiment of the invention discloses a data transmission device based on a content delivery network, comprising:
[0114] Data prediction module 11 is used to acquire real-time data access requests from target terminals based on a preset time interval, determine the target feature information corresponding to each real-time data access request, and use the target feature information and target content prediction model to predict the data to be accessed within a preset time period with an access probability not less than a preset probability threshold.
[0115] The data cache update module 12 is used to obtain the data to be accessed from the target data source, cache the data to be accessed in the first target CDN node corresponding to the target terminal, and dynamically update the preset data cache map based on the physical distance and network topology between each CDN node and the data access records in each CDN node; wherein, the preset data cache map includes the node ID of each CDN node, the data ID of the network data stored in each CDN node, and the physical distance between each CDN node;
[0116] The first data transmission module 13 is used to obtain the current data access request of the target terminal, determine whether there is target data corresponding to the current data access request in the data to be accessed, and if there is target data corresponding to the current data access request in the data to be accessed, then directly transmit the target data from the first target CDN node to the target terminal.
[0117] The second data transmission module 14 is used to determine the second target CDN node corresponding to the target data using the preset data cache map if the target data corresponding to the current data access request does not exist in the data to be accessed, and to transmit the target data from the second target CDN node to the target terminal.
[0118] In some specific embodiments, the data prediction module 11 further includes:
[0119] The training set construction unit is used to obtain the historical data access requests corresponding to the target terminal, construct the target training set using each of the historical data access requests, and set the target learning rate corresponding to the preset Adam optimizer.
[0120] The model training unit is used to train a preset initial content prediction model based on the target training set, the target learning rate, and the target loss function to obtain the target content prediction model; wherein the target content prediction model is a two-layer LSTM structure.
[0121] In some specific embodiments, the map update module 12 may specifically include:
[0122] The first data caching unit is used to determine whether the total data volume of each of the data to be accessed is greater than the target data capacity of the first target CDN node. If the total data volume of each of the data to be accessed is not greater than the target data capacity of the first target CDN node, then each of the data to be accessed is directly cached to the first target CDN node.
[0123] The second data caching unit is configured to, if the total data volume of each of the data to be accessed is greater than the target data capacity of the first target CDN node, determine the access probability and data volume corresponding to each of the data to be accessed, calculate the target ratio of the access probability to the data volume of each of the data to be accessed, filter out data to be cached from each of the data to be accessed based on the target ratio, and cache each of the data to be cached to the first target CDN node.
[0124] In some specific embodiments, the second data transmission module 14 may specifically include:
[0125] The network hop count determination unit is used to determine the target physical distance between each CDN node and the first target CDN node using the preset data cache map, and to determine the target network hop count between each CDN node and the first target CDN node.
[0126] A node determination unit is used to determine the second target CDN node from each of the CDN nodes based on the physical distance to each target and the number of network hops to each target.
[0127] In some specific embodiments, the second data transmission module 14 may specifically include:
[0128] The first data transmission unit is used to determine whether the data volume of the target data is less than a preset data volume threshold. If the data volume of the target data is not less than the preset data volume threshold, the unit determines the bandwidth of each data transmission path between the target terminal and the second target CDN node, performs fragmentation processing on the target data according to the bandwidth of each data transmission path to obtain corresponding target data fragments, and uses each data transmission path to transmit each target data fragment from the second target CDN node to the target terminal.
[0129] In some specific embodiments, the second data transmission module 14 may specifically include:
[0130] The second data transmission unit is configured to, if the data volume of the target data is less than the preset data volume threshold, dynamically determine the target data transmission path from each of the data transmission paths based on the target node status and target routing decision model corresponding to each of the CDN nodes, and use the target data transmission path to transmit the target data from the second target CDN node to the target terminal; wherein, the target node status includes the CPU utilization, memory usage, and bandwidth utilization corresponding to each of the CDN nodes.
[0131] In some specific embodiments, the second data transmission module 14 further includes:
[0132] The fragment verification unit is used to verify whether the target data fragments received by the target terminal are complete based on the target hash values corresponding to each target data fragment. If the target data fragments received by the target terminal are incomplete, the missing data fragments in the target data fragments are transmitted to the target terminal using a preset backup data transmission path.
[0133] Furthermore, embodiments of this application also disclose an electronic device, Figure 5 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.
[0134] Figure 5 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the data transmission method based on a content delivery network disclosed in any of the foregoing embodiments. Alternatively, the electronic device 20 in this embodiment may specifically be an electronic computer.
[0135] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.
[0136] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.
[0137] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the content delivery network-based data transmission method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks.
[0138] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned data transmission method based on a content delivery network. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.
[0139] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.
[0140] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0141] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.
[0142] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0143] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. A data transmission method based on a content delivery network, characterized in that, include: Based on a preset time interval, real-time data access requests from the target terminal are obtained, target feature information corresponding to each real-time data access request is determined, and the target feature information and target content prediction model are used to predict the data to be accessed within a preset time period with an access probability not less than a preset probability threshold. The data to be accessed is obtained from the target data source, cached in the first target CDN node corresponding to the target terminal, and a preset data cache map is dynamically updated based on the physical distance and network topology between CDN nodes and the data access records in each CDN node; wherein, the preset data cache map includes the node ID of each CDN node, the data ID of the network data stored in each CDN node, and the physical distance between each CDN node; The current data access request of the target terminal is obtained, and it is determined whether the target data corresponding to the current data access request exists in the data to be accessed. If the target data corresponding to the current data access request exists in the data to be accessed, the target data is directly transmitted from the first target CDN node to the target terminal. If the target data corresponding to the current data access request does not exist in the data to be accessed, the second target CDN node corresponding to the target data is determined using the preset data cache map, and the target data is transmitted from the second target CDN node to the target terminal.
2. The data transmission method based on a content delivery network according to claim 1, characterized in that, Before using the target feature information and target content prediction model to predict the data to be accessed within a preset time period with an access probability not less than a preset probability threshold, the method further includes: Obtain historical data access requests corresponding to the target terminal, construct a target training set using each historical data access request, and set a target learning rate corresponding to the preset Adam optimizer; The target content prediction model is trained based on the target training set, the target learning rate, and the target loss function to obtain the target content prediction model; wherein the target content prediction model is a two-layer LSTM structure.
3. The data transmission method based on a content delivery network according to claim 1, characterized in that, The step of caching the data to be accessed to the first target CDN node corresponding to the target terminal includes: Determine whether the total data volume of each of the data to be accessed is greater than the target data capacity of the first target CDN node. If the total data volume of each of the data to be accessed is not greater than the target data capacity of the first target CDN node, then directly cache each of the data to be accessed to the first target CDN node. If the total data volume of each of the data to be accessed is greater than the target data capacity of the first target CDN node, then the access probability and data volume corresponding to each of the data to be accessed are determined, the target ratio of the access probability to the data volume of each of the data to be accessed is calculated, and data to be cached is selected from each of the data to be accessed based on the target ratio, and each of the data to be cached is cached to the first target CDN node.
4. The data transmission method based on a content delivery network according to claim 1, characterized in that, The step of determining the second target CDN node corresponding to the target data using the preset data cache map includes: The target physical distance between each CDN node and the first target CDN node is determined using the preset data cache map, and the target network hop count between each CDN node and the first target CDN node is also determined. The second target CDN node is determined from the CDN nodes based on the physical distance to each target and the number of network hops to each target.
5. The data transmission method based on a content delivery network according to any one of claims 1 to 4, characterized in that, The step of transmitting the target data from the second target CDN node to the target terminal includes: Determine whether the data volume of the target data is less than a preset data volume threshold. If the data volume of the target data is not less than the preset data volume threshold, determine the bandwidth of each data transmission path between the target terminal and the second target CDN node, perform fragmentation processing on the target data according to the bandwidth of each data transmission path to obtain corresponding target data fragments, and use each data transmission path to transmit each target data fragment from the second target CDN node to the target terminal.
6. The data transmission method based on a content delivery network according to claim 5, characterized in that, The step of transmitting the target data from the second target CDN node to the target terminal includes: If the data volume of the target data is less than the preset data volume threshold, then based on the target node status and target routing decision model corresponding to each CDN node, the target data transmission path is dynamically determined from each data transmission path, and the target data is transmitted from the second target CDN node to the target terminal using the target data transmission path; wherein, the target node status includes the CPU utilization, memory usage, and bandwidth utilization corresponding to each CDN node.
7. The data transmission method based on a content delivery network according to claim 5, characterized in that, After transmitting the target data from the second target CDN node to the target terminal, the process further includes: The target data fragments received by the target terminal are verified to be complete based on the target hash values corresponding to each target data fragment. If the target data fragments received by the target terminal are incomplete, the missing data fragments in the target data fragments are transmitted to the target terminal using a preset backup data transmission path.
8. A data transmission device based on a content delivery network, characterized in that, include: The data prediction module is used to acquire real-time data access requests from the target terminal based on a preset time interval, determine the target feature information corresponding to each real-time data access request, and use the target feature information and the target content prediction model to predict the data to be accessed within a preset time period with an access probability not less than a preset probability threshold. The data cache update module is used to obtain the data to be accessed from the target data source, cache the data to be accessed in the first target CDN node corresponding to the target terminal, and dynamically update the preset data cache map based on the physical distance and network topology between each CDN node and the data access records in each CDN node; wherein, the preset data cache map includes the node ID of each CDN node, the data ID of the network data stored in each CDN node, and the physical distance between each CDN node; The first data transmission module is used to obtain the current data access request of the target terminal, determine whether the target data corresponding to the current data access request exists in the data to be accessed, and if the target data corresponding to the current data access request exists in the data to be accessed, then the target data is directly transmitted from the first target CDN node to the target terminal. The second data transmission module is used to determine the second target CDN node corresponding to the target data using the preset data cache map if the target data corresponding to the current data access request does not exist in the data to be accessed, and to transmit the target data from the second target CDN node to the target terminal.
9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the data transmission method based on a content delivery network as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, Used to store a computer program, which, when executed by a processor, implements the data transmission method based on a content delivery network as described in any one of claims 1 to 7.