Method for caching content in content delivery network and device for executing the method

Abstract

A method for caching content in a content delivery network and an apparatus for performing the same are disclosed. A content caching apparatus in a content delivery network according to an embodiment of the present invention comprises one or more processors and a memory for storing one or more programs executed by the one or more processors, and includes: a content request detection unit that detects a content request from a user terminal and extracts content information regarding the requested content; a content request processing unit that extracts a node where the requested content is stored based on the extracted content information and network information, and transmits the content stored in the node closest to the user terminal among the extracted nodes to the user terminal; a content caching determination unit that determines that the requested content is a cache miss when the node where the requested content is stored is an original server, and determines a cache node to store the cache miss content using a previously stored reinforcement learning model based on the extracted content information and the network information; and a content caching processing unit that caches the cache miss content in the cache node determined by the content caching determination unit.

Description

Technology Field

[0001] Embodiments of the present invention relate to content caching technology in a content delivery network. Background Technology

[0003] In networks prior to the existence of content delivery networks, when a user terminal requested content, it was received from the origin server. Consequently, this caused user requests to be concentrated on the content origin server, significantly reducing the quality of service for content transmission.

[0004] Accordingly, Content Delivery Networks (CDNs) have recently been proposed, and to improve the service quality of content delivery, multiple servers storing content are located at the network edge close to user terminals.

[0005] The most important technology in such content delivery networks is content caching technology, which aims to provide fast content and reduce network traffic by storing frequently requested data in the network environment at intermediate nodes or edge servers.

[0006] However, existing content caching technologies are primarily based on static patterns, making it increasingly difficult to respond to the dynamic content request patterns of users resulting from the advancement of the digital age. Prior art literature

[0008] Korean Patent Publication No. 10-1508349 (March 27, 2015) The problem to be solved

[0009] Embodiments of the present invention are intended to provide a content caching method in a content delivery network. means of solving the problem

[0011] According to an exemplary embodiment of the present invention, a content caching device is provided, comprising: one or more processors and a memory for storing one or more programs executed by said one or more processors; a content request detection unit that detects a content request from a user terminal and extracts content information regarding said requested content; a content request processing unit that extracts a node where said requested content is stored based on said extracted content information and network information, and transmits content stored in a node among said extracted nodes that is close to said user terminal to said user terminal; a content caching determination unit that determines that said requested content is a cache miss when said node where said requested content is stored is an original server, and determines a cache node to store said cache miss content using a previously stored reinforcement learning model based on said extracted content information and said network information; and a content caching processing unit that caches said cache miss content in a cache node determined by said content caching determination unit.

[0012] The above content information includes the ID of the content and the popularity of the content, and the above network information may include at least one of the location of a node within the network, a list of content stored in the node, the cache size of the node, the connectivity of the node, the popularity of all content within the network, and the frequency of requests for the content.

[0013] The above reinforcement learning model can perform reinforcement learning for content caching through Q-learning using a Deep Q-Network.

[0014] The above content caching decision unit may include a machine learning module including the reinforcement learning model, a storage module that stores training data generated based on the input and output data of the reinforcement learning model, and a replication module that includes a target model replicated from the reinforcement learning model.

[0015] The machine learning module determines the optimal cache node and caching technique at the time the content is requested using the reinforcement learning model, and the caching technique may be the LRU (Least Recently Used) technique or the LFU (Least Frequently Used) technique.

[0016] The above content caching decision unit can be trained such that, at each preset period, the error is minimized by comparing the Q-value output by the reinforcement learning model at the current time when the content is requested with the Q'-value output by the target model at the current time using the training data.

[0017] The error between the Q-value output by the above reinforcement learning model and the Q'-value output by the above target model can be calculated through the Bellman error.

[0018] According to another exemplary embodiment of the present invention, a content caching method is provided for a computing device comprising one or more processors and a memory for storing one or more programs executed by said one or more processors, the method comprising: detecting a content request from a user terminal and extracting content information for said requested content; extracting a node in which said requested content is stored based on said extracted content information and network information; transmitting content stored in a node among said extracted nodes that is close to said user terminal to said user terminal; determining that said requested content is a cache miss if the node in which said requested content is stored is an original server; determining a cache node to store said cache miss content using a previously stored reinforcement learning model based on said extracted content information and said network information; and caching said cache miss content in said determined cache node. Effects of the invention

[0020] According to embodiments of the present invention, by learning long-term content request patterns to which reinforcement learning is applied, the cache hit rate can be maximized and network performance can be significantly improved. Brief explanation of the drawing

[0022] FIG. 1 is a block diagram illustrating the configuration of a content caching device in a content transmission network according to an embodiment of the present invention. FIG. 2 is a drawing for explaining a content caching decision unit of a content caching device in a content transmission network according to an embodiment of the present invention. FIG. 3 is a flowchart illustrating a content caching method in a content transmission network according to an embodiment of the present invention. FIG. 4 is a block diagram illustrating a computing environment including a computing device suitable for use in exemplary embodiments. Specific details for implementing the invention

[0023] Hereinafter, specific embodiments of the present invention will be described with reference to the drawings. The following detailed description is provided to facilitate a comprehensive understanding of the methods, apparatuses, and / or systems described herein. However, this is merely illustrative and the present invention is not limited thereto.

[0024] In describing the embodiments of the present invention, detailed descriptions of known technologies related to the present invention are omitted if it is determined that such detailed descriptions may unnecessarily obscure the essence of the present invention. Furthermore, the terms described below are defined in consideration of their functions within the present invention, and these may vary depending on the intentions or practices of the user or operator. Therefore, such definitions should be based on the content throughout this specification. Terms used in the detailed description are intended merely to describe the embodiments of the present invention and should not be limiting in any way. Unless explicitly stated otherwise, expressions in the singular form include the meaning of the plural form. In this description, expressions such as "include" or "comprise" are intended to refer to certain characteristics, numbers, steps, actions, elements, parts thereof, or combinations thereof, and should not be interpreted to exclude the existence or possibility of one or more other characteristics, numbers, steps, actions, elements, parts thereof, or combinations thereof other than those described.

[0025] In the following description, terms such as "transmission," "communication," "transmission," "reception," and other terms of similar meaning regarding signals or information include not only the direct transfer of signals or information from one component to another but also the transfer through other components. In particular, "transmission" or "transmitting" a signal or information to one component refers to the final destination of the signal or information and does not mean a direct destination. The same applies to the "reception" of signals or information. Furthermore, in this specification, two or more data or information are "related" means that if one data (or information) is obtained, at least a portion of another data (or information) can be obtained based thereon.

[0026] Additionally, terms such as "first," "second," etc., may be used to describe various components, but said components should not be limited by said terms. These terms may be used for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component.

[0027] Meanwhile, embodiments of the present invention may include a program for performing the methods described herein on a computer, and a computer-readable recording medium containing said program. The computer-readable recording medium may include program instructions, local data files, local data structures, etc., either alone or in combination. The medium may be one specifically designed and configured for the present invention, or one that is commonly available in the field of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of said programs may include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

[0028] Additionally, the term '~part' used in this embodiment refers to a software or hardware component such as an FPGA (field programmable gate array) or an ASIC, and the '~part' performs certain roles. However, the meaning of '~part' is not limited to software or hardware. The '~part' may be configured to reside in an addressable storage medium or may be configured to run one or more processors. Accordingly, as an example, the '~part' may include components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables.

[0029] The functions provided within the components and 'parts' may be combined into a smaller number of components and 'parts' or separated from additional components and 'parts'. In addition, the components and 'parts' may be implemented to utilize one or more CPUs within the device or secure multimedia card.

[0031] FIG. 1 is a block diagram illustrating the configuration of a content caching device in a content transmission network according to an embodiment of the present invention.

[0032] Referring to FIG. 1, a content caching device (100) in a content transmission network according to one embodiment of the present invention may include a content request detection unit (200), a content request processing unit (300), a content caching determination unit (400), and a content caching processing unit (500).

[0033] In the following, "node" includes networking equipment such as routers, and "source server" may include a server that stores the source of the content requested by the user terminal.

[0035] The content request detection unit (200) detects a content request from a user terminal (not shown) and can extract content information regarding the requested content.

[0037] The content request processing unit (300) can extract a node where the requested content is stored based on extracted content information and network information, and transmit the content from the node to the user terminal. Here, the content information may include the ID of the content, the popularity of the content, etc. Additionally, the network information may include the location of the node within the network, a list of content stored in the node, the cache size of the node, the node connectivity, the popularity of all content within the network, and the frequency of requests for the content.

[0038] Specifically, the content request processing unit (300) extracts the node where the requested content is stored based on the extracted content information and network information, and can check whether the content is stored sequentially starting from the node closest to the user terminal. That is, in order to transmit the content from the node closest to the user terminal that requested the content, the node where the content is stored can be extracted by gradually increasing the distance starting from the node closest to the user terminal.

[0039] Additionally, the content request processing unit (300) can transmit the content stored in the extracted node to the user terminal. At this time, the extracted node may be the node closest to the user terminal.

[0041] The content caching decision unit (400) can determine a cache node to store cached content. Specifically, if the node where the requested content is stored is the source server, the content caching decision unit (400) determines that the requested content is cached and can determine a cache node to store the cached content using a previously stored reinforcement learning model based on extracted content information and network information. Here, the content information may include the ID of the content, the popularity of the content, etc. Additionally, the network information may include the location of the node within the network, a list of content stored in the node, the cache size of the node, the node connectivity, the popularity of all content within the network, and the frequency of requests for the content.

[0042] In this case, the stored reinforcement learning model refers to a model trained to take the optimal action by updating the Q-value, which represents the expected reward for the action that can be taken from the current state.

[0043] According to one embodiment, the content caching decision unit (400) can perform reinforcement learning for content caching through Q-learning using a Deep Q-network (hereinafter referred to as Q-network). At this time, the state may be the popularity of the content, the location of a node within the network, a list of content stored in the node, the cache size of the node, the node connectivity, the popularity of all content within the network, the frequency of requests for content, etc., and the action may be whether to cache at each node within the network and the caching technique (e.g., LRU (Least Recently Used) technique or LFU (Least Frequently Used) technique).

[0044] Additionally, the content caching decision unit (400) can determine the optimal node to store cached content using a reinforcement learning model, and can perform learning at preset intervals. Accordingly, it is possible to learn content request patterns that accumulate over time, thereby learning long-term patterns and effectively processing content requests even in dynamic environments to optimize caching performance.

[0045] To this end, as shown in FIG. 2, the content caching decision unit (400) may include a machine learning module (410) including a reinforcement learning model, a storage module (420) that stores training data generated based on the input and output data of the reinforcement learning model, and a replication module (430) that includes a replicated target model whenever the reinforcement learning model is trained.

[0046] The machine learning module (410) can determine the optimal cache node and cache technique at the time when content is requested using a reinforcement learning model.

[0047] In an exemplary embodiment, the reinforcement learning model may be configured as a Transformer decoder and trained through Q-learning. This Transformer decoder can use a Masked Self-Attention algorithm to selectively encode important information from past points in time without referencing future information, thereby enabling efficient analysis of the relationship between content requests and cache misses. Additionally, the Transformer decoder enables parallel processing while maintaining temporal order through positional encoding, thereby enabling fast learning and real-time performance.

[0048] Additionally, the machine learning module (410) can train the reinforcement learning model so that the error is minimized by comparing the Q-value output at a specific time point (t) at each preset period with the Q'-value output from the target model. The training process of this reinforcement learning model will be explained in detail below.

[0049] The storage module (420) can generate and store training data based on the input and output data of the reinforcement learning model. Here, the training data is data required for Q-learning (e.g., the state (s) of the current time point (t). t ), action selected at the current time (a t ), reward at the current point in time (r t ), state at the next time point (t+1) (s t+1 Training data can be generated by forming a set of )). At this time, the storage module (420) stores the training data in the order in which it was generated, but if the stored size is exceeded, the data stored in order of oldest training data can be deleted.

[0050] The replication module (430) can generate a target value (Q-value) for training the reinforcement learning model using the replicated target model and training data whenever the reinforcement learning model is trained. That is, the replication module (430) is configured identically to the reinforcement learning model and can generate a Q'-value using the training data stored in the storage module (420) to enable the reinforcement learning model to be trained. Additionally, when the reinforcement learning model is trained, the replication module (430) can generate a target model by replicating the completed reinforcement learning model. That is, by generating a target model that replicates the reinforcement learning model whenever the reinforcement learning model is trained, it becomes possible to learn content request patterns that accumulate over time.

[0051] Below, we will explain the learning process of the reinforcement learning model.

[0052] A content caching decision unit (400) according to one embodiment of the present invention can train a reinforcement learning model at a preset period. Here, the preset period can be set based on the number of times a cache miss occurs (e.g., 100 times).

[0053] For example, the content caching decision unit (400) can train the reinforcement learning model to minimize the error by comparing the Q-value output at a specific time point (t) where a cache miss occurred in the reinforcement learning model with the Q'-value output at a specific time point (t) in the target model based on the training data.

[0054] At this time, the error between the Q-value of the reinforcement learning model and the Q'-value of the target model can be calculated through the Bellman Error as shown in Equation 1 below.

[0055] [Mathematical Formula 1]

[0056]

[0057] (Here, δ is the Bellman error, r is the reward for the action performed at a specific time point (t), is the maximum expected reward obtainable at the next time point (t+1) calculated by the target model, is the Q-value at a specific time point (t) calculated by the reinforcement learning model

[0058] In other words, since the weights (θ) of the reinforcement learning model are updated according to reinforcement learning, the weights (θ) of the target model, which have a fixed value - By training a reinforcement learning model based on ), it becomes possible to prevent the performance of the reinforcement learning model from degrading.

[0059] Using these Bellman errors, the Q-value of the reinforcement learning model can be updated as shown in Equation 2 below.

[0060] [Mathematical Formula 2]

[0061]

[0062] (Here, α is the Q-value at a specific time point (t) calculated in the reinforcement learning model, where α is the learning rate.

[0063] Meanwhile, in the process of training a reinforcement learning model, the present invention allows for training using partial state information rather than utilizing all state information. Furthermore, reward values ​​for cache hits between the time a cache miss occurs and the time of the previous cache miss occur are applied at the time of the cache miss, and additional rewards are applied if a cache hit for the corresponding content occurs within a certain period from the time of the cache miss. That is, by simultaneously reflecting past cache hit rewards and additional rewards at the time of the cache miss and performing training based on partial state information, past rewards are reflected all at once at the current time of the cache miss, future state information is processed through partial state information, and maximum rewards are predicted, thereby maximizing processing speed and resource efficiency. Here, partial state information may include the location of nodes within the network, the list of content stored in the nodes, the cache size of the nodes, node connectivity, the average popularity of all content within the network, and the request frequency of the content. In other words, resources can be saved by utilizing the average popularity of all content within the network and cached content, rather than utilizing the popularity of all content within the network.

[0064] That is, the Q-value of the reinforcement learning model can be updated as shown in Equation 3 below.

[0065] [Mathematical Formula 3]

[0066]

[0067] (Here, : Partial state information at the time a cache miss occurred, Point of view Action selected in, : Learning rate, r miss : Cash miss penalty, ( r rS + r rM + r rL ) : Recent request match rate reward, r totalP : Total match rate reward, Point of view at Cash Hit Rewards up to, : Additional reward (if cached content is successfully hit), : Discount rate that reflects future rewards in the present, : Maximum Q-value among possible actions in a state (output from the target model), : Q-value of a reinforcement learning model)

[0069] The content caching processing unit (500) can cache cached content in a cache node determined by the content caching decision unit (400). That is, the content caching processing unit (500) can store content determined to be cached according to a determined technique (LRU technique or LFU technique) in a cache node determined by the content caching decision unit (400).

[0070] Accordingly, according to the embodiments of the present invention, by learning long-term content request patterns to which reinforcement learning is applied, the cache hit rate can be maximized and network performance can be significantly improved.

[0072] FIG. 3 is a flowchart illustrating a content caching method in a content delivery network according to an embodiment of the present invention. The method illustrated in FIG. 3 may be performed, for example, by a content caching device in the aforementioned content delivery network. Although the method is described in the illustrated flowchart by dividing it into a plurality of steps, at least some of the steps may be performed in a different order, combined with other steps and performed together, omitted, divided into detailed steps, or performed with one or more steps not illustrated added.

[0073] In step 310, the content caching device (100) can detect a content request. Specifically, the content caching device (100) can detect a content request from a user terminal and extract content information for the requested content.

[0074] In step 320, the content caching device (100) can provide content depending on whether the detected content is cached. Specifically, the content caching device (100) can extract a node where the requested content is stored based on extracted content information and network information, and transmit the content from the node to the user terminal.

[0075] In step 330, the content caching device (100) can determine the location to cache the cached content. Specifically, if the node where the requested content is stored is the source server, the content caching device (100) determines that the content is cached, and can determine the cache node to store the requested content using a previously stored reinforcement learning model based on content information and network information.

[0076] In step 340, the content caching device (100) can cache the content at a determined location. Specifically, the content caching device (100) can store content that is determined to be cached miss according to a determined technique (LRU technique or LFU technique) at a determined cache node.

[0078] FIG. 4 is a block diagram illustrating a computing environment including a computing device suitable for use in exemplary embodiments. In the illustrated embodiments, each component may have different functions and capabilities in addition to those described below, and may include additional components in addition to those described below.

[0079] The illustrated computing environment (10) includes a computing device (12). In one embodiment, the computing device (12) may be a content caching device (100) in a content delivery network.

[0080] The computing device (12) includes at least one processor (14), a computer-readable storage medium (16), and a communication bus (18). The processor (14) can cause the computing device (12) to operate according to the exemplary embodiment described above. For example, the processor (14) can execute one or more programs stored in the computer-readable storage medium (16). The one or more programs may include one or more computer-executable instructions, and the computer-executable instructions may be configured to cause the computing device (12) to perform operations according to the exemplary embodiment when executed by the processor (14).

[0081] A computer-readable storage medium (16) is configured to store computer-executable instructions or program code, program data and / or other suitable forms of information. A program (20) stored in the computer-readable storage medium (16) includes a set of instructions executable by a processor (14). In one embodiment, the computer-readable storage medium (16) may be memory (volatile memory such as random access memory, non-volatile memory, or a suitable combination thereof), one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other forms of storage media that are accessed by a computing device (12) and capable of storing desired information, or a suitable combination thereof.

[0082] The communication bus (18) interconnects various other components of the computing device (12), including the processor (14) and the computer-readable storage medium (16).

[0083] The computing device (12) may also include one or more input / output interfaces (22) and one or more network communication interfaces (26) that provide interfaces for one or more input / output devices (24). The input / output interfaces (22) and network communication interfaces (26) are connected to a communication bus (18). The input / output devices (24) may be connected to other components of the computing device (12) through the input / output interfaces (22). An exemplary input / output device (24) may include an input device such as a pointing device (such as a mouse or trackpad), a keyboard, a touch input device (such as a touchpad or touchscreen), a voice or sound input device, various types of sensor devices and / or imaging devices, and / or an output device such as a display device, a printer, a speaker and / or a network card. An exemplary input / output device (24) may be included inside the computing device (12) as a component constituting the computing device (12), or it may be connected to the computing device (12) as a separate device distinct from the computing device (12).

[0085] Although representative embodiments of the present invention have been described in detail above, those skilled in the art will understand that various modifications can be made to the above-described embodiments without departing from the scope of the present invention. Therefore, the scope of the present invention should not be limited to the described embodiments, but should be defined by the claims set forth below as well as equivalents thereof. Explanation of the symbols

[0087] 10: Computing Environment 12: Computing device 14: Processor 16: Computer-readable storage media 18: Communication bus 20: Program 22: Input / Output Interface 24: Input / Output Devices 26: Network communication interface 100: Content caching device in a content delivery network 200: Content request detection unit 300: Content Request Processor 400: Content Caching Decision Unit 410: Machine Learning Module 420: Storage module 430: Replication Module 500: Content caching processing unit

Claims

Claim 1 A content caching device comprising one or more processors and a memory for storing one or more programs executed by said one or more processors, the device comprising: a content request detection unit that detects a content request from a user terminal and extracts content information regarding said requested content; a content request processing unit that extracts a node where said requested content is stored based on said extracted content information and network information, and transmits the content stored in the node closest to said user terminal among said extracted nodes to said user terminal; and a content caching determination unit that, if the node where said requested content is stored is an original server, determines that said requested content is a cache miss and determines a cache node to store said cache miss content using a previously stored reinforcement learning model based on said extracted content information and said network information.The system includes a content caching processing unit that caches the cached miss content in a cache node determined by the content caching decision unit, wherein the content caching decision unit, when it is determined that the requested content is a cached miss at the time the content is requested, determines the optimal cache node to store the cached miss content and the caching technique to apply to the cache node using the reinforcement learning model, wherein the caching technique is the LRU (Least Recently Used) technique or the LFU (Least Frequently Used) technique, wherein in the process of training the reinforcement learning model, partial state information is used instead of all state information, wherein the partial state information includes the average popularity of all content within the network and the average popularity of cached content, and applies reward values ​​for cache hits that occurred between the time the current cache miss occurred and the time the previous cache miss occurred at the time the current cache miss occurred, and applies an additional reward if a cache hit for the cached miss content occurs within a certain time from the time the current cache miss occurred, and based on the partial state information, the reward values ​​for the cache hit, and the additional reward, the reinforcement learning model corresponding to the time the current cache miss occurred A content caching device that updates a Q-value, wherein the content caching processing unit caches the cached miss content in the determined cache node according to the determined caching technique. Claim 2 A content caching device according to claim 1, wherein the content information includes an ID of the content and the popularity of the content, and the network information includes at least one of the location of a node in the network, a list of content stored in the node, the cache size of the node, the connectivity of the node, the popularity of all content in the network, and the frequency of requests for the content. Claim 3 A content caching device according to claim 1, wherein the reinforcement learning model performs reinforcement learning for content caching through Q-learning using a Deep Q-Network. Claim 4 delete Claim 5 delete Claim 6 A content caching device according to claim 3, wherein the content caching decision unit trains the reinforcement learning model at each preset period to minimize the error by comparing the Q-value output by the reinforcement learning model at the current time when the content is requested with the Q'-value output by the target model generated by replicating the reinforcement learning model using the training data at the current time. Claim 7 A content caching device according to claim 6, wherein the error between the Q-value output by the reinforcement learning model and the Q'-value output by the target model is calculated through the Bellman Error. Claim 8 A content caching method performed in a computing device having one or more processors and a memory for storing one or more programs executed by said one or more processors, comprising: detecting a content request from a user terminal and extracting content information for said requested content; extracting a node where said requested content is stored based on said extracted content information and network information; transmitting content stored in a node among said extracted nodes that is closest to said user terminal to said user terminal; determining that said requested content is a cache miss if the node where said requested content is stored is an original server; and determining a cache node to store said cache miss content using a previously stored reinforcement learning model based on said extracted content information and said network information.The method includes the step of caching the cache miss content in the cache node determined above, wherein the step of determining the cache node includes, when the requested content is determined to be a cache miss at the time the content is requested, determining the optimal cache node to store the cache miss content and the caching technique to apply to the cache node using the previously stored reinforcement learning model, wherein the caching technique is the LRU (Least Recently Used) technique or the LFU (Least Frequently Used) technique, and the content caching method further includes the step of training the previously stored reinforcement learning model, wherein the step of training the previously stored reinforcement learning model uses partial state information rather than all state information, wherein the partial state information includes the average popularity of all content within the network and the average popularity of cached content, and applies reward values ​​for cache hits that occurred between the time the cache miss occurred and the time the previous cache miss occurred at the time the cache miss occurred, and applies an additional reward if a cache hit for the cache miss content occurs within a certain time from the time the cache miss occurred, and based on the partial state information, the reward values ​​for the cache hit, and the additional reward A content caching method that updates the Q-value of the previously stored reinforcement learning model corresponding to the time when the cache miss occurs, and the caching step caches the cache miss content in the determined cache node according to the determined caching technique. Claim 9 A content caching method according to claim 8, wherein the content information includes an ID of the content and the popularity of the content, and the network information includes at least one of the location of a node in the network, a list of content stored in the node, the cache size of the node, the node connectivity, the popularity of all content in the network, and the frequency of requests for the content. Claim 10 A content caching method according to claim 8, wherein the reinforcement learning model performs reinforcement learning for content caching through Q-learning using a Deep Q-Network. Claim 11 delete Claim 12 A content caching method according to claim 10, wherein the previously stored reinforcement learning model utilizes a target model generated by replicating the previously stored reinforcement learning model, and is trained at predetermined intervals such that the error is minimized by comparing the Q-value output at the current time when the content is requested with the Q'-value output at the current time by the target model using training data. Claim 13 A content caching method according to claim 12, wherein the error between the Q-value output by the previously stored reinforcement learning model and the Q'-value output by the target model is calculated through the Bellman Error.

Images