Method and system for managing context cache

Abstract

The application is suitable for the technical field of data processing, and provides a management method and management system of context cache, the method comprising: updating a directory entry corresponding to a context cache unit in a global cache directory based on cache attribute information of the context cache unit; determining a subsequent access smart chip set of the context cache unit within a prediction window according to task execution relationship information and cache access association information corresponding to the directory entry; determining a strategy cost of each candidate cache layout strategy according to the subsequent access smart chip set, access prediction information of the context cache unit, and cache transmission information; determining a target cache layout strategy according to the strategy cost of each candidate cache layout strategy; and adjusting cache layout of the context cache unit according to the target cache layout strategy. The above scheme can reduce cross-device data interaction cost, improve context cache resource utilization rate and collaborative reasoning efficiency.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method and system for managing context caching. Background Technology

[0002] As the scale of artificial intelligence models continues to increase, applications such as generative AI, edge intelligence, multimodal interaction, image generation, and video generation are placing ever-growing demands on inference computing power and storage resources. To improve inference throughput, existing solutions typically employ multiple intelligent chips to collaboratively execute inference tasks, distributing model computation through tensor parallelism, pipelined parallelism, and task parallelism. During inference, data such as intermediate model states, historical context, key features, latent variables, visual features, or inter-frame information often need to be reused in subsequent inference processes, thus consuming a certain amount of cache resources.

[0003] Existing cache management solutions primarily focus on cache retention, eviction, swapping in, swapping out, or data offloading within a single device or general-purpose storage system, typically based on factors such as cache capacity, cache size, access frequency, or storage tier. In multi-chip collaborative inference scenarios, cached data may be distributed across different chips, local storage tiers, or external storage spaces, with varying overhead for data access and migration between these locations. If a general cache management approach is still adopted, issues such as insufficient cache resource utilization, high cross-device data interaction overhead, duplicate cache data usage, unstable cache migration benefits, or high costs associated with rereading after offloading can easily arise, thus impacting the overall efficiency and stability of multi-chip collaborative inference. Summary of the Invention

[0004] This application provides a context cache management method and system that can match the cache layout of the context cache unit with the cache access requirements and data transmission overhead in the multi-smart chip collaborative inference process, thereby reducing cross-device data interaction costs and improving context cache resource utilization and collaborative inference efficiency.

[0005] Firstly, a context cache management method is provided, applied to a context cache management system including multiple smart chips and at least one context cache management node; the management method includes: Based on the cache attribute information of the context cache unit, update the directory entry in the global cache directory corresponding to the context cache unit; Based on the task execution relationship information and the cache access association information corresponding to the directory item, the set of subsequent access smart chips of the context cache unit within the prediction window is determined. The task execution relationship information is used to characterize the task execution relationship when the multiple smart chips perform inference tasks, and the cache access association information is used to characterize the subsequent access relationship of the context cache unit. Based on the subsequent access smart chip set, the access prediction information of the context cache unit, and the cache transfer information, the strategy cost of each candidate cache layout strategy is determined. Based on the policy cost of each candidate cache layout strategy, the target cache layout strategy is determined; The cache layout of the context cache unit is adjusted according to the target cache layout strategy.

[0006] In this technical solution, the global cache directory is first updated based on the cache attribute information of the context cache unit, enabling the context cache management node to grasp the current storage state and access-related state of the context cache unit. Then, by combining task execution relationship information and cache access association information corresponding to the directory entries, the set of subsequent access smart chips for the context cache unit within the prediction window is determined. This allows cache management to no longer rely solely on cache size or hotness, but to adapt to the task execution relationship and subsequent access relationship in the multi-smart chip collaborative inference process. Furthermore, the strategy cost of each candidate cache layout strategy is determined based on the set of subsequent access smart chips, access prediction information, and cache transfer information, and the target cache layout strategy is determined accordingly. Finally, the cache layout of the context cache unit is adjusted according to the target cache layout strategy. This reduces the overhead caused by cross-chip access and invalid cache relocation, improves the utilization rate of context cache resources, and enhances the efficiency of multi-smart chip collaborative inference.

[0007] In one possible implementation, determining the set of subsequent access smart chips for the context cache unit within the prediction window based on the cache access association information and task execution relationship information corresponding to the directory entry includes: Based on the task execution relationship information, determine the set of candidate smart chips that are qualified to access within the prediction window; Based on the cache access association information corresponding to the directory item and the task execution relationship information, determine the access score corresponding to each candidate smart chip in the candidate smart chip set; The set of subsequent access smart chips is determined based on the access score corresponding to each candidate smart chip.

[0008] In this implementation, by first determining the set of candidate smart chips that are qualified to access, and then determining the set of smart chips to be accessed subsequently based on the access score, the subsequent access objects of the context cache are effectively filtered, avoiding the inclusion of irrelevant chips in the cache layout decision.

[0009] In one possible implementation, determining the access score corresponding to each candidate smart chip in the candidate smart chip set based on the cache access association information corresponding to the directory entry and the task execution relationship information includes: Based on the cache access association information corresponding to the directory entry, determine the number of accesses, the most recent access time, and the number of active requests of the context cache unit relative to the candidate smart chip; Based on the task execution relationship information, determine the future scheduling position of the candidate intelligent chip within the prediction window; The access score corresponding to the candidate smart chip is determined based on the number of accesses, the most recent access time, the number of active requests, and the future scheduling position.

[0010] In this implementation, by combining the number of accesses, the most recent access time, the number of active requests, and the future scheduling location to determine the access score, a comprehensive assessment of the likelihood of subsequent access to candidate smart chips is achieved, thereby improving the accuracy of subsequent access to the set of smart chips.

[0011] In one possible implementation, determining the set of subsequent access smart chips for the context cache unit within the prediction window based on the cache access association information and task execution relationship information corresponding to the directory entry includes: Based on the cache access association information corresponding to the directory entry, determine the target cache unit associated with the context cache unit; Based on the task execution relationship information, determine the execution smart chip corresponding to the target cache unit; Based on the execution smart chip corresponding to the target cache unit, determine the set of subsequent access smart chips for the context cache unit.

[0012] In this implementation, by first determining the target cache unit associated with the context cache unit, and then determining the subsequent access smart chip set based on the execution smart chip of the target cache unit, the adaptation of cross-region, cross-frame, or cross-temporal cache access relationships in image or video inference scenarios is achieved.

[0013] In one possible implementation, determining the target cache unit associated with the context cache unit based on the cache access association information corresponding to the directory entry includes: From the set of context cache units corresponding to the same inference task, determine the candidate associated cache unit corresponding to the context cache unit; Based on the spatial-temporal association information in the cache access association information, the default association information between the context cache unit and the candidate association cache unit is determined; Dynamic association information is determined based on at least one of the feature similarity, attention association, motion association, and common access degree between the context cache unit and the candidate association cache unit; Based on the default association information and the dynamic association information, the association score between the context cache unit and the candidate association cache unit is determined; The target cache unit is determined from the candidate associated cache units based on the association score.

[0014] In this implementation, the association score is determined by combining default association information and dynamic association information, and the target cache unit is filtered based on the association score, so as to achieve accurate identification of associated cache objects in image or video tasks.

[0015] In one possible implementation, determining the strategy cost of each candidate cache layout strategy based on the subsequent access smart chip set, the access prediction information of the context cache unit, and the cache transfer information includes: Based on the set of subsequent access smart chips, a candidate cache layout strategy corresponding to the context cache unit is determined. The candidate cache layout strategy includes at least two of the following: remote access strategy, replication sharing strategy, ownership migration strategy, and offload strategy. When the candidate cache layout strategy includes a remote access strategy, the strategy cost of the remote access strategy is determined based on the expected number of accesses in the access prediction information and the remote read cost in the cache transmission information. When the candidate cache layout strategy includes a replication-sharing strategy, the strategy cost of the replication-sharing strategy is determined based on the replication transmission cost, local access cost, and space occupation cost in the cache transmission information. When the candidate cache layout strategy includes an ownership migration strategy, the strategy cost of the ownership migration strategy is determined based on the migration transmission cost, post-migration read cost, and directory update cost in the cache transmission information. When the candidate cache layout strategy includes an offloading strategy, the strategy cost of the offloading strategy is determined based on the offloading transmission cost in the cache transmission information, the migration probability in the access prediction information, and the external storage occupation cost.

[0016] In this implementation, a quantitative comparison between different cache layout methods is achieved by determining the policy costs of candidate cache layout strategies such as remote access, replication sharing, ownership migration, and offloading.

[0017] In one possible implementation, the cached transmission information is determined based on a pre-defined lookup table or a piecewise linear model; The lookup table or the piecewise linear model is generated based on at least one of the following: read latency, write latency, link latency, and effective bandwidth, corresponding to the cache block size, source and target storage levels, and link type.

[0018] In this implementation, cache transmission information is determined by a pre-calibrated lookup table or piecewise linear model, thereby achieving low-overhead estimation of policy costs and reducing the impact of runtime cache layout decisions on the inference process.

[0019] In one possible implementation, determining the target cache layout strategy based on the strategy cost of each candidate cache layout strategy includes: For a candidate cache layout strategy with a target storage location, determine whether the candidate cache layout strategy meets the capacity constraint based on the available capacity of the target storage location. For candidate cache layout strategies that require data transmission to be completed before the first access, it is determined whether the candidate cache layout strategy meets the completion time constraint based on the expected completion time corresponding to the candidate cache layout strategy and the expected first access time corresponding to the context cache unit. From the candidate cache layout strategies that satisfy the corresponding constraints, determine the candidate cache layout strategy with the minimum strategy cost; If the candidate cache layout strategy with the lowest strategy cost satisfies a preset benefit condition relative to the current cache layout corresponding to the directory entry, the candidate cache layout strategy with the lowest strategy cost is determined as the target cache layout strategy.

[0020] In this implementation, the target cache layout strategy is determined by combining capacity constraints, completion time constraints, strategy costs, and preset benefit conditions, thereby enabling feasibility assessment and stable switching of cache layout adjustments.

[0021] In one possible implementation, adjusting the cache layout of the context cache unit according to the target cache layout strategy includes: When the target cache layout strategy is a replication-sharing strategy or an ownership migration strategy, cache space is reserved for the context cache unit at the target location; Set the status of the directory entry corresponding to the context cache unit to "copying" or "migrating"; The context cache unit data is transferred to the target location; After the data at the target location passes verification, the directory entry corresponding to the context cache unit in the global cache directory is updated.

[0022] In this implementation, the context cache layout adjustment process is reliably executed by reserving space, marking status, transmitting data, and updating the directory after verification during the copy sharing or ownership migration process.

[0023] Secondly, a context cache management system is provided, the management system including multiple smart chips and at least one context cache management node; The context cache management node is used in the context cache management method described in any of the first aspects above to manage the context cache during the collaborative inference process of the multiple smart chips.

[0024] Thirdly, a context cache management device is provided, configured in a context cache management system, the management device comprising: The directory entry update module is used to update the directory entry in the global cache directory corresponding to the context cache unit based on the cache attribute information of the context cache unit; The chip set determination module is used to determine the set of intelligent chips that the context cache unit will subsequently access within the prediction window based on the task execution relationship information and the cache access association information corresponding to the directory item. The task execution relationship information is used to characterize the task execution relationship when the multiple intelligent chips perform inference tasks, and the cache access association information is used to characterize the subsequent access relationship of the context cache unit. The strategy cost determination module is used to determine the strategy cost of each candidate cache layout strategy based on the subsequent access smart chip set, the access prediction information of the context cache unit, and the cache transmission information. The layout strategy determination module is used to determine the target cache layout strategy based on the strategy cost of each candidate cache layout strategy. The layout adjustment module is used to adjust the cache layout of the context cache unit according to the target cache layout strategy.

[0025] Fourthly, an electronic device is provided, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method as described in any of the embodiments of the first aspect above.

[0026] Fifthly, a computer-readable storage medium is provided that stores a computer program, which, when executed by a processor, implements the method as described in any embodiment of the first aspect.

[0027] In a sixth aspect, a computer program product is provided, comprising a computer program that, when executed by a processor, enables the implementation of the method as described in any embodiment of the first aspect.

[0028] The beneficial effects of the second to sixth aspects mentioned above can be referred to the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the structure of a context caching management system provided in an embodiment of this application; Figure 2 This is a flowchart illustrating a context cache management method provided in an embodiment of this application; Figure 3 This is a schematic diagram illustrating the process for determining the set of intelligent chips for subsequent access in a text / multimodal scenario provided in this application embodiment; Figure 4 This is a schematic diagram illustrating the process for determining the set of intelligent chips for subsequent access in an image / video scene, as provided in an embodiment of this application. Figure 5 This is a schematic diagram of the process for determining the cost of candidate cache layout strategies provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of a context cache management device provided in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0030] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0031] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0032] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0033] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0034] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0035] It should be understood that the sequence number of each step in this embodiment does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of this application embodiment.

[0036] As the scale of artificial intelligence models continues to increase, applications such as generative AI, edge intelligence, multimodal interaction, image generation, and video generation are placing ever-growing demands on inference computing power and storage resources. To improve inference throughput, existing solutions typically employ multiple intelligent chips to collaboratively execute inference tasks, distributing model computation through tensor parallelism, pipelined parallelism, and task parallelism. During inference, data such as intermediate model states, historical context, key features, latent variables, visual features, or inter-frame information often need to be reused in subsequent inference processes, thus consuming a certain amount of cache resources.

[0037] Existing cache management solutions primarily focus on cache retention, eviction, swapping in, swapping out, or data offloading within a single device or general-purpose storage system, typically based on factors such as cache capacity, cache size, access frequency, or storage tier. In multi-chip collaborative inference scenarios, cached data may be distributed across different chips, local storage tiers, or external storage spaces, with varying overhead for data access and migration between these locations. If a general cache management approach is still adopted, issues such as insufficient cache resource utilization, high cross-device data interaction overhead, duplicate cache data usage, unstable cache migration benefits, or high costs associated with rereading after offloading can easily arise, thus impacting the overall efficiency and stability of multi-chip collaborative inference.

[0038] To address the aforementioned issues, this application provides a context cache management method and system. The method first updates the global cache directory based on the cache attribute information of the context cache unit, enabling the context cache management node to grasp the current storage and access-related states of the context cache unit. Then, by combining task execution relationship information and cache access association information corresponding to directory entries, the method determines the set of subsequent access smart chips for the context cache unit within the prediction window. This allows cache management to adapt to the task execution and subsequent access relationships in the multi-smart chip collaborative inference process, rather than solely relying on cache size or hotness. Furthermore, the method determines the strategy cost of each candidate cache layout strategy based on the set of subsequent access smart chips, access prediction information, and cache transfer information, and accordingly determines the target cache layout strategy. Finally, the method adjusts the cache layout of the context cache unit according to the target cache layout strategy. This reduces the overhead of cross-chip access and invalid cache relocation, improves context cache resource utilization, and enhances the efficiency of multi-smart chip collaborative inference.

[0039] The following detailed description, with reference to the accompanying drawings, describes the context caching management method, management system, management device, electronic device, computer-readable storage medium, and computer program product of this application.

[0040] See Figure 1 This illustration shows a schematic diagram of the structure of a context caching management system provided in an embodiment of this application; as shown... Figure 1 As shown, the context cache management system may include multiple smart chips and at least one context cache management node. The multiple smart chips are used to collaboratively execute inference tasks, and the context cache management node is used to manage the context cache during the collaborative inference process of the multiple smart chips.

[0041] The context cache management node can be implemented by a main smart chip among multiple smart chips, an independent cache management server, or a combination of both. This embodiment does not limit this implementation. The context cache management node can maintain a global cache directory, which records the directory entries corresponding to each context cache unit. Directory entries may include at least one of the following: cache identifier, request or task identifier, cache type, cache segment range, current storage level and address, owner chip, copy location, version number, validity status, number of references, last access time, number of accesses within a statistical window, estimated first access time, and the set of smart chips for subsequent accesses.

[0042] Each smart chip can maintain a local cache state table, which records the state information of the corresponding smart chip's local cache blocks. For example, the local cache state table can record at least one of the following: actual occupied space, access count, last access time, read / write status, and version number of the local cache block. The local cache blocks can be stored in the smart chip's local storage space, such as SRAM or DDR. The cache data corresponding to the context cache unit can also be stored in a cache data plane such as shared DDR, SSD, or external cache server.

[0043] In some embodiments, each smart chip can periodically report status information from its local cache status table to the context cache management node; or, when at least one of the following events occurs: cache creation, cache block sealing, cache access, cache copying, cache migration, cache invalidation, task rescheduling, and available capacity reaching a threshold, it can report corresponding cache attribute information to the context cache management node. The context cache management node can update the directory entry corresponding to the context cache unit in the global cache directory based on the received cache attribute information.

[0044] Furthermore, the context cache management node can determine the set of subsequent access smart chips for the context cache unit within the prediction window based on task execution relationship information and cache access association information corresponding to directory entries; and determine the policy cost of each candidate cache layout strategy based on the set of subsequent access smart chips, the access prediction information of the context cache unit, and cache transfer information. Then, the context cache management node can determine the target cache layout strategy based on the policy costs of each candidate cache layout strategy, and adjust the cache layout of the context cache unit according to the target cache layout strategy.

[0045] It should be noted that after adjusting the cache layout of the context cache unit, the context cache management node can also update the directory entry corresponding to that context cache unit in the global cache directory. This ensures that information such as the current storage level and address, owner chip, replica location, version number, validity status, and subsequent access to the smart chip set in that directory entry matches the adjusted cache layout. This allows the cache layout of the context cache unit to match the cache access requirements and data transfer overhead during the collaborative inference process of multiple smart chips.

[0046] Based on the above system embodiments, the context cache management method provided in this application embodiment will be described below. It should be noted that this management method can be applied to... Figure 1The context cache management system shown can be executed by a context cache management node. In some embodiments, the context cache management node can manage the context cache during the collaborative inference process of multiple smart chips through information interaction with each smart chip, the local cache state table, the global cache directory, and the cache data plane. The embodiments of this application will be further described below in conjunction with the method flow.

[0047] See Figure 2 The diagram illustrates a flowchart of a context cache management method provided in an embodiment of this application; as shown below. Figure 2 As shown, the method may include the following steps: Step 201: Based on the cache attribute information of the context cache unit, update the directory entry in the global cache directory corresponding to the context cache unit.

[0048] In this context, a context cache unit refers to a cache management unit within the context cache that can be independently addressed and migrated. Context cache units for different inference tasks can be partitioned in different ways. For example, for text or multimodal Transformer models, a context cache unit can be a KV cache page or cache block formed by partitioning according to requests, model layers, tensor parallel partitioning, and token intervals; for image generation tasks, a context cache unit can be a cache unit partitioned according to image regions, pixel blocks, image blocks, patches, visual tokens, latent variable blocks, feature map blocks, attention windows, or resolution levels; for video generation tasks, a context cache unit can be a cache unit partitioned according to video frames, frame groups, time segments, intra-frame image blocks, video patches, spatiotemporal tokens, latent variable blocks, feature map blocks, or spatiotemporal blocks formed across consecutive frames.

[0049] Cache attribute information refers to information used to characterize the current state and management attributes of a context cache unit. Cache attribute information may include at least one of the following: cache identifier, request or task identifier, cache type, cache segment range, current storage level and address, owner chip, copy location, version number, validity status, reference count, last access time, number of accesses within a statistical window, estimated first access time, and the set of smart chips for subsequent accesses. Cache attribute information may also include at least one of the following: actual occupied space, access count, last access time, read / write status, and version number of the local cache block maintained by each smart chip.

[0050] The global cache directory is a cache management directory maintained by the context cache management node. It is used to uniformly record the directory entries of each context cache unit during the collaborative inference process of multiple intelligent chips, so that the context cache management node can know the storage location, owner, copy, version, validity status, and access-related status of the context cache unit.

[0051] A directory entry is a record in the global cache directory that corresponds to a context cache unit. A directory entry can be used to record at least one of the following for a context cache unit: cache identifier, request or task identifier, cache type, cache segment range, current storage level and address, owner chip, copy location, version number, validity status, number of references, last access time, number of accesses within the statistics window, estimated first access time, and the set of smart chips for subsequent accesses.

[0052] In the specific implementation, each smart chip can maintain a local cache status table, recording information such as the actual occupied space, access count, last access time, read / write status, and version number of the local cache block. Each smart chip can periodically send cache attribute information corresponding to the context cache unit to the context cache management node; or, when at least one of the following events occurs: cache creation, cache block sealing, access, copying, migration, invalidation, task rescheduling, or available capacity reaching a threshold, the smart chip can send cache attribute information corresponding to the context cache management node. Upon receiving the cache attribute information, the context cache management node can determine the context cache unit corresponding to the cache attribute information and search for the directory entry corresponding to that context cache unit in the global cache directory. If a corresponding directory entry already exists, it updates the directory entry based on the received cache attribute information; if no corresponding directory entry exists, it creates the directory entry corresponding to the context cache unit and writes the received cache attribute information into that directory entry.

[0053] For example, for text or multimodal Transformer models, a context cache unit can be defined as a KV cache corresponding to a specific request at a specific model layer, a specific tensor parallel shard, and a specific token range. Once this context cache unit is created or archived, the smart chip responsible for maintaining it can report cache attribute information such as the cache identifier, request or task identifier, cache type, cache segment range, current storage level and address, owner chip, version number, validity status, and local access count to the context cache management node. Based on this cache attribute information, the context cache management node can create or update the corresponding directory entry for this context cache unit in the global cache directory, enabling the global cache directory to record the current location, version status, and access-related information of this context cache unit.

[0054] Step 202: Based on the task execution relationship information and the cache access association information corresponding to the directory item, determine the set of smart chips for subsequent access of the context cache unit within the prediction window.

[0055] The task execution relationship information is used to characterize the task execution relationships when multiple intelligent chips perform inference tasks. This task execution relationship information can be used to represent the division of labor, execution paths, or scheduling relationships of inference tasks among multiple intelligent chips. For example, the task execution relationship information may include at least one of the following: model execution graph, tensor parallel mapping, pipelined parallel mapping, hybrid parallel mapping, or task scheduling table.

[0056] The cache access association information corresponding to the directory entry is used to characterize the subsequent access relationships of the context cache unit. In other words, the cache access association information can reflect which access objects, task stages, or other cached content the context cache unit may have access associations with during the execution of the inference task. This cache access association information can be recorded in the directory entries of the global cache directory, or it can be determined based on access statistics, reference relationships, and spatial-temporal associations recorded in the directory entries.

[0057] The prediction window refers to the future execution range used to determine the subsequent access status of the context cache unit. This prediction window can correspond to a future period of time, or it can correspond to several future scheduling slots, model execution stages, or inference steps; this embodiment of the application does not limit this.

[0058] The subsequent access smart chip set refers to the set of smart chips that may need to access the context cache unit within the prediction window. This subsequent access smart chip set can serve as the data basis for subsequently determining candidate cache layout strategies and strategy costs.

[0059] In the specific implementation, the context cache management node can obtain task execution relationship information for multiple smart chips and read the directory entry corresponding to the context cache unit from the global cache directory. Then, the context cache management node can combine the task execution relationship information and the cache access association information corresponding to the directory entry to determine which smart chips might access the context cache unit within the prediction window, thereby determining the set of subsequent accessing smart chips corresponding to the context cache unit. It should be noted that this set of subsequent accessing smart chips can be updated based on changes in inference task execution, changes in cache access status, or task rescheduling.

[0060] For example, in text or multimodal inference tasks, a context cache unit can be a key-value cache block corresponding to a request at a certain model layer. The context cache management node can determine the execution relationships of multiple smart chips in subsequent inference processes based on the model execution graph, parallel mapping relationships, or task scheduling table corresponding to the inference task. Simultaneously, based on the access-related information in the directory entry corresponding to the context cache unit, it can determine the access association between the context cache unit and subsequent inference processes. Based on the above information, the context cache management node can determine the smart chips that may need to access the key-value cache block within the prediction window and form a set of subsequent access smart chips corresponding to the context cache unit.

[0061] In one possible implementation, a schematic diagram illustrating the process for determining the set of intelligent chips for subsequent access in a text / multimodal scenario can be found here. Figure 3 Specifically, this may include the following steps: Step 301: Based on the task execution relationship information, determine the set of candidate smart chips that are qualified to access within the prediction window.

[0062] For text / multimodal scenarios, task execution relationship information can be used to characterize the task execution relationships of multiple intelligent chips when performing text or multimodal inference tasks. For text or multimodal models, this task execution relationship information can include at least one of the following: model execution graph, tensor parallel mapping, pipelined parallel mapping, and task scheduling table. Model execution graph can be used to characterize the execution relationships between different computational stages or model layers in the model; tensor parallel mapping can be used to characterize the mapping relationship between model tensor slices and intelligent chips; pipelined parallel mapping can be used to characterize the mapping relationship between different stages of the model and intelligent chips; and task scheduling table can be used to characterize the scheduling arrangement of different intelligent chips executing relevant task stages within a future scheduling window.

[0063] In text or multimodal scenarios, the prediction window can be specifically a future scheduling window, and the prediction window H can represent the number of scheduling locations contained in the future scheduling window.

[0064] In this embodiment, the context cache management node can read the model execution graph, tensor parallel mapping, pipeline parallel mapping, and task scheduling table to determine the set I(c) of candidate smart chips that are qualified to access within the prediction window H from multiple smart chips. Here, H can represent the number of scheduling positions included in the prediction window, and c represents the context cache unit of the object to be subsequently accessed.

[0065] For intelligent chips that do not belong to the model execution path, or those that, although belonging to the system, have not been scheduled to execute the model stage related to the context cache unit c, they can be excluded from the candidate intelligent chip set I(c). This allows the scope of subsequent access to the intelligent chip set to be matched with the actual execution path of the text or multimodal model.

[0066] Step 302: Determine the access score corresponding to each candidate smart chip in the candidate smart chip set based on the cache access association information and task execution relationship information corresponding to the directory item.

[0067] In this embodiment, the cache access association information corresponding to the directory entry can be used to characterize the access association relationship between the context cache unit c and the candidate smart chip. For candidate smart chip i in the candidate smart chip set I(c), each smart chip can report to the context cache management node the number of accesses N(c,i) within the statistical window W, the time Δt(c,i) since the most recent access, the first execution position s(c,i) of the smart chip in the future scheduling window, and the number of active requests L(c,i) referencing the shared prefix cache.

[0068] Among them, the number of accesses N(c,i) can represent the number of times the candidate smart chip i accesses the context cache unit c within the statistical window W; the time since the most recent access Δt(c,i) can represent the time interval between the candidate smart chip i's most recent access to the context cache unit c and the current time; the first execution position s(c,i) can represent the position of the candidate smart chip i in the first execution of the relevant model stage within the prediction window; and the number of active requests L(c,i) can represent the number of active requests that reference the shared prefix cache and are related to the candidate smart chip i.

[0069] In one possible implementation, based on the cache access association information and task execution relationship information corresponding to the directory entries, the access score corresponding to each candidate smart chip in the candidate smart chip set is determined, including: Based on the cache access association information corresponding to the directory entries, determine the number of accesses, the most recent access time, and the number of active requests for the context cache unit relative to the candidate smart chip; Based on the task execution relationship information, determine the future scheduling position of the candidate intelligent chip within the prediction window; The access score for each candidate smart chip is determined based on the number of visits, the most recent access time, the number of active requests, and the future scheduling location.

[0070] Specifically, the context cache management node can normalize the aforementioned raw data to obtain multiple normalized parameters for calculating the access score. For example, the number of accesses can be normalized as F(c,i)=min[N(c,i) / Nref,1], where Nref is the normalization benchmark for the number of accesses; the time since the most recent access can be normalized as E(c,i)=exp[-Δt(c,i) / τ], where τ is the time decay constant; the future scheduling location can be normalized as Q(c,i)=max[0,1-s(c,i) / H], where H is the number of scheduling locations included in the prediction window; and the number of active requests can be normalized as G(c,i)=min[L(c,i) / Lref,1], where Lref is the normalization benchmark for the number of shared prefix references. When the candidate smart chip does not execute the relevant model stage within the prediction window, Q(c,i) can be set to 0.

[0071] After obtaining the above normalized parameters, the context cache management node can calculate the access score P(c,i) corresponding to the candidate smart chip i. For example, the access score can be calculated according to the following formula: P(c,i)=w1F(c,i)+w2E(c,i)+w3Q(c,i)+w4G(c,i), where w1, w2, w3, and w4 are all non-negative numbers, and w1+w2+w3+w4=1. As an example, w1=0.30, w2=0.20, w3=0.35, and w4=0.15 can be taken; the above values ​​are only used to illustrate one possible implementation and do not constitute a limitation on the scope of protection. Through this access score, the historical access situation, access frequency, future scheduling position, and shared prefix reference situation of the candidate smart chip i to the context cache unit c can be comprehensively characterized.

[0072] Step 303: Determine the set of smart chips to be accessed next based on the access scores corresponding to each candidate smart chip.

[0073] In this embodiment, the context cache management node can determine the set of subsequent access smart chips for context cache unit c within the prediction window based on the access score P(c,i) corresponding to each candidate smart chip in the candidate smart chip set I(c). Specifically, candidate smart chips with access scores P(c,i) not less than the access threshold θP can be added to the set of subsequent access smart chips. If the number of candidate smart chips that meet the access threshold θP is less than a preset minimum number, the top K candidate smart chips with the highest access scores can be added from the candidate smart chip set I(c) to form the set of subsequent access smart chips.

[0074] For example, for a text or multimodal Transformer model, a context cache unit c can be a KV cache block corresponding to a shared prefix. The context cache management node can first determine the set I(c) of candidate smart chips eligible for access within the prediction window H based on the model execution graph, parallel mapping relationships, and task scheduling table. Then, for each candidate smart chip i in the set I(c), the context cache management node can calculate an access score P(c,i) based on the access count N(c,i), the time since the most recent access Δt(c,i), the future scheduling position s(c,i), and the number of active requests L(c,i) corresponding to candidate smart chip i. Finally, candidate smart chips with access scores not less than the access threshold θP are determined as smart chips in the subsequent access smart chip set, thus obtaining the subsequent access smart chip set of context cache unit c within the prediction window.

[0075] It should be noted that the access score can be used to determine the subsequent access subject of context cache unit c within the prediction window. For the access prediction information required for subsequent policy cost calculations, the expected number of accesses and the expected first access time can also be determined based on access statistics and scheduling information. For example, the expected number of accesses can be obtained using an exponential moving average: n_hat(c,i)=ρNnew(c,i)+(1-ρ)Nold(c,i), where Nnew is the number of accesses in the most recent statistical window, Nold is the previous prediction result, and ρ is between 0 and 1. The expected first access time can be calculated based on the future scheduling position as t_first(c,i)=t_now+s(c,i)×t_slot, where t_slot is the average execution time of several recent scheduling positions; when the scheduler has given a definite start time, the definite start time can be used directly. The access score is used to determine the access subject, and the expected number of accesses and the expected first access time can be used for subsequent policy cost calculations.

[0076] In one possible implementation, for image / video scenarios, a schematic diagram of the subsequent access to the smart chip set determination process can be found here. Figure 4 Specifically, this may include the following steps: Step 401: Determine the target cache unit associated with the context cache unit based on the cache access association information corresponding to the directory entry.

[0077] In this embodiment, for image or video generation tasks, context cache units may have spatial, temporal, attentional, motion, or shared access relationships. The cache access association information corresponding to the directory entry can be used to characterize these relationships. For the current context cache unit, a target cache unit with a high degree of association can be determined from other context cache units corresponding to the same inference task based on this cache access association information.

[0078] In one possible implementation, determining the target cache unit associated with the context cache unit based on the cache access association information corresponding to the directory entry includes: From the set of context cache units corresponding to the same inference task, determine the candidate associated cache units corresponding to the context cache units; Based on the spatial-temporal association information in the cache access association information, determine the default association information between the context cache unit and the candidate associated cache unit; Dynamic association information is determined based on at least one of the following: feature similarity, attention association, motion association, and common access degree between the context cache unit and the candidate association cache unit. Based on the default association information and the dynamic association information, the association score between the context cache unit and the candidate association cache unit is determined; The target cache unit is determined from the candidate associated cache units based on the association score.

[0079] Specifically, candidate associated cache units can be other context cache units obtained by dividing the same image or video generation task according to image regions, patches, visual token positions, video frames, time segments, or spatiotemporal blocks. The task scheduler can establish default association information between the current context cache unit i and the candidate associated cache unit j based on region coordinates, patch or visual token positions, video frame numbers, time segments, and spatiotemporal block positions. This default association information can be represented by a default association value A(i,j). In one embodiment, A(i,j) = η1IoU(i,j) + η2exp[-Ds(i,j) / σs] + η3exp[-Dt(i,j) / σt] + η4W(i,j). Where IoU is the overlap ratio of two spatial regions, Ds is the normalized distance between the centers or boundaries of two spatial units, and Dt is the frame number difference, time segment difference, or spatiotemporal tube time distance; W is 1 when the two buffer units belong to the same attention window, have a preset keyframe dependency, or belong to a preset adjacency list, otherwise it is 0; η1 to η4 are non-negative weights and their sum is 1. For pure image tasks, η3 can be set to 0; for tasks without overlapping regions, η1 can be set to 0.

[0080] Furthermore, during model operation, each intelligent chip can also report operational data used to determine dynamic association information. For example, each intelligent chip can report the feature vectors Vi and Vj of the current context cache unit i and the candidate association cache unit j, the average attention value Att(i,j) between cache units, the motion vectors Ui and Uj, and the number of common visits Nco(i,j) within the statistical window. Among them, feature similarity can be defined as S(i,j)=[1+cos(Vi,Vj)] / 2; attention association can be defined as Matt(i,j)=Att(i,j) / maxk Att(i,k); motion association can be defined as Mmotion(i,j)=exp[-||Ui-Uj||² / σm]; and common visit can be defined as C(i,j)=min[Nco(i,j) / Cref,1]. For models with attention output, attention correlation Matt(i,j) can be used; for video models with motion estimation results, motion correlation Mmotion(i,j) can be used; or a weighted combination of attention correlation and motion correlation can be used.

[0081] After determining the default association information and dynamic association information, the context cache management node can calculate the association score between the current context cache unit i and the candidate association cache unit j. For example, the association score can be expressed as R(i,j) = aA(i,j) + bS(i,j) + cM(i,j) + dC(i,j). Here, A(i,j) represents the default association information, S(i,j) represents the feature similarity, M(i,j) can represent attention association, motion association, or a weighted combination of both, and C(i,j) represents the co-access degree; a, b, c, and d are all non-negative numbers and a + b + c + d = 1. In one example, a = 0.35, b = 0.20, c = 0.30, and d = 0.15 can be taken. These values ​​are only used to illustrate one possible implementation and do not constitute a limitation on the scope of protection.

[0082] After obtaining the association scores, the context cache management node can determine the candidate association cache units with association scores R(i,j) not less than the association threshold θR as the target cache units, or select the top K candidate association cache units with the highest association scores from each context cache unit as the target cache units. Default association information ensures that certain association objects are obtained during the first execution, while dynamic association information can be used to add, delete, or adjust association objects after the model runs, thus enabling the determination of target cache units to adapt to actual association changes during the image or video generation process.

[0083] Step 402: Determine the execution smart chip corresponding to the target cache unit based on the task execution relationship information.

[0084] In this application scenario, task execution relationship information can include a task scheduling table corresponding to the image or video generation task. This task scheduling table can record which smart chips execute different image regions, patches, visual tokens, video frames, time segments, or spatiotemporal blocks. After determining the target cache unit, the context cache management node can determine the executing smart chip corresponding to each target cache unit based on the task execution relationship information. That is, if the image region, video frame segment, or spatiotemporal block corresponding to a certain target cache unit is executed by a certain smart chip, then that smart chip can be used as the executing smart chip corresponding to that target cache unit.

[0085] Step 403: Determine the set of subsequent access smart chips for the context cache unit based on the execution smart chip corresponding to the target cache unit.

[0086] In this embodiment, since there is a spatial-temporal or dynamic association between the target cache unit and the current context cache unit, the smart chip responsible for executing the tasks related to the target cache unit may need to access the current context cache unit. Based on this, the context cache management node can add the executing smart chip corresponding to the target cache unit to the set of subsequent access smart chips for the current context cache unit. If multiple target cache units correspond to the same executing smart chip, the set of subsequent access smart chips can be formed by deduplicating the executing smart chips.

[0087] For example, in image generation tasks, if the association score between the context cache unit corresponding to the boundary of an image region and the cache units corresponding to adjacent regions reaches an association threshold, then the intelligent chips responsible for both the current region and the adjacent regions can be added to the subsequent access intelligent chip set. Similarly, in video generation tasks, if the motion correlation between the context cache unit corresponding to a video spatiotemporal block and the cache unit corresponding to a subsequent frame is high, and the association score reaches an association threshold, then the intelligent chip responsible for that subsequent frame can be added to the subsequent access intelligent chip set. Thus, the spatial-temporal relationships in image or video tasks can be converted into executable cache-sharing objects, and the corresponding subsequent access intelligent chip set can be further determined.

[0088] In one possible implementation, after determining the set of smart chips that will subsequently access the context cache unit within the prediction window, the context cache management node can also determine the access frequency of the context cache unit. This access frequency can be used to characterize the overall access activity of the context cache unit during the collaborative inference process of multiple smart chips, and can serve as reference information for subsequently determining candidate cache layout strategies.

[0089] Specifically, for a context cache unit c, the context cache management node may determine the access popularity Hot(c) of the context cache unit c according to the global access count, the most recent access time, the access score corresponding to the candidate intelligent chip, and the number of active request references. In an embodiment, the access popularity may be expressed as: Hot(c)=u1Fglobal(c)+u2Eglobal(c)+u3Pmax(c)+u4Gglobal(c). Wherein Fglobal(c) is a value obtained by normalizing the sum of the access counts of all intelligent chips accessing the context cache unit c within a statistical window by Nref_total; Eglobal(c)=exp[-Δtglobal(c) / τglobal], where Δtglobal(c) is the time elapsed since any intelligent chip most recently accessed the context cache unit c; Pmax(c) is the maximum value among the access scores P(c,i) of each candidate intelligent chip; Gglobal(c) is a value obtained by normalizing the total number of active requests referencing the context cache unit c by Lref_total; u1 to u4 are non-negative numbers and u1+u2+u3+u4=1.

[0090] After obtaining the access popularity Hot(c), the context cache management node may determine the cache hot-cold type of the context cache unit c according to the access popularity Hot(c). For example, when Hot(c)≥θhot, the context cache unit c may be marked as a hot cache; when θcold<Hot(c)<θhot, the context cache unit c may be marked as a warm cache; when Hot(c)≤θcold, the context cache unit c may be marked as a cold cache. In an example, θhot may be 0.70, and θcold may be 0.30; the above numerical values are only used to illustrate one implementable manner, and do not constitute a limitation on the protection scope.

[0091] Further, the context cache management node may further correct the cache hot-cold type in combination with the life cycle state of the context cache unit. For example, an unsealed writable tail cache cannot be directly evicted as a cold cache; a context cache unit whose estimated first access time is less than the prefetch time threshold may be promoted to at least a warm cache; a sealed cache whose requests have ended and whose reference count is 0 may enter a recyclable state. As a result, the hot-cold judgment result of the context cache unit can take into account both the access activity degree and the cache life cycle state, so as to provide a reference for the subsequent determination of the candidate cache placement strategy.

[0092] Step 203: Determine the strategy cost of each candidate cache placement strategy according to the subsequent access intelligent chip set, the access prediction information of the context cache unit, and the cache transmission information.

[0093] The access prediction information is used to characterize the expected access of the context cache unit during subsequent inference. The access prediction information may include at least one of the following: the expected number of accesses of the context cache unit relative to each smart chip in the subsequent access set, the expected first access time, and the probability of being migrated back within the prediction window after being unloaded. For example, the expected number of accesses may represent the number of times a smart chip is expected to access the context cache unit within the prediction window, and the expected first access time may represent the time when the smart chip first needs to access the context cache unit.

[0094] The cache transfer information is used to characterize the transfer overhead involved when the context cache unit is accessed, copied, migrated, or unloaded between different storage locations or smart chips. The cache transfer information may include at least one of the following: cache unit data volume, source storage read latency, target storage write latency, link latency, effective bandwidth, remote read cost, copy transfer cost, migration transfer cost, local access cost, directory update cost, wait penalty, and external storage occupancy cost.

[0095] In one possible implementation, for a context cache unit c transferred from storage location x to storage location y, the time for a complete transfer can be defined as Tcopy(x,y,c) = Dread(x) + L(x,y) + Size(c) / B(x,y) + Dwrite(y). Here, Size(c) is the data size of the context cache unit c, B(x,y) is the effective bandwidth between storage location x and storage location y obtained through runtime monitoring or offline calibration, L(x,y) is the fixed link latency between storage location x and storage location y, Dread(x) is the read latency of the source storage location x, and Dwrite(y) is the write latency of the target storage location y. For remote reads that do not require a complete copy to be stored at the target, the remote read time can be defined as Tread(x,i,c) = Dread(x) + L(x,i) + Size(c) / B(x,i) + Dconsume(i). Where i is the smart chip accessing the context cache unit c, and Dconsume(i) is the processing latency of smart chip i receiving cached data and using it for inference.

[0096] In one possible implementation, the aforementioned cached transmission information can be determined based on a pre-calibrated lookup table or piecewise linear model. Specifically, during the initialization or offline calibration phase, the context cache management node can measure parameters such as read latency, write latency, link latency, and effective bandwidth according to the cache block size range, source and target storage levels, and link type, and generate a lookup table or piecewise linear model. During runtime, the context cache management node can preferentially read the lookup table or call the piecewise linear model to determine the cached transmission information required for the aforementioned migration time, remote read time, and subsequent policy cost calculations.

[0097] In some embodiments, when the deviation between the measured bandwidth and the calibrated value is detected to exceed the update threshold during operation, the context cache management node can update the bandwidth parameters of the corresponding link without regenerating the complete model. This reduces the runtime overhead of policy cost calculation and minimizes the impact of cache layout decisions on the inference process.

[0098] In some embodiments, the context cache management node can also set a time budget for policy cost calculation. When the calculation time reaches the time budget, subsequent candidate comparisons can be stopped, and a candidate cache layout strategy can be determined according to preset fast rules. For example, the cache creation phase can adopt a default layout corresponding to the model parallel approach; hot caches can be preferentially retained in near-end storage; warm caches can be retained in DDR and allowed remote access; cold caches or overflow caches can be offloaded to external storage. Furthermore, policy cost calculation can be triggered when cache blocks are sealed, access sets change, hotness crosses a threshold, remaining capacity falls below a threshold, or tasks are rescheduled, rather than being performed on every cache read. In one possible implementation, see [link to implementation details]. Figure 5 The diagram illustrates the process of determining the cost of candidate cache layout strategies. Based on the subsequent access to the smart chip set, access prediction information of the context cache unit, and cache transfer information, the strategy cost of each candidate cache layout strategy is determined, which may include the following steps: Step 501: Based on the set of smart chips to be accessed subsequently, determine the candidate cache layout strategy corresponding to the context cache unit. The candidate cache layout strategy includes at least two of the following: remote access strategy, replication sharing strategy, ownership migration strategy, and offloading strategy.

[0099] Specifically, for a context cache unit c to be decided, the context cache management node can construct at least two candidate cache layout strategies corresponding to the context cache unit c based on the set of subsequent access smart chips I(c). The remote access strategy can represent keeping the context cache unit c at the current storage location x, accessible by the subsequent access smart chip via remote read; the replication sharing strategy can represent copying the context cache unit c to the near-end location corresponding to one or more subsequent access smart chips to form a read-only copy; the ownership migration strategy can represent migrating the owner location of the context cache unit c from the current storage location to the target smart chip; and the offloading strategy can represent offloading the context cache unit c to an external storage location, such as external storage z. The above candidate cache layout strategies can be determined based on information such as the set of subsequent access smart chips, the data volume of the context cache unit, its storage location, and transmission conditions.

[0100] Step 5011: If the candidate cache layout strategy includes a remote access strategy, determine the strategy cost of the remote access strategy based on the expected number of accesses in the access prediction information and the remote read cost in the cache transmission information.

[0101] Specifically, under the remote access strategy, the context cache unit c can be kept at the current storage location x. Subsequent smart chips in the smart chip set I(c) access the context cache unit c via remote reading. The context cache management node can determine the strategy cost of the remote access strategy based on the expected number of accesses n_hat(c,i) for each smart chip i and the remote read time Tread(x,i,c) from the current storage location x to the smart chip i. In one embodiment, the strategy cost of the remote access strategy can be expressed as: Cremote(c) = Σi∈I(c)n_hat(c,i) × Tread(x,i,c). Where I(c) is the set of smart chips subsequently accessing the context cache unit c, and n_hat(c,i) is the expected number of accesses of the smart chip i to the context cache unit c.

[0102] Step 5012: If the candidate cache layout strategy includes the replication sharing strategy, determine the strategy cost of the replication sharing strategy based on the replication transmission cost, local access cost, and space occupation cost in the cache transmission information.

[0103] Specifically, under the replication-sharing strategy, the context cache management node can determine the chip set J for establishing a near-end replica from the subsequent access smart chip set I(c), and copy the context cache unit c from the current storage location x to the near-end location corresponding to the smart chip in chip set J. The strategy cost of the replication-sharing strategy can include replication transmission cost, subsequent local access cost, and space occupation cost. In one embodiment, the strategy cost of the replication-sharing strategy can be expressed as: Ccopy(c)=Σi∈J Tcopy(x,i,c)+Σi∈I(c)n_hat(c,i)×Dlocal(i)+λs×Size(c)×|J|. Wherein, J is the chip set for establishing a near-end replica, Tcopy(x,i,c) is the replication transmission cost of copying the context cache unit c from the current storage location x to the near-end location corresponding to the smart chip i, Dlocal(i) is the single access cost of the local storage of the smart chip i, λs is the space occupation penalty coefficient, and |J| is the number of smart chips in chip set J.

[0104] Step 5013: If the candidate cache layout strategy includes an ownership migration strategy, determine the strategy cost of the ownership migration strategy based on the migration transmission cost, post-migration read cost, and directory update cost in the cache transmission information.

[0105] Specifically, under the ownership migration strategy, the context cache management node can migrate the context cache unit c from its current storage location x to the target smart chip t, and set the location corresponding to the target smart chip t as the new owner location. The strategy cost of the ownership migration strategy may include the migration and transmission cost of migrating the context cache unit c to the target smart chip t, the read cost of other subsequent accessing smart chips reading the context cache unit c from the target smart chip t after migration, and the directory update cost of updating the global cache directory, synchronizing versions, and waiting for in-transit accesses to finish. In one embodiment, the strategy cost of the ownership migration strategy can be expressed as: Cmigrate(c,t)=Tcopy(x,t,c)+Σi∈I(c)n_hat(c,i) ×Tread(t,i,c)+Cdir. Where Tcopy(x,t,c) is the migration cost of moving the context cache unit c from the current storage location x to the target smart chip t, Tread(t,i,c) is the read cost of smart chip i reading the context cache unit c from the corresponding location of the target smart chip t, and Cdir is the cost of updating the global cache directory, synchronizing the version, and waiting for the in-transit access to finish.

[0106] Step 5014: If the candidate cache layout strategy includes an offload strategy, determine the strategy cost of the offload strategy based on the offload transfer cost in the cache transfer information, the migration probability in the access prediction information, and the external storage occupation cost.

[0107] Specifically, under the offloading strategy, the context cache management node can offload the context cache unit c from the current storage location x to the external storage location z. The strategy cost of the offloading strategy may include the offloading transfer cost of offloading the context cache unit c to the external storage location z, the migration cost incurred by the possible migration back of the context cache unit c within the prediction window, and the external storage occupation cost. In one embodiment, the strategy cost of the offloading strategy can be expressed as: Coffload(c,z)=Tcopy(x,z,c)+precall(c)×[Tcopy(z,t,c)+Cwait(c)]+λz×Size(c). Where Tcopy(x,z,c) is the offloading transfer cost of transferring context cache unit c from the current storage location x to the external storage location z, precall(c) is the relocation probability of context cache unit c within the prediction window, t is the expected relocation target, Tcopy(z,t,c) is the transfer cost of transferring context cache unit c from the external storage location z to the expected relocation target t, Cwait(c) is the waiting penalty when the relocation is not completed before the first access, and λz is the external storage occupancy coefficient.

[0108] Step 204: Determine the target cache layout strategy based on the strategy cost of each candidate cache layout strategy.

[0109] In this embodiment, the target cache layout strategy refers to the strategy determined from among the candidate cache layout strategies, used for subsequent cache layout adjustments to the context cache unit. Candidate cache layout strategies may include at least two of the following: remote access strategy, replication sharing strategy, ownership migration strategy, and offloading strategy. The context cache management node can determine the target cache layout strategy based on the strategy costs of each candidate cache layout strategy.

[0110] In one possible implementation, the target cache layout strategy is determined based on the strategy cost of each candidate cache layout strategy, including: For a candidate cache layout strategy with a target storage location, determine whether the candidate cache layout strategy meets the capacity constraint based on the available capacity of the target storage location. For candidate cache layout strategies that require data transmission to be completed before the first access, determine whether the candidate cache layout strategy meets the completion time constraint based on the expected completion time of the candidate cache layout strategy and the expected first access time of the context cache unit. From the candidate cache layout strategies that satisfy the corresponding constraints, determine the candidate cache layout strategy with the minimum strategy cost; If the candidate cache layout strategy with the lowest strategy cost satisfies the preset benefit conditions relative to the current cache layout corresponding to the directory entry, the candidate cache layout strategy with the lowest strategy cost is determined as the target cache layout strategy.

[0111] Specifically, for candidate cache layout strategies with target storage locations, such as replication-sharing, ownership migration, or offloading strategies, the context cache management node can determine whether the available capacity of the target storage location is sufficient to accommodate the context cache units. For example, when the available capacity of the candidate target location is less than Size(c) + Reserve(y), the candidate cache layout strategy can be set to unavailable. Here, Size(c) is the data size of the context cache unit c, and Reserve(y) is the reserved capacity of the target storage location y. In this way, cache replication, migration, or offloading can be avoided when the capacity of the target storage location is insufficient.

[0112] For candidate cache layout strategies that require data transfer to be completed before the first access, such as replication-sharing strategies or ownership migration strategies, the context cache management node can also calculate the estimated completion time corresponding to the candidate cache layout strategy. For example, the estimated completion time can be expressed as Tfinish = t_now + Tcopy + Tqueue. Here, t_now is the current time, Tcopy is the transmission time under the corresponding candidate cache layout strategy, and Tqueue is the current transmission queue waiting time. In one implementation, the candidate cache layout strategy is considered to satisfy the completion time constraint only if Tfinish + Tsafety ≤ min[t_first(c,i)]. Here, Tsafety is the safety margin, and t_first(c,i) is the estimated time for the subsequent access of smart chip i to first access context cache unit c. This method avoids situations where the cache layout adjustment is not yet complete before the subsequent inference stage requires access to the context cache unit.

[0113] After obtaining candidate cache layout strategies that satisfy both capacity and completion time constraints, the context cache management node can select the candidate cache layout strategy Snew with the lowest policy cost from the feasible candidate strategies. Furthermore, to avoid frequent replication or migration due to slight fluctuations in policy cost, the candidate cache layout strategy Snew with the lowest policy cost can only be determined as the target cache layout strategy if it satisfies a preset benefit condition relative to the current cache layout Scurrent corresponding to the directory entry. For example, the preset benefit condition can be expressed as C(Snew) ≤ C(Scurrent) × (1-ε), where ε is the hysteresis coefficient. In one example, ε can be 0.10; this value is only used to illustrate one possible implementation and does not constitute a limitation on the scope of protection. If no candidate cache layout strategy simultaneously satisfies the capacity constraint, completion time constraint, and preset benefit condition, the current cache layout can be maintained.

[0114] In some embodiments, strategy selection can also employ fast rules to prune candidate strategies. For example, when the subsequent access set of smart chips includes only one new smart chip and the original owner chip no longer accesses the context cache unit, the ownership migration strategy can be considered a preferred candidate; when the subsequent access set of smart chips includes multiple smart chips and the expected number of accesses reaches the replication threshold, the replication sharing strategy can be considered a preferred candidate; when the access probability is low or the expected first access time is greater than the remote access threshold, the remote access strategy can be considered a preferred candidate; when the context cache unit is marked as cold cache, the remaining capacity is lower than the capacity threshold, or the cache contains overflow data, the offloading strategy can be considered a preferred candidate. Afterward, the context cache management node can perform only an approximate cost comparison among the retained candidate cache layout strategies.

[0115] In some embodiments, in multi-request or multi-user concurrent inference scenarios, if multiple requests have the same system prompt word, historical prefix, image condition, video prefix, or task configuration, a shared cache identifier can be generated for the shared portion, and the request reference relationship can be maintained in the global cache directory. The context cache management node can determine whether to keep the shared cache in shared DDR, copy it to multiple near-ends of smart chips, provide remote reading by an external cache management server, or offload it to SSD when the request is inactive, based on the number of requests referencing the context cache unit, the set of smart chips where the request is located, the expected number of accesses, and the expected first access time.

[0116] Step 205: Adjust the cache layout of the context cache unit according to the target cache layout strategy.

[0117] In this embodiment, cache layout adjustment refers to adjusting the storage location, replica location, owner location, or offloading location of a context cache unit according to a target cache layout strategy. Different target cache layout strategies may correspond to different adjustment methods. For example, under a remote access strategy, the current location of the context cache unit can be maintained, allowing subsequent access to the smart chip for remote reading; under a replication-sharing strategy, a replica of the context cache unit can be established near the target smart chip; under an ownership migration strategy, the owner location of the context cache unit can be migrated to the target smart chip; under an offloading strategy, the context cache unit can be transferred to an external cache management server or external storage location such as an SSD.

[0118] In one possible implementation, the cache layout of the context cache unit is adjusted according to the target cache layout strategy, including: When the target cache layout strategy is a replication-sharing strategy or an ownership migration strategy, cache space is reserved for the context cache unit at the target location; Set the status of the directory entry corresponding to the context cache unit to "copying" or "migrating"; Transfer the context buffer unit to the target location; After the data at the target location passes verification, update the directory entry in the global cache directory corresponding to the context cache unit.

[0119] Specifically, when the target cache layout strategy is a replication-sharing strategy or an ownership migration strategy, the context cache management node can first reserve cache space for the context cache unit at the target location and set the status of the directory entry corresponding to the context cache unit in the global cache directory to "replicating" or "migrating". Afterwards, cached data can be asynchronously transferred via P2P DMA, shared memory replication, or storage read / write. During data transfer, the original valid location can continue to serve existing reads, and the new location will not provide services externally as a valid copy or a new owner location until verification is completed.

[0120] After data transmission is complete, the target location can verify the cache identifier, data length, checksum, and version number. If the verification passes, the context cache management node can update the directory entry corresponding to the context cache unit in the global cache directory, for example, setting the target location to a valid copy location or a new owner location, and synchronously updating information such as the version number, validity status, copy location, or owner chip. In some embodiments, the context cache management node can also wait for the in-transit access to the old location to finish before releasing the old cache. If the transmission times out or the verification fails, the new location can be revoked, and the original valid location can continue to be used. This improves data reliability during replication sharing or ownership migration, preventing subsequent access to the smart chip from reading incomplete or inconsistent cache data.

[0121] For example, taking a large text model with multiple requests for inference as an example, a 256MB KV cache block is currently located in the DDR of chip 1. The model execution graph and task scheduling table determine that the candidate access chips are chip 2, chip 3, and chip 4. After calculating the access score, the access scores of chip 2, chip 3, and chip 4 are 0.86, 0.79, and 0.24, respectively; when the access threshold θP is 0.60, the subsequent set of accessing smart chips is chip 2 and chip 3. If the effective bandwidth of chip 1 to chip 2 and chip 3 is 64GB / s, ignoring the small amount of fixed latency in the example, a single transmission of 256MB takes about 4ms. Within the prediction window, chip 2 and chip 3 are expected to be accessed 8 times and 7 times, respectively. Therefore, the main transmission time for maintaining a single cache and remotely accessing it is about (8+7)×4=60ms; the main transmission time for copying a cache to each of the two chips is about 2×4=8ms. If the capacities of the two target chips meet the constraints, replication can be completed before the first access, and the policy cost of the replication-sharing strategy after adding space occupancy penalties is still less than the policy cost of the remote access strategy multiplied by (1-ε), then the context cache management node can determine the replication-sharing strategy as the target cache layout strategy and replicate the KV cache block to the near-end storage of chip 2 and chip 3 respectively. The above values ​​are only used to illustrate the calculation process of the formula and do not limit the actual cache size, bandwidth, and threshold.

[0122] For example, taking video generation as an example, the task scheduling table can assign adjacent frames to different smart chips. The context cache management node can establish the association between adjacent frames based on the default temporal adjacency relationship, and after the model runs, determine whether a certain keyframe cache will affect the subsequent two frames based on motion correlation and attention score. At this time, the smart chip responsible for the two frames can be added to the set of smart chips that will access the next frame, and the keyframe cache can be copied according to the policy cost. For earlier spatiotemporal block caches that are not expected to be accessed in the short term, they can be offloaded to an external cache management server or SSD, and migrated back in advance according to the expected first access time.

[0123] In summary, the above-described method embodiments form a closed-loop management system from cache attribute information collection, directory entry update, determination of subsequent access smart chip set, calculation of candidate cache layout strategy cost, determination of target cache layout strategy, cache layout adjustment to directory entry update. This enables the cache layout of the context cache unit to match the task execution relationship, cache access association relationship and data transmission overhead in the multi-smart chip collaborative inference process, thereby reducing the overhead caused by cross-chip access and invalid cache relocation, and improving the utilization rate of context cache resources and the efficiency of collaborative inference.

[0124] See Figure 6The diagram shows a schematic of the structure of a context cache management device provided in an embodiment of this application; for ease of explanation, only the parts related to the embodiments of this application are shown.

[0125] Specifically, the context cache management device 600 can be configured as follows: Figure 1 The context cache management system shown may include: The directory entry update module 601 is used to update the directory entry corresponding to the context cache unit in the global cache directory based on the cache attribute information of the context cache unit; The chip set determination module 602 is used to determine the set of intelligent chips that the context cache unit will subsequently access within the prediction window based on the task execution relationship information and the cache access association information corresponding to the directory item. The task execution relationship information is used to characterize the task execution relationship when multiple intelligent chips perform inference tasks, and the cache access association information is used to characterize the subsequent access relationship of the context cache unit. The strategy cost determination module 603 is used to determine the strategy cost of each candidate cache layout strategy based on the access prediction information of the subsequent access smart chip set, the context cache unit, and the cache transfer information. The layout strategy determination module 604 is used to determine the target cache layout strategy based on the strategy cost of each candidate cache layout strategy. The layout adjustment module 605 is used to adjust the cache layout of the context cache unit according to the target cache layout strategy.

[0126] In this embodiment of the application, the chip set determination module 602 may include a candidate chip determination unit, an access score determination unit, and a first set determination unit.

[0127] The candidate chip determination unit is used to determine the set of candidate smart chips that are eligible for access within the prediction window based on task execution relationship information.

[0128] The access score determination unit is used to determine the access score of each candidate smart chip in the candidate smart chip set based on the cache access association information and task execution relationship information corresponding to the directory entry.

[0129] The first set determination unit is used to determine the set of smart chips to be accessed subsequently based on the access scores corresponding to each candidate smart chip.

[0130] In some embodiments, the access score determination unit is further configured to determine the number of accesses, the most recent access time, and the number of active requests of the context cache unit relative to the candidate smart chip based on the cache access association information corresponding to the directory entry; determine the future scheduling position of the candidate smart chip within the prediction window based on the task execution relationship information; and determine the access score corresponding to the candidate smart chip based on the number of accesses, the most recent access time, the number of active requests, and the future scheduling position.

[0131] In another embodiment of this application, the chip set determination module 602 may further include a target cache unit determination unit, an execution chip determination unit, and a second set determination unit.

[0132] The target cache unit determination unit is used to determine the target cache unit associated with the context cache unit based on the cache access association information corresponding to the directory entry.

[0133] The execution chip determination unit is used to determine the execution smart chip corresponding to the target cache unit based on the task execution relationship information.

[0134] The second set determination unit is used to determine the set of subsequent access smart chips for the context cache unit based on the execution smart chip corresponding to the target cache unit.

[0135] In some embodiments, the target cache unit determining unit is further configured to: determine candidate associated cache units corresponding to the context cache units from the set of context cache units corresponding to the same inference task; determine default association information between the context cache units and the candidate associated cache units based on the spatial-temporal association information in the cache access association information; determine dynamic association information based on at least one of feature similarity, attention association, motion association, and co-access degree between the context cache units and the candidate associated cache units; determine the association score between the context cache units and the candidate associated cache units based on the default association information and the dynamic association information; and determine the target cache unit from the candidate associated cache units based on the association score.

[0136] In this embodiment, the strategy cost determination module 603 may include a candidate strategy determination unit and a cost determination unit.

[0137] The candidate strategy determination unit is used to determine the candidate cache layout strategy corresponding to the context cache unit based on the subsequent access to the smart chip set. The candidate cache layout strategy includes at least two of the following: remote access strategy, replication sharing strategy, ownership migration strategy, and offloading strategy.

[0138] The cost determination unit is used to determine the policy cost of a remote access policy based on the expected number of accesses in the access prediction information and the remote read cost in the cache transfer information when the candidate cache layout policy includes a remote access policy; to determine the policy cost of a replication sharing policy based on the replication transfer cost, local access cost, and space occupation cost in the cache transfer information when the candidate cache layout policy includes a replication sharing policy; to determine the policy cost of an ownership migration policy based on the migration transfer cost, post-migration read cost, and directory update cost in the cache transfer information when the candidate cache layout policy includes an offload policy; and to determine the policy cost of an offload policy based on the offload transfer cost in the cache transfer information, the migration probability in the access prediction information, and the external storage occupation cost when the candidate cache layout policy includes an offload policy.

[0139] In some embodiments, the policy cost determination module 603 may further include a transmission information determination unit. The transmission information determination unit is configured to determine cached transmission information based on a pre-calibrated lookup table or piecewise linear model. The lookup table or piecewise linear model is generated based on at least one of the following: cache block size, source / destination storage level, and link type, corresponding to read latency, write latency, link latency, and effective bandwidth.

[0140] In this embodiment of the application, the layout strategy determination module 604 may include a capacity constraint determination unit, a completion time constraint determination unit, a minimum cost strategy determination unit, and a target strategy determination unit.

[0141] The capacity constraint determination unit is used to determine whether a candidate cache layout strategy meets the capacity constraint based on the available capacity of the target storage location, for a candidate cache layout strategy with a target storage location.

[0142] The completion time constraint determination unit is used to determine whether a candidate cache layout strategy meets the completion time constraint based on the expected completion time of the candidate cache layout strategy and the expected first access time of the context cache unit.

[0143] The minimum cost strategy determination unit is used to determine the candidate cache layout strategy with the minimum strategy cost from the candidate cache layout strategies that satisfy the corresponding constraints.

[0144] The target strategy determination unit is used to determine the candidate cache layout strategy with the lowest strategy cost as the target cache layout strategy when the candidate cache layout strategy with the lowest strategy cost satisfies the preset benefit conditions relative to the current cache layout corresponding to the directory entry.

[0145] In this embodiment, the layout adjustment module 605 may include a space reservation unit, a status setting unit, a data transmission unit, and a directory update unit.

[0146] Space reservation unit is used to reserve cache space for context cache unit at the target location when the target cache layout strategy is a copy-sharing strategy or an ownership migration strategy.

[0147] The status setting unit is used to set the status of the directory entry corresponding to the context cache unit to "copying" or "migrating".

[0148] The data transmission unit is used to transmit the data of the context buffer unit to the target location.

[0149] The directory update unit is used to update the directory entry corresponding to the context cache unit in the global cache directory after the data at the target location has passed verification.

[0150] It should be noted that each module and unit in the above device embodiments can be used to execute the corresponding steps in the aforementioned method embodiments. For specific implementation processes, please refer to the relevant descriptions in the aforementioned method embodiments, which will not be repeated here.

[0151] See Figure 7 This illustration shows a structural schematic diagram of an electronic device provided in an embodiment of this application; as shown Figure 7 As shown, the electronic device 700 of this embodiment includes: at least one processor 710 ( Figure 7 Only one is shown in the diagram), memory 720, and computer program 721 stored in said memory 720 and executable on said at least one processor 710, wherein the processor 710 executes said computer program 721 to implement the steps in the above-described context cache management method embodiment.

[0152] The electronic device 700 can be a standalone cache management device, server, or intelligent computing device, or it can be a context cache management node, main intelligent chip, or other electronic device component integrated into a multi-intelligent chip collaborative inference system. The electronic device 700 may include, but is not limited to, a processor 710 and a memory 720. Those skilled in the art will understand that... Figure 7 This is merely an exemplary structure of the electronic device 700 and does not constitute a limitation on the electronic device 700. The electronic device 700 may include more or fewer components than shown, or combine certain components, or use different component connection methods.

[0153] The processor 710 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), microcontroller units (MCUs), image signal processing units, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 710 can be any conventional or special-purpose processing unit that performs control, scheduling, data processing, or interface management functions.

[0154] In some embodiments, the memory 720 may be an internal storage unit of the electronic device 700, such as an on-chip memory, a cache, a register array, or other semiconductor storage unit; in other embodiments, the memory 720 may also be an external storage unit communicatively connected to the electronic device 700. Furthermore, the memory 720 may simultaneously include both internal and external storage units.

[0155] In specific implementation, the processor 710, memory 720, and computer program 721 described in the embodiments of this application can execute the steps in the context cache management method embodiments of this application, which will not be repeated here.

[0156] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0157] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0158] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0159] In the embodiments provided in this application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / simulator embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0160] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0161] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0162] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0163] The implementation of all or part of the processes in the methods of the above embodiments can also be accomplished by a computer program product. When the computer program product is run on an electronic device, the electronic device can implement the steps in the various method embodiments described above.

[0164] The embodiments described above are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for managing context caching, characterized in that, A context cache management system comprising multiple smart chips and at least one context cache management node; the management method includes: Based on the cache attribute information of the context cache unit, update the directory entry in the global cache directory corresponding to the context cache unit; Based on the task execution relationship information and the cache access association information corresponding to the directory item, the set of subsequent access smart chips of the context cache unit within the prediction window is determined. The task execution relationship information is used to characterize the task execution relationship when the multiple smart chips perform inference tasks, and the cache access association information is used to characterize the subsequent access relationship of the context cache unit. Based on the subsequent access smart chip set, the access prediction information of the context cache unit, and the cache transfer information, the strategy cost of each candidate cache layout strategy is determined. Based on the policy cost of each candidate cache layout strategy, the target cache layout strategy is determined; The cache layout of the context cache unit is adjusted according to the target cache layout strategy.

2. The management method as described in claim 1, characterized in that, The step of determining the set of smart chips for subsequent access within the prediction window of the context cache unit based on the cache access association information and task execution relationship information corresponding to the directory entry includes: Based on the task execution relationship information, determine the set of candidate smart chips that are qualified to access within the prediction window; Based on the cache access association information corresponding to the directory item and the task execution relationship information, determine the access score corresponding to each candidate smart chip in the candidate smart chip set; The set of subsequent access smart chips is determined based on the access scores corresponding to each candidate smart chip.

3. The management method as described in claim 2, characterized in that, The step of determining the access score corresponding to each candidate smart chip in the candidate smart chip set based on the cache access association information corresponding to the directory entry and the task execution relationship information includes: Based on the cache access association information corresponding to the directory entry, determine the number of accesses, the most recent access time, and the number of active requests of the context cache unit relative to the candidate smart chip; Based on the task execution relationship information, determine the future scheduling position of the candidate intelligent chip within the prediction window; The access score corresponding to the candidate smart chip is determined based on the number of accesses, the most recent access time, the number of active requests, and the future scheduling position.

4. The management method as described in claim 1, characterized in that, The step of determining the set of smart chips for subsequent access within the prediction window of the context cache unit based on the cache access association information and task execution relationship information corresponding to the directory entry includes: Based on the cache access association information corresponding to the directory entry, determine the target cache unit associated with the context cache unit; Based on the task execution relationship information, determine the execution smart chip corresponding to the target cache unit; Based on the execution smart chip corresponding to the target cache unit, determine the set of subsequent access smart chips for the context cache unit.

5. The method as described in claim 4, characterized in that, The step of determining the target cache unit associated with the context cache unit based on the cache access association information corresponding to the directory entry includes: From the set of context cache units corresponding to the same inference task, determine the candidate associated cache unit corresponding to the context cache unit; Based on the spatial-temporal association information in the cache access association information, the default association information between the context cache unit and the candidate association cache unit is determined; Dynamic association information is determined based on at least one of the feature similarity, attention association, motion association, and common access degree between the context cache unit and the candidate association cache unit; Based on the default association information and the dynamic association information, the association score between the context cache unit and the candidate association cache unit is determined; The target cache unit is determined from the candidate associated cache units based on the association score.

6. The management method as described in claim 1, characterized in that, The step of determining the strategy cost of each candidate cache layout strategy based on the subsequent access smart chip set, the access prediction information of the context cache unit, and the cache transfer information includes: Based on the set of subsequent access smart chips, a candidate cache layout strategy corresponding to the context cache unit is determined. The candidate cache layout strategy includes at least two of the following: remote access strategy, replication sharing strategy, ownership migration strategy, and offload strategy. When the candidate cache layout strategy includes a remote access strategy, the strategy cost of the remote access strategy is determined based on the expected number of accesses in the access prediction information and the remote read cost in the cache transmission information. When the candidate cache layout strategy includes a replication-sharing strategy, the strategy cost of the replication-sharing strategy is determined based on the replication transmission cost, local access cost, and space occupation cost in the cache transmission information. When the candidate cache layout strategy includes an ownership migration strategy, the strategy cost of the ownership migration strategy is determined based on the migration transmission cost, post-migration read cost, and directory update cost in the cache transmission information. When the candidate cache layout strategy includes an offloading strategy, the strategy cost of the offloading strategy is determined based on the offloading transmission cost in the cache transmission information, the migration probability in the access prediction information, and the external storage occupation cost.

7. The management method as described in claim 6, characterized in that, The cached transmission information is determined based on a pre-defined lookup table or a piecewise linear model; The lookup table or the piecewise linear model is generated based on at least one of the following: read latency, write latency, link latency, and effective bandwidth, corresponding to the cache block size, source and target storage levels, and link type.

8. The management method as described in claim 1, characterized in that, The step of determining the target cache layout strategy based on the strategy cost of each candidate cache layout strategy includes: For a candidate cache layout strategy with a target storage location, determine whether the candidate cache layout strategy meets the capacity constraint based on the available capacity of the target storage location. For candidate cache layout strategies that require data transmission to be completed before the first access, it is determined whether the candidate cache layout strategy meets the completion time constraint based on the expected completion time corresponding to the candidate cache layout strategy and the expected first access time corresponding to the context cache unit. From the candidate cache layout strategies that satisfy the corresponding constraints, determine the candidate cache layout strategy with the minimum strategy cost; If the candidate cache layout strategy with the lowest strategy cost satisfies a preset benefit condition relative to the current cache layout corresponding to the directory entry, the candidate cache layout strategy with the lowest strategy cost is determined as the target cache layout strategy.

9. The management method as described in claim 1, characterized in that, The step of adjusting the cache layout of the context cache unit according to the target cache layout strategy includes: When the target cache layout strategy is a replication-sharing strategy or an ownership migration strategy, cache space is reserved for the context cache unit at the target location; Set the status of the directory entry corresponding to the context cache unit to "copying" or "migrating"; The context cache unit data is transferred to the target location; After the data at the target location passes verification, the directory entry corresponding to the context cache unit in the global cache directory is updated.

10. A context caching management system, characterized in that, The management system includes multiple smart chips and at least one context cache management node; The context cache management node is used to execute the context cache management method according to any one of claims 1 to 9, so as to manage the context cache in the collaborative inference process of the multiple smart chips.

Images