Cache management method, inference method, cache management unit, inference unit, electronic device, and storage medium

Abstract

Embodiments of the present application provide a cache management method, an inference method, a cache management unit, an inference unit, an electronic device and a storage medium. The cache management method comprises: receiving a data access instruction, the data access instruction comprising verification information of a session and a predetermined virtual address for storing inference cache data of the session; comparing the verification information of the data access instruction with target verification information, the target verification information being verification information associated with the predetermined virtual address; if the verification information of the data access instruction and the target verification information are the same, reading inference cache data from a physical address associated with the predetermined virtual address, and sending the read inference cache data to a source unit of the data access instruction; and if the verification information of the data access instruction and the target verification information are not the same, no longer reading inference cache data.

Description

Technical Field

[0001] This application relates to the field of large language model technology, and in particular to a cache management method, inference method, cache management unit, inference unit, electronic device, and storage medium. Background Technology

[0002] In recent years, Large Language Models (LLMs) have achieved revolutionary breakthroughs in fields such as natural language processing, intelligent dialogue, and content generation. To meet the service demands of high-concurrency scenarios, advanced LLM inference systems generally adopt a single-process, multi-session architecture. In this architecture, multiple user sessions are executed concurrently within the same process space to share model parameters and fully utilize hardware computing resources.

[0003] To accelerate the decoding phase of the inference process, some technical solutions employ key-value caching (KVCache) technology to maintain a set of key-value pairs for each session, used to store intermediate computation results generated during inference. The cached key-value pairs, or KVCache, serve as the core state data of the session, and their security is paramount. However, current technologies typically manage the KVCaches of different sessions within a unified virtual and physical address. This allows a computational unit accessing the KVCache of one session to potentially access the KVCaches of other sessions, leading to the leakage of KVCaches from those other sessions. Summary of the Invention

[0004] In view of this, embodiments of this application provide a cache management method, an inference method, a cache management unit, an inference unit, an electronic device, and a storage medium to at least solve or alleviate the above-mentioned problems.

[0005] According to a first aspect of the embodiments of this application, a cache management method is provided, comprising: receiving a data access instruction, the data access instruction including session verification information and a predetermined virtual address for storing inference cache data of the session; comparing the verification information of the data access instruction with the target verification information, the target verification information being verification information associated with the predetermined virtual address; if the verification information of the data access instruction and the target verification information are the same, then reading inference cache data from the physical address associated with the predetermined virtual address and sending the read inference cache data to the source unit of the data access instruction; if the verification information of the data access instruction and the target verification information are different, then no further reading of inference cache data.

[0006] According to a second aspect of the embodiments of this application, an inference method is provided, comprising: after receiving session information, determining the verification information of the session according to the session to which the session information belongs, and obtaining a predetermined virtual address for storing inference cache data of the session; generating a data access instruction according to the verification information of the session to which the session information belongs and the predetermined virtual address, and sending the data access instruction to a cache management unit so that the cache management unit executes the cache management method described in any of the above embodiments; if inference cache data sent by the cache management unit is received, performing inference operations based on the received inference cache data; if inference cache data not received by the cache management unit, stopping the execution of inference operations.

[0007] According to a third aspect of the embodiments of this application, a cache management unit is provided, comprising: a receiving subunit configured to receive a data access instruction, the data access instruction including session verification information and a predetermined virtual address for storing inference cache data of the session; a verification subunit configured to compare the verification information of the data access instruction with target verification information, the target verification information being verification information associated with the predetermined virtual address; and a data transmission subunit configured to, if the verification information of the data access instruction and the target verification information are the same, read inference cache data from the physical address associated with the predetermined virtual address and send the read inference cache data to the source unit of the data access instruction; and if the verification information of the data access instruction and the target verification information are different, not read the inference cache data.

[0008] According to a fourth aspect of the embodiments of this application, an inference unit is provided, comprising: a preparation subunit configured to, upon receiving session information, determine verification information of the session according to the session to which the session information belongs, and obtain a predetermined virtual address for storing inference cache data of the session; an access subunit configured to, based on the verification information of the session to which the session information belongs and the predetermined virtual address, generate a data access instruction, and send the data access instruction to a cache management unit, so that the cache management unit executes the cache management method described in any of the above embodiments; and an inference subunit configured to, if the access subunit receives inference cache data sent by the cache management unit, perform inference operations based on the received inference cache data; and if the access subunit does not receive inference cache data sent by the cache management unit, stop performing inference operations.

[0009] According to a fifth aspect of the present application, an electronic device is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; the memory is used to store at least one executable instruction, wherein the executable instruction causes the processor to execute the method described in any of the above embodiments.

[0010] According to a sixth aspect of the present application, a computer storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the methods described in any of the above embodiments.

[0011] In this embodiment, a data access instruction is received. The verification information of the data access instruction is compared with target verification information, which is the verification information associated with a predetermined virtual address. If the verification information of the data access instruction and the target verification information are the same, inference cache data is read from the physical address associated with the predetermined virtual address, and the read inference cache data is sent to the source unit of the data access instruction. If the verification information of the data access instruction and the target verification information are different, inference cache data is not read. By introducing a verification information-based comparison mechanism, a mandatory verification information comparison step is added to the data access path, establishing a security gate for the cached data of each session. Whether the predetermined virtual address is calculated incorrectly due to a software defect or an unauthorized access is maliciously initiated, as long as the verification information carried by the data access instruction does not match the verification information associated with the predetermined virtual address, inference cache data cannot be read. This effectively prevents cross-session cache data leakage and pollution, ensuring the security of session cached data. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0013] Figure 1 This is a system architecture diagram of the inference system of some optional embodiments of this application.

[0014] Figure 2 This is a flowchart of the steps of a cache management method in some optional embodiments of this application.

[0015] Figure 3 This is a flowchart of the steps of the reasoning method in some optional embodiments of this application.

[0016] Figure 4This is a structural block diagram of the cache management unit in some optional embodiments of this application.

[0017] Figure 5 This is a structural block diagram of the inference unit in some optional embodiments of this application.

[0018] Figure 6 This is a schematic diagram of an electronic device according to some optional embodiments of this application. Detailed Implementation

[0019] The present application is described below based on embodiments, but it is not limited to these embodiments. In the detailed description of the present application below, certain specific details are described in detail. Those skilled in the art can fully understand the present application without these details. To avoid obscuring the essence of the present application, well-known methods, processes, and flows are not described in detail. Furthermore, the accompanying drawings are not necessarily drawn to scale.

[0020] refer to Figure 1 The diagram illustrates a system architecture of an inference system. System 100 includes, but is not limited to, an interaction unit 110, an execution engine 120, a computing unit 130, and a memory 140.

[0021] The interaction unit 110 is used to receive conversation information input by the user and send the conversation information to the execution engine 120, wherein the conversation information can be the dialogue content input by the user.

[0022] The execution engine 120 may include a driver unit 121 and a cache management unit 122. The driver unit 121 coordinates the execution order of inference tasks corresponding to multiple concurrent sessions, avoids resource conflicts, and allocates a unique verification information to each new session. The cache management unit 122 manages the virtual and physical addresses of the storage space. After the execution engine 120 receives the session information of a new session, the cache management unit 122 allocates virtual and physical addresses for the inference cache data (i.e., KVCache) of that session. Furthermore, the cache management unit 122 can execute the cache management method described in the following embodiments to manage access to the inference cache data. Optionally, the cache management unit 122 may include an MMU (Memory Management Unit) to execute the cache management method.

[0023] The computation unit 130 is used to run computation graphs of large language models such as Transformer. After the execution engine 120 sends session information to the computation unit 130, the computation unit 130 can perform corresponding inference computations based on the received session information, such as inference computations using a Transformer architecture with a KVCache optimization mechanism, to generate corresponding session responses and complete the dialogue response. When the computation unit 130 needs to read inference cache data, it generates a data access instruction and sends the data access instruction to the cache management unit 122 so that the cache management unit can read the required inference cache data.

[0024] The memory 140 is used to store inference cache data for the session, and the cache management unit 122 can manage the physical address and the corresponding virtual address of its storage space.

[0025] Cache management methods:

[0026] This application provides a cache management method to at least partially solve the problem of potential KVCache leakage.

[0027] Figure 2 This is a flowchart of a cache management method according to an embodiment of this application. Figure 2 As shown, this cache management method includes the following steps:

[0028] S210. Receive a data access instruction, which includes session verification information and a predetermined virtual address for storing inference cache data of the session.

[0029] The data access instruction received in this embodiment can be a data access instruction issued by the computing unit 130. When the computing unit 130 needs to read inference cache data for inference calculation of the current session, it will issue a request to read the inference cache data, i.e., a data access instruction. This instruction includes at least a predetermined virtual address and session verification information. The predetermined virtual address is a pre-allocated virtual address used to address the inference cache data it needs. The session verification information corresponds one-to-one with the session, and can be an identifier such as a session ID specific to the session, or a password specifically set for the session. The session verification information is used to verify whether the virtual address is associated with a certain session, i.e., to verify whether the inference cache data stored at the virtual address belongs to a certain session. The verification information associated with the predetermined virtual address is the verification information of the session to which the inference cache data stored at the predetermined virtual address belongs.

[0030] S220. Compare the verification information of the data access instruction with the target verification information, whereby the target verification information is the verification information associated with a predetermined virtual address. The target verification information can be obtained from a specified storage space, and its specific storage location is not limited in this embodiment.

[0031] S231. If the verification information of the data access instruction is the same as the target verification information, then read the inference cache data from the physical address associated with the predetermined virtual address, and send the read inference cache data to the source unit of the data access instruction.

[0032] The physical address associated with the virtual address is the physical address of the storage space actually used to store the data, which is pointed to by the virtual address. The source unit of the data access instruction can be the aforementioned computing unit 130, thereby enabling the computing unit 130 to obtain inference cache data and perform corresponding inference calculations based on the inference data. For specific inference calculation steps, please refer to relevant technologies, which will not be elaborated here.

[0033] S232. If the verification information of the data access instruction is different from the target verification information, then the inference cache data will not be read again.

[0034] In some optional embodiments, the verification information of the data access instruction is compared with the target verification information. If the verification information of the data access instruction and the target verification information are the same, inference cache data is read from the physical address associated with the predetermined virtual address, including:

[0035] The target verification information is queried in the first address mapping page table according to the predetermined virtual address, and the verification information of the data access instruction is compared with the queried target verification information. The first address mapping page table includes multiple virtual addresses, as well as verification information and physical addresses associated with each virtual address.

[0036] The first address mapping page table can be used to represent the relationship between virtual address, check information, and physical address. For example, associated virtual address, check information, and physical address can be located in the same row or the same page table entry of the first address mapping page table. Of course, the first address mapping page table can also represent the relationship between virtual address, check information, and physical address in other suitable ways, which will not be elaborated here. The first address mapping page table can be stored in a predetermined memory, such as in the page table area of ​​the memory used to store page tables. After receiving a data access instruction, the data access instruction or its check information and the predetermined virtual address can be stored in a predetermined register. When comparing the check information of the data access instruction with the target check information, the first address mapping page table can be retrieved from the predetermined memory, and the check information and the predetermined virtual address can be retrieved from the predetermined register.

[0037] As a feasible implementation, querying target verification information in the first address mapping page table based on a predetermined virtual address may include: querying the first address mapping page table for a virtual address identical to the predetermined virtual address; after finding a virtual address identical to the predetermined virtual address, using the verification information associated with that virtual address in the first address mapping page table as the target verification information. For example, when a virtual address identical to the predetermined virtual address is found, the verification information and physical address located in the same row or the same page table entry can be queried, and this verification information and physical address can be used as the verification information and physical address associated with the predetermined virtual address. In some optional embodiments, a certain number of historical first address mapping page tables can be stored in a TLB (Translation Lookaside Buffer), which can be the most recently used first address mapping page tables. When querying the first address mapping page table for a virtual address identical to the predetermined virtual address, the predetermined virtual address can be queried first in the first address mapping page table in the TLB (Translation Lookaside Buffer). If no virtual address identical to the predetermined virtual address is found, the predetermined virtual address can then be queried in the first address mapping page table stored in a predetermined memory.

[0038] If the verification information of the data access instruction is the same as the queried target verification information, then the physical address associated with the predetermined virtual address is retrieved from the first address mapping page table based on the predetermined virtual address, and the inference cache data is read from the physical address associated with the predetermined virtual address. When retrieving the physical address associated with the predetermined virtual address from the first address mapping page table based on the predetermined virtual address, the physical address associated with the aforementioned virtual address in the first address mapping page table can be used as the physical address associated with the predetermined virtual address. It should be understood that the aforementioned predetermined virtual address refers to the same virtual address as described above.

[0039] In this embodiment, if the verification information of the data access instruction and the target verification information are different, the inference cache data will no longer be read. This may include: if the verification information of the data access instruction and the target verification information are different, the physical address associated with the predetermined virtual address will no longer be queried, and the operation of reading the inference cache data will be stopped.

[0040] By integrating verification information with the physical address into the first address mapping page table, security verification performed by comparing verification information can be seamlessly embedded into the inherent physical address translation process. This makes security verification a necessary, pre-processing step in physical address translation. When the security verification, i.e., the verification information of the data access instruction, differs from the target verification information, the physical address access process can be terminated immediately early in the physical address translation process, before the physical address is queried. This achieves a rapid termination mechanism for data access, ensuring the security of inference cache data and avoiding unnecessary physical address queries and memory access operations, thus saving computational resources. Furthermore, by extending the verification information field in the page table entries of the first address mapping page table, logical isolation between session virtual addresses can be achieved, overcoming the limitations of physical hardware storage resources. This allows for support of hundreds of concurrent sessions within a single process, fully utilizing hardware computing resources.

[0041] In some optional embodiments, if the verification information of the data access instruction and the target verification information are different, the physical address associated with the predetermined virtual address will not be queried, including:

[0042] If the verification information of the data access instruction is different from the target verification information, the inference cache data will no longer be read, and an exception message indicating that an exception has occurred will be issued.

[0043] For example, an exception prompt can be sent to the interaction unit 110, causing the interaction unit 110 to display an access exception to the user. Alternatively, the exception prompt can be sent to other suitable units, such as the driving unit 121 or the source unit of the data access instruction, to trigger the driving unit 121 or the source unit of the data access instruction to execute a preset security handling process, such as recording a security log or terminating the session's inference operations. This enhances the defense against abnormal access and provides protection for identifying and tracking potential security threats. Of course, an exception prompt can also be sent to the driving unit 121, and then sent from the driving unit 121 to the interaction unit 110. This embodiment of the application does not limit the transmission path of the exception prompt.

[0044] In existing technologies, incorrect address acquisition may lead some sessions to obtain virtual addresses corresponding to other sessions, causing them to access the physical address storing inference cache data of other sessions based on the virtual address. Alternatively, unauthorized access may change the virtual address corresponding to a session to the virtual address of another session, thereby accessing the physical address storing inference cache data of other sessions based on the virtual address, in order to illegally obtain the inference cache data of other sessions.

[0045] In this embodiment, a data access instruction is received. The verification information of the data access instruction is compared with target verification information, which is the verification information associated with a predetermined virtual address. If the verification information of the data access instruction and the target verification information are the same, inference cache data is read from the physical address associated with the predetermined virtual address, and the read inference cache data is sent to the source unit of the data access instruction. If the verification information of the data access instruction and the target verification information are different, inference cache data is not read. By introducing a verification information-based comparison mechanism, a mandatory verification information comparison step is added to the data access path, establishing a security gate for the cached data of each session. Whether the predetermined virtual address is calculated incorrectly due to a software defect or an unauthorized access is maliciously initiated, as long as the verification information carried by the data access instruction does not match the verification information associated with the predetermined virtual address, inference cache data cannot be read. This effectively prevents cross-session cache data leakage and pollution, ensuring the security of session cached data.

[0046] In some optional embodiments, the method further includes, before receiving a data access instruction:

[0047] The system receives session verification information; allocates a predetermined virtual address and a physical address associated with the predetermined virtual address to the session. Based on the predetermined virtual address, the verification information associated with the predetermined virtual address, and the allocated physical address, it generates page table entries in a first address mapping page table. Each page table entry may include a set of associated verification information, a virtual address, and a physical address.

[0048] Optionally, the verification information can be the verification information sent by the driver unit 121. When the execution engine 120 receives the first session information of a new session, the driver unit 121 can assign a unique verification information to the session and send the verification information to the cache management unit 122.

[0049] In this embodiment, when receiving session verification information, a predetermined virtual address and a physical address associated with the predetermined virtual address are allocated to the session. Page table entries in the first address mapping page table are generated based on the predetermined virtual address, the verification information associated with the predetermined virtual address, and the allocated physical address. This allows the mapping relationship between the verification information, the corresponding virtual address, and the physical address to be written into the first address mapping page table simultaneously with the allocation of physical memory for the session. This ensures that the session's inference cache data is protected by the verification information from the moment it is stored in the physical address, thereby better protecting the security of the inference cache data.

[0050] In some optional embodiments, a predetermined virtual address and a physical address associated with the predetermined virtual address are assigned to the session, including:

[0051] Assign a reserved virtual address to the session and a dedicated storage space to the session, and set the physical address of the dedicated storage space to the physical address associated with the reserved virtual address.

[0052] When allocating a predetermined virtual address for a session, a contiguous range of virtual addresses of a specified size can be drawn from all unoccupied virtual addresses to obtain the predetermined virtual address. The allocated storage space is the storage space of memory 140, which can be one or more physical page frames of memory 140. The process of allocating the predetermined virtual address and physical address can be executed by the aforementioned driving unit, directly by the cache management unit 122, or by other suitable hardware or software units; this application does not impose any limitations on this. The dedicated storage space allocated for a session is used only to store the inference cache data of that session and not for storing the inference cache data of other sessions. Of course, in some optional embodiments, the dedicated storage space allocated for a session can be reclaimed. In this case, the storage space will no longer belong exclusively to that session and can be allocated to other sessions to store their inference cache data.

[0053] In this embodiment, dedicated storage space can be allocated to a session, and the physical address of the dedicated storage space can be set to a physical address associated with a predetermined virtual address. This separates the inference cache data of different sessions at the physical storage space level, avoiding the problem of easy leakage of inference cache data caused by mixed storage of inference cache data of different sessions.

[0054] In some optional embodiments, the verification information of the data access instruction is compared with the target verification information. If the verification information of the data access instruction and the target verification information are the same, inference cache data is read from the physical address associated with the predetermined virtual address, including:

[0055] The physical address associated with a predetermined virtual address is retrieved from the second address mapping page table. The second address mapping page table includes multiple virtual addresses and the physical addresses associated with each virtual address. The second address mapping page table can be stored in a predetermined memory, such as a page table area used for storing page tables. When it is necessary to retrieve information such as a physical address from the second address mapping page table, it can be retrieved from the predetermined memory. It should be understood that the second address mapping page table can be used to represent the association between virtual addresses and physical addresses. For example, associated virtual addresses and physical addresses can be located in the same row or the same page table entry in the second address mapping page table. When a virtual address identical to the predetermined virtual address is found in the second address mapping page table, the physical address associated with that virtual address is retrieved from the first address mapping page table by querying the physical address located in the same row or the same page table entry. Of course, the second address mapping page table can also represent the association between virtual addresses and physical addresses in other suitable ways, which will not be elaborated here.

[0056] The verification information of the data access instruction and the physical address associated with the predetermined virtual address are written into the Direct Memory Access (DMA) descriptor.

[0057] The direct memory access controller compares the verification information in the direct memory access descriptor with the target verification information. In this embodiment, a physical address and the verification information associated with that physical address can be stored in a predetermined memory. Before comparing the verification information in the direct memory access descriptor with the target verification information, the verification information associated with the physical address can be queried from the predetermined memory based on the physical address in the direct memory access descriptor, thereby obtaining the target verification information.

[0058] If the verification information in the Direct Memory Access Descriptor is the same as the target verification information, then the inference cache data is read from the physical address associated with the predetermined virtual address through the Direct Memory Access controller. DMA is a technology in computer systems that allows related units to directly transfer data with memory. Its data transfer is performed through the DMA controller, without CPU intervention, enabling high-speed data transfer and reducing CPU load. In this embodiment, inference cache data can be read from the physical address associated with the predetermined virtual address through the DMA controller.

[0059] If the verification information of the data access instruction and the target verification information are different, then the inference cache data will not be read again, including:

[0060] If the verification information in the direct memory access descriptor is different from the target verification information, then the inference cache data will no longer be read through the direct memory access controller.

[0061] In this embodiment of the application, by embedding session verification information in the DMA descriptor, the DMA controller can verify the access permission by comparing the verification information in the direct memory access descriptor with the target verification information before reading the inference cache data. This adds security protection to the DMA transfer process and prevents unauthorized access from obtaining the inference cache data of other sessions through DMA.

[0062] In some optional embodiments, retrieving the physical address associated with the predetermined virtual address from a first address mapping page table based on the predetermined virtual address, and reading inference cache data from the physical address associated with the predetermined virtual address includes:

[0063] Based on the predetermined virtual address, the system queries the first address mapping page table for the physical address associated with the predetermined virtual address. The verification information of the data access instruction and the physical address associated with the predetermined virtual address are then written into the direct memory access descriptor. The direct memory access controller compares the verification information in the direct memory access descriptor with the target verification information. If the verification information in the direct memory access descriptor matches the target verification information, the system reads the inference cache data from the physical address associated with the predetermined virtual address through the direct memory access controller.

[0064] If the verification information of the data access instruction is different from the target verification information, the inference cache data will no longer be read. This includes: if the verification information in the direct memory access descriptor is different from the target verification information, the inference cache data will no longer be read through the direct memory access controller.

[0065] In this embodiment, the scheme of verifying the verification information through the first address mapping page table can be combined with the scheme of verifying the verification information in the DMA transmission stage, which can protect the inference cache data at multiple levels and further avoid the leakage of inference cache data.

[0066] Reasoning method:

[0067] like Figure 3 As shown in the embodiments of this application, a reasoning method is also provided, including:

[0068] S310. After receiving the session information, determine the session verification information according to the session to which the session information belongs, and obtain the reserved virtual address for storing the inference cache data of the session.

[0069] In some optional embodiments, all session information input by the same user can be categorized into the same session, and a first mapping relationship and a second mapping relationship can be preset. The user ID of the user is mapped to the corresponding session ID through the first mapping relationship, so as to mark the session to which the user's session information belongs through the session ID. The session ID is mapped to the session's verification information through the second mapping relationship. Of course, other first mapping relationships can also be set to map other known information bound to the user to the session ID through the first mapping relationship. This application embodiment does not limit this. After the interaction unit 110 receives the session information input by the user, it can send the session information to the execution engine 120. After receiving the session information, the execution engine 120 can determine the session ID of the session to which the session information belongs through the driver unit 121 according to the user ID of the user who input the session information and the first mapping relationship. According to the session ID and the second mapping relationship, it determines the verification information of the session. And obtain the predetermined virtual address from the predetermined memory that stores the predetermined virtual address used to store the inference cache data of the session.

[0070] S320. Based on the verification information of the session to which the session information belongs and the predetermined virtual address, generate a data access instruction and send the data access instruction to the cache management unit so that the cache management unit executes the cache management method in any of the above embodiments.

[0071] When the driving unit 121 determines that it needs to execute the inference task corresponding to the session, it can send the session's verification information and the predetermined virtual address to the computing unit 130. When it is necessary to read inference cache data, the computing unit 130 can generate a data access instruction based on the session's verification information and the predetermined virtual address, and send the data access instruction to the cache management unit. The cache management unit can execute the cache management method in any of the above embodiments based on the data access instruction.

[0072] S331. If inference cache data sent by the cache management unit is received, inference operations are performed based on the received inference cache data.

[0073] If the cache management unit sends the read inference cache data to the computing unit 130, the computing unit 130 can perform inference operations based on the received inference cache data after receiving the inference cache data sent by the cache management unit, so as to complete the inference task corresponding to the session.

[0074] S332. If no inference cache data is received from the cache management unit, the inference operation is stopped.

[0075] If the cache management unit no longer queries the physical address associated with the predetermined virtual address, it can no longer send inference cache data to the computing unit 130. The computing unit 130 can stop performing inference operations if it does not receive inference cache data from the cache management unit after a predetermined time has elapsed. This prevents inference results from being derived based on erroneous or illegally accessed inference cache data, ensuring the security and accuracy of the inference results. It should be understood that the predetermined time should be greater than the time required for the cache management unit to send the read inference cache data to the computing unit 130.

[0076] The reasoning method in this embodiment is based on the same inventive concept as the aforementioned cache management method embodiment and has the beneficial effects of the corresponding method embodiment, which will not be repeated here. The implementation of the relevant steps in the reasoning method of this embodiment can be referred to the description of the corresponding parts in the aforementioned cache management method embodiment, which will also not be repeated here.

[0077] Cache management unit:

[0078] like Figure 4 As shown in the figure, this application embodiment also provides a cache management unit 400, including:

[0079] The receiving subunit 410 is configured to receive a data access instruction, which includes session verification information and a predetermined virtual address for storing inference cache data of the session.

[0080] The verification subunit 420 is configured to compare the verification information of the data access instruction with the target verification information, wherein the target verification information is the verification information associated with a predetermined virtual address.

[0081] The data transmission subunit 430 is configured to read inference cache data from the physical address associated with the predetermined virtual address and send the read inference cache data to the source unit of the data access instruction if the verification information of the data access instruction and the target verification information are the same; otherwise, it will not read the inference cache data.

[0082] The cache management unit 400 of this embodiment is based on the same inventive concept as the aforementioned cache management method embodiments, and is used to implement the corresponding cache management methods in the aforementioned multiple method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here. The functional implementation of each module or unit in the cache management unit 400 of this embodiment can refer to the description of the corresponding part in the aforementioned cache management method embodiments, which will also not be repeated here.

[0083] Reasoning Unit:

[0084] like Figure 5As shown in the embodiment of this application, an inference unit 500 is also provided, including:

[0085] The preparation subunit 510 is configured to, upon receiving session information, determine the session verification information based on the session to which the session information belongs, and obtain a predetermined virtual address for storing the inference cache data of the session.

[0086] Access subunit 520 is configured to generate a data access instruction based on the verification information of the session to which the session information belongs and a predetermined virtual address, and send the data access instruction to the cache management unit so that the cache management unit executes the cache management method described in any of the above embodiments;

[0087] The inference subunit 530 is configured to perform inference operations based on the received inference cache data if the access subunit 520 receives inference cache data sent by the cache management unit; and to stop performing inference operations if the access subunit 520 does not receive inference cache data sent by the cache management unit.

[0088] The reasoning unit 500 of this embodiment is based on the same inventive concept as the aforementioned reasoning method embodiments, and is used to implement the corresponding reasoning methods in the aforementioned multiple method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here. The functional implementation of each module or unit in the reasoning unit 500 of this embodiment can refer to the description of the corresponding part in the aforementioned reasoning method embodiments, which will also not be repeated here.

[0089] Electronic devices:

[0090] Figure 6 This is a schematic block diagram of an electronic device provided in an embodiment of this application. Specific embodiments of this application do not limit the specific implementation of the electronic device. Figure 6 As shown, the electronic device may include: a processor 602, a communications interface 604, a memory 606, and a communications bus 608.

[0091] in:

[0092] The processor 602, communication interface 604, and memory 606 communicate with each other via communication bus 608.

[0093] Communication interface 604 is used to communicate with other electronic devices or servers, such as the aforementioned source device.

[0094] The processor 602 is used to execute computer software 610, specifically to execute the relevant steps in the embodiments of the cache management method or inference method described above.

[0095] Specifically, computer software 610 may include program code, which includes executable instructions, which may be computer operation instructions.

[0096] The processor 602 may be a CPU, an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The smart device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0097] Memory 606 is used to store computer software 610. Memory 606 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0098] The computer software 610 may include multiple executable instructions. Specifically, the computer software 610 may use multiple executable instructions to cause the processor 602 to perform the operations corresponding to the cache management method or inference method described in any of the foregoing embodiments.

[0099] The specific implementation of each step in computer software 610 can be found in the corresponding descriptions of the steps and units in the above method embodiments, and has corresponding beneficial effects, which will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.

[0100] Computer storage media:

[0101] This application also provides a computer storage medium storing a computer program thereon. When executed by a processor, the program implements the methods described in any of the embodiments of the aforementioned cache management method or inference method. The computer storage medium includes, but is not limited to, compact disc read-only memory (CD-ROM), random access memory (RAM), floppy disk, hard disk, or magneto-optical disk.

[0102] It should be noted that the user-related information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to sample data used for training the model, data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0103] It should be noted that, depending on the implementation needs, the various components / steps described in the embodiments of this application can be broken down into more components / steps, or two or more components / steps or parts of the operation of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of this application.

[0104] The methods described in the embodiments of this application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code downloaded over a network that is originally stored in a remote recording medium or a non-transitory machine-readable medium and will be stored in a local recording medium. Thus, the methods described herein can be processed by software stored on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., RAM, ROM, flash memory, etc.) capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods described herein. Furthermore, when a general-purpose computer accesses code used to implement the methods shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for executing the methods shown herein.

[0105] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0106] The above embodiments are only used to illustrate the embodiments of this application, and are not intended to limit the embodiments of this application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of this application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of this application, and the patent protection scope of the embodiments of this application should be defined by the claims.

Claims

1. A cache management method, comprising: Receive a data access instruction, the data access instruction including session verification information and a predetermined virtual address for storing inference cache data of the session, the session and the verification information correspond one-to-one; The verification information of the data access instruction is compared with the target verification information, wherein the target verification information is the verification information associated with the predetermined virtual address; If the verification information of the data access instruction is the same as the target verification information, then the inference cache data is read from the physical address associated with the predetermined virtual address, and the read inference cache data is sent to the source unit of the data access instruction; wherein, the physical address associated with the predetermined virtual address is a physical address of the storage space dedicated to the corresponding session, and the storage space dedicated to the corresponding session is only used to store the inference cache data of that session, and not used to store the inference cache data of other sessions. If the verification information of the data access instruction is different from the target verification information, then the inference cache data will no longer be read; the step of no longer reading the inference cache data includes: stopping the operation of reading the inference cache data.

2. The method according to claim 1, wherein, The verification information of the data access instruction is compared with the target verification information. If the verification information of the data access instruction and the target verification information are the same, inference cache data is read from the physical address associated with the predetermined virtual address, including: The target verification information is queried in the first address mapping page table according to the predetermined virtual address, and the verification information of the data access instruction is compared with the queried target verification information; wherein, the first address mapping page table includes multiple virtual addresses, as well as verification information and physical addresses associated with each virtual address; If the verification information of the data access instruction is the same as the target verification information found, then the physical address associated with the predetermined virtual address is queried from the first address mapping page table according to the predetermined virtual address, and inference cache data is read from the physical address associated with the predetermined virtual address. If the verification information of the data access instruction and the target verification information are different, then the inference cache data will not be read again, including: If the verification information of the data access instruction is different from the target verification information, then the physical address associated with the predetermined virtual address will no longer be queried.

3. The method according to claim 2, wherein, Before receiving the data access instruction, the method further includes: Receive the verification information of the session; allocate the predetermined virtual address and the physical address associated with the predetermined virtual address to the session; generate page table entries in the first address mapping page table based on the predetermined virtual address, the verification information associated with the predetermined virtual address and the allocated physical address.

4. The method according to claim 3, wherein, The process of allocating the predetermined virtual address and the physical address associated with the predetermined virtual address to the session includes: Assign the predetermined virtual address to the session and allocate dedicated storage space to the session, and set the physical address of the dedicated storage space to the physical address associated with the predetermined virtual address.

5. The method according to claim 1, wherein, The step of comparing the verification information of the data access instruction with the target verification information, and if the verification information of the data access instruction and the target verification information are the same, then reading the inference cache data from the physical address associated with the predetermined virtual address, includes: The physical address associated with the predetermined virtual address is queried in the second address mapping page table according to the predetermined virtual address, wherein the second address mapping page table includes multiple virtual addresses and physical addresses associated with each virtual address; Write the verification information of the data access instruction and the physical address associated with the predetermined virtual address into the direct memory access descriptor; The direct memory access controller compares the verification information in the direct memory access descriptor with the target verification information. If the verification information in the direct memory access descriptor is the same as the target verification information, then the inference cache data is read from the physical address associated with the predetermined virtual address through the direct memory access controller. If the verification information of the data access instruction and the target verification information are different, then the inference cache data will not be read again, including: If the verification information in the direct memory access descriptor is different from the target verification information, then the inference cache data will no longer be read through the direct memory access controller.

6. The method according to any one of claims 1-5, wherein, If the verification information of the data access instruction and the target verification information are different, then the physical address associated with the predetermined virtual address will no longer be queried, including: If the verification information of the data access instruction is different from the target verification information, the inference cache data will no longer be read, and an exception prompt will be issued to indicate that an exception has occurred.

7. A reasoning method, comprising: Upon receiving session information, the verification information of the session is determined based on the session to which the session information belongs, and a predetermined virtual address for storing the inference cache data of the session is obtained. Based on the verification information of the session to which the session information belongs and the predetermined virtual address, a data access instruction is generated and sent to the cache management unit so that the cache management unit executes the method of any one of claims 1-6; If inference cache data sent by the cache management unit is received, inference operations are performed based on the received inference cache data; if inference cache data is not received from the cache management unit, inference operations are stopped.

8. A cache management unit, comprising: The receiving subunit is configured to receive a data access instruction, the data access instruction including session verification information and a predetermined virtual address for storing inference cache data of the session, wherein the session corresponds one-to-one with the verification information; The verification subunit is configured to compare the verification information of the data access instruction with the target verification information, wherein the target verification information is the verification information associated with the predetermined virtual address; The data transmission subunit is configured to, if the verification information of the data access instruction and the target verification information are the same, read inference cache data from the physical address associated with the predetermined virtual address and send the read inference cache data to the source unit of the data access instruction; wherein, the physical address associated with the predetermined virtual address is a physical address of a storage space dedicated to the corresponding session, and the storage space dedicated to the corresponding session is only used to store the inference cache data of that session and not to store the inference cache data of other sessions; if the verification information of the data access instruction and the target verification information are different, then the inference cache data is no longer read; the no-reading of inference cache data includes: stopping the operation of reading inference cache data.

9. A reasoning unit, comprising: The preparation subunit is configured to, upon receiving session information, determine the verification information of the session based on the session to which the session information belongs, and obtain a predetermined virtual address for storing the inference cache data of the session. The access subunit is configured to generate a data access instruction based on the verification information of the session to which the session information belongs and the predetermined virtual address, and send the data access instruction to the cache management unit so that the cache management unit executes the method of any one of claims 1-6; The inference subunit is configured to perform inference operations based on the received inference cache data if the access subunit receives inference cache data sent by the cache management unit; and to stop performing inference operations if the access subunit does not receive inference cache data sent by the cache management unit.

10. An electronic device, comprising: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform the method as described in any one of claims 1-7.

11. A computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any one of claims 1-7.

Images