Reasoning result generation method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a reasoning result generation method and device, electronic equipment and storage medium, and relates to the fields of large models, natural language processing, distributed storage, intelligent cloud and the like. The method can include: determining a target request to be processed; determining a target page corresponding to the target request from a hot page stored in high-bandwidth memory of a target chip and / or a cold page stored in low-bandwidth memory of the target chip according to the principle of querying the high-bandwidth memory of the target chip first and then querying the low-bandwidth memory of the target chip if not all hits are obtained, wherein the key-value pages of the key-value cache generated in the reasoning process of the reasoning engine are divided, and each key-value page is determined as a hot page or a cold page according to the access frequency, and the reasoning engine runs on the target chip; generating a reasoning task according to the target page, sending the reasoning task to the reasoning engine, and obtaining the reasoning result generated by the reasoning engine and returning it. The present disclosure can reduce reasoning delay and the like.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, particularly to the fields of large models, natural language processing, distributed storage and intelligent cloud, and especially to methods, devices, electronic devices and storage media for generating inference results. Background Technology

[0002] Currently, in the process of promoting information technology application innovation (IT innovation) transformation, power companies are required to migrate their core business systems to designated chip platforms, such as domestically produced Neural Processing Units (NPUs). These chips differ significantly from general-purpose graphics processing units (GPUs) in terms of memory hierarchy and instruction set architecture. For example, these chips typically employ a two-tier storage hierarchy of high-bandwidth memory (HBM) and low-bandwidth memory. Summary of the Invention

[0003] This disclosure provides a method, apparatus, electronic device, and storage medium for generating inference results.

[0004] A method for generating reasoning results, comprising:

[0005] Identify the target request to be processed;

[0006] Following the principle of prioritizing querying the high-bandwidth memory of the target chip, and querying the low-bandwidth memory of the target chip if not all pages are hit, the target page corresponding to the target request is determined from the hot pages stored in the high-bandwidth memory and / or the cold pages stored in the low-bandwidth memory. In this process, the key-value cache generated by the inference engine is divided into key-value pages, and each key-value page is determined as the hot page or the cold page according to the access frequency. The inference engine runs on the target chip.

[0007] A reasoning task is generated based on the target page, the reasoning task is sent to the reasoning engine, and the reasoning result generated by the reasoning engine is returned.

[0008] An inference result generation device includes: a request scheduling module, a hierarchical compression module, and a quantization execution module;

[0009] The request scheduling module is used to determine the target request to be processed;

[0010] The layered compression module is used to determine the target page corresponding to the target request from the hot pages stored in the high-bandwidth memory and / or the cold pages stored in the low-bandwidth memory, according to the principle of first querying the high-bandwidth memory of the target chip and then querying the low-bandwidth memory of the target chip if not all pages are hit. The key-value cache generated during the inference process of the inference engine is divided into key-value pages, and each key-value page is determined as the hot page or the cold page according to the access frequency. The inference engine runs on the target chip.

[0011] The quantization execution module is used to generate an inference task based on the target page, send the inference task to the inference engine, and return the inference result generated by the inference engine.

[0012] An electronic device, comprising:

[0013] At least one processor; and

[0014] A memory communicatively connected to the at least one processor; wherein,

[0015] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described above.

[0016] A non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the methods described above.

[0017] A computer program product includes a computer program / instructions that, when executed by a processor, implement the method described above.

[0018] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0019] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0020] Figure 1 This is a flowchart of an embodiment of the reasoning result generation method described in this disclosure;

[0021] Figure 2 This is a flowchart of an embodiment of the inference request scheduling method described in this disclosure;

[0022] Figure 3 This is a flowchart illustrating an embodiment of the method for obtaining the target page as described in this disclosure;

[0023] Figure 4 A flowchart illustrating an embodiment of the vocabulary tag mapping table generation method described in this disclosure;

[0024] Figure 5 A flowchart illustrating an embodiment of the method for generating the quantization embedding matrix as described in this disclosure;

[0025] Figure 6 A flowchart illustrating an embodiment of the target embedding matrix generation method described in this disclosure;

[0026] Figure 7 This is a schematic diagram of the composition structure of the first embodiment 700 of the reasoning result generation device described in this disclosure;

[0027] Figure 8 This is a schematic diagram of the composition structure of the second embodiment 800 of the reasoning result generation device described in this disclosure;

[0028] Figure 9 A schematic block diagram of an electronic device 900 that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0029] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0030] Furthermore, it should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0031] Figure 1 This is a flowchart illustrating an embodiment of the reasoning result generation method described in this disclosure. Figure 1 As shown, the specific implementation methods are as follows.

[0032] In step 101, the target request to be processed is determined.

[0033] In step 102, following the principle of prioritizing querying the high-bandwidth memory of the target chip, and querying the low-bandwidth memory of the target chip if not all pages are hit, the target page corresponding to the target request is determined from the hot pages stored in the high-bandwidth memory and / or the cold pages stored in the low-bandwidth memory. Specifically, the key-value cache (KV Cache) generated during the inference process of the inference engine is divided into key-value pages, and each key-value page is determined as a hot page or a cold page according to the access frequency. The inference engine runs on the target chip.

[0034] In step 103, an inference task is generated based on the target page, the inference task is sent to the inference engine, and the inference result generated by the inference engine is returned.

[0035] During the inference process of the inference engine, the read and write of key-value cache is the main performance bottleneck. For chips that use a two-level storage hierarchy of high-bandwidth memory and low-bandwidth memory, the traditional approach is to store all key-value cache in high-bandwidth memory or all in low-bandwidth memory. Although high-bandwidth memory has sufficient bandwidth, its capacity is limited. Storing all key-value cache can easily trigger data eviction and recalculation due to insufficient space. On the other hand, low-bandwidth memory has low bandwidth, and storing all key-value cache will face bandwidth bottlenecks. In short, either approach may cause increased inference latency and decreased throughput.

[0036] The scheme described in the above method embodiment can divide the key-value cache into key-value pages, and further distinguish them into hot pages and cold pages based on access frequency, storing them in high-bandwidth memory and low-bandwidth memory respectively. Accordingly, when processing a target request, the hot pages in high-bandwidth memory can be queried first, and if not all are hit, the cold pages in low-bandwidth memory can be queried. This hierarchical scheduling method makes reasonable use of the characteristics of two-level storage, avoids the defects of a single storage method as much as possible, and thus reduces inference latency and improves inference throughput.

[0037] It should be noted that the target requests and inference results in the embodiments described in this disclosure are not targeted at any specific user and are not intended to reflect the personal information of any specific user. The collection, storage, use, processing, transmission, provision, and disclosure of any type of information, such as user personal information, involved in the technical solutions of this disclosure comply with relevant laws and regulations and do not violate public order and good morals.

[0038] In some embodiments of this disclosure, in response to receiving any inference request, the request type of the inference request can be identified first. If the request type is a realtime request, the inference request can be added to the real-time queue. If the request type is an offline request, the inference request can be added to the offline queue. Accordingly, in each scheduling cycle, in response to the real-time queue being non-empty, M inference requests can be taken from the real-time queue and determined as target requests to be processed, where M is a positive integer. In response to the real-time queue being empty and the offline queue being non-empty, N inference requests can be taken from the offline queue and determined as target requests to be processed, where N is a positive integer and M is less than N.

[0039] Taking power dispatching systems as an example, there are usually two types of inference requests. One type is real-time queries from dispatchers, such as "suggestions for handling after the current main transformer protection action", which requires low-latency response and has relatively relaxed throughput requirements. The other type is offline knowledge retrieval and document parsing tasks (such as batch analysis of historical fault reports), which are relatively insensitive to latency but require high throughput. Traditional methods lack differentiated dispatching methods for different request types.

[0040] By adopting the above processing method, differentiated scheduling of real-time and offline requests can be achieved. Real-time requests are executed first through small-batch scheduling to avoid being blocked by large-batch offline tasks, effectively ensuring low-latency response of real-time requests and meeting the service level agreement (SLA) constraints of power dispatching and other systems. Offline requests are executed through large-batch scheduling, which can make full use of computing resources and maintain high throughput. In other words, it can ensure the response speed of real-time requests while taking into account the processing efficiency of offline tasks, and achieve a balance of performance in both scenarios.

[0041] Based on the above introduction, Figure 2 This is a flowchart illustrating an embodiment of the inference request scheduling method described in this disclosure. Figure 2 As shown, the specific implementation methods are as follows.

[0042] In step 201, an inference request is obtained.

[0043] In step 202, determine whether the request type is a real-time request or an offline request. If it is a real-time request, proceed to step 203; if it is an offline request, proceed to step 204.

[0044] For example, by parsing the type flag field in the inference request, it can be determined whether the inference request is a real-time request or an offline request.

[0045] In step 203, the inference request is added to the real-time queue (Queue), and then step 205 is executed.

[0046] In step 204, the inference request is added to the offline queue.

[0047] It can maintain two independent request queues, one for storing real-time requests and the other for storing offline requests. Each queue can adopt a first-in-first-out (FIFO) queue structure.

[0048] In step 205, in each scheduling cycle, it is determined whether the real-time queue is empty. If not, step 206 is executed; if so, step 207 is executed.

[0049] In step 206, M inference requests are retrieved from the real-time queue, identified as target requests, and then the process ends.

[0050] The specific value of M can be determined according to actual needs, but it is usually small to reduce batch assembly waiting time, etc.

[0051] In step 207, determine whether the offline queue is empty. If not, proceed to step 208; if yes, end the process.

[0052] In step 208, N inference requests are retrieved from the offline queue, identified as target requests, and then the process ends.

[0053] The specific value of N can be determined according to actual needs, but it is usually large in order to improve resource utilization and throughput of offline request processing, etc.

[0054] For example, if 3 real-time requests and 50 offline requests are acquired at a certain time, the real-time requests can be added to the real-time queue and the offline requests can be added to the offline queue. The real-time queue can be scheduled first, such as scheduling these 3 real-time requests as a batch, and then scheduling the offline queue after the real-time queue is empty.

[0055] In addition, in some embodiments of this disclosure, one or any combination of the following may also be performed: 1) In response to the inference engine needing to process a real-time request while it is processing an offline request, a preemption event indication is sent to the inference engine to instruct it to pause processing of the offline request and save the intermediate state of the key-value cache corresponding to the offline request. After the real-time request is processed, the processing of the offline request is resumed according to the intermediate state; 2) In response to the target request being a real-time request and the inference engine not generating an inference result within a specified time, a degradation event indication is sent to the inference engine to instruct it to generate a degradation result according to a predetermined degradation strategy. The degradation result is the inference result after content reduction; 3) In response to obtaining the inference result, an audit log corresponding to the target request is generated. The audit log includes: the request type of the target request, the number of target requests, the proportion of hot pages in the target page, whether a preemption event occurred during the processing of the target request, and whether a degradation event occurred during the processing of the target request.

[0056] While the inference engine is processing an offline request, it may need to process a real-time request for some reason. In this case, a preemption mechanism can be triggered, which sends a preemption event indicator (Preemption-Signal) to the inference engine. Accordingly, the inference engine can pause the processing of the offline request and save the intermediate state of the key-value cache corresponding to the offline request. For example, the inference engine can pause the processing of the offline request after completing the currently processing decoding step, without immediately interrupting it, to avoid errors in subsequent recovery due to incomplete key-value cache intermediate state. The key-value cache intermediate state refers to the key-value cache corresponding to the intermediate results that have been generated but have not yet completed all output. When saving the intermediate state, the hot page corresponding to the offline request can be migrated from high-bandwidth memory to low-bandwidth memory to release high-bandwidth memory resources. After the real-time request is processed, the processing of the offline request can be resumed according to the intermediate state. The key-value page (original hot page) migrated to low-bandwidth memory can also be migrated back to high-bandwidth memory to continue executing the remaining decoding steps, etc.

[0057] The decoding step is a single forward computation process in which the Large Language Model (LLM) of the inference engine generates inference results word by word. Each time the model completes a decoding step, it outputs a word (token). Multiple iterations of decoding generate the complete inference result. Furthermore, in each decoding step, the Large Language Model performs attention calculations based on the current input and historical key-value caches (such as the target page), thereby generating the current word and its corresponding key-value cache.

[0058] By adopting the above processing method and using the preemption mechanism, real-time requests can be processed in a timely manner without waiting for all offline requests to be completed, thereby effectively shortening the response latency of real-time requests. At the same time, the intermediate state of the key-value cache of offline requests is completely saved and can be seamlessly restored after the real-time request is processed, avoiding duplicate calculations and wasted computing power caused by interruption. Thus, it balances the low-latency response of real-time requests with the processing efficiency of offline requests.

[0059] Additionally, if the target request is a real-time request and the inference engine fails to generate an inference result within the specified time, a degradation mechanism can be triggered. This involves sending a degradation event instruction to the inference engine, which instructs the engine to generate a degradation result based on a predetermined degradation strategy. The degradation result is the inference result with reduced content.

[0060] For example, the scheduling timestamp of the target request (the timestamp taken from the queue) can be recorded, and a delay budget can be configured for the target request according to actual needs. Accordingly, the time consumed in the inference process can be continuously monitored.

[0061] That is:

[0062] Telapsed(t)=Tcurrent-Tarrive; (1)

[0063] Where Tcurrent represents the current time, Tarrive represents the scheduling timestamp, and Telapsed(t) represents the elapsed time.

[0064] Telapsed(t) can be compared with Tbudget, and if the following conditions are met:

[0065] Telapsed(t)>γ*Tbudget; (2)

[0066] Then a degradation event indication can be sent to the inference engine. γ represents the warning coefficient, and the specific value can be determined according to actual needs. It is usually greater than 0 and less than 1. Tbudget represents the delay budget.

[0067] Accordingly, the inference engine can directly use the currently generated content (generated vocabulary) as the degradation result, that is, terminate the inference early and return the truncated result. Alternatively, if the generated content includes sentence-ending punctuation, such as a period or question mark, it can be truncated to the nearest sentence-ending punctuation to improve the semantic integrity of the obtained degradation result.

[0068] The aforementioned degradation mechanism can be triggered in a timely manner when the inference time approaches the latency budget limit, thus preventing real-time requests from violating the SLA due to excessive inference time.

[0069] In addition, after obtaining the inference results, an audit log corresponding to the target request can be generated. The audit log may include: the request type of the target request, the number of target requests, the proportion of hot pages in the target page, whether a preemption event occurred during the processing of the target request, and whether a degradation event occurred during the processing of the target request. If necessary, the audit log may also include other information, such as batch identifier, end-to-end latency, scheduling timestamp, etc.

[0070] This allows for the generation of corresponding audit logs for each batch of scheduling, which may include:

[0071] batch_id: Batch identifier, which can be used for cross-system tracking;

[0072] batch_type: The type of request to be made, whether it is a real-time request or an offline request;

[0073] request_count: The number of target requests;

[0074] e2e_latency_ms: End-to-end latency (milliseconds), which can be the time interval between the scheduling timestamp and the acquisition of the inference result;

[0075] kv_hit_rate: The percentage of hot pages in the target page (hot page hit rate);

[0076] preemption_occurred: Whether a preemption event occurred (Boolean value);

[0077] degradation_occurred: Whether a degradation event occurred (Boolean value);

[0078] timestamp: The scheduling timestamp.

[0079] Audit logs can be written to the operation and maintenance audit component of the power dispatching system in a structured format to support SLA compliance monitoring, inference service performance analysis (such as hot spot hit rate trend analysis), and fault tracing, thereby meeting the information security and compliance audit requirements of the power industry and ensuring the observability of the information technology innovation system.

[0080] For the identified target request, the target page corresponding to the target request can be determined from the hot pages stored in the high-bandwidth memory and / or the cold pages stored in the low-bandwidth memory, following the principle of prioritizing querying high-bandwidth memory and querying low-bandwidth memory if not all pages are hit. Specifically, the key-value cache generated during the inference process of the inference engine is divided into key-value pages, and each key-value page is determined as a hot page or a cold page according to the access frequency.

[0081] For example, the key-value cache can be divided into multiple key-value pages according to a fixed size (i.e., segmented). Each key-value page includes multiple key-value tensors of consecutive words, and a global page table can be maintained to record the metadata information of each key-value page.

[0082] The metadata information may include:

[0083] page_id: The page identifier of the key-value page;

[0084] session_id: The identifier of the session to which the key-value page belongs;

[0085] token_range: The starting and ending index range of the words covered by the key-value page;

[0086] storage_tier: The current storage location. For example, HOT can be used to represent high-bandwidth memory, and COLD can be used to represent low-bandwidth memory.

[0087] access_freq: Normalized access frequency (floating-point number, updated exponentially);

[0088] last_access_ts: The timestamp of the most recent access;

[0089] compressed_size: The number of compressed bytes when stored in low-bandwidth memory (0 when stored in high-bandwidth memory).

[0090] In some embodiments of this disclosure, L newly generated key-value pages during the inference engine's generation of inference results can also be obtained and identified as pages to be processed, where L is a positive integer. In response to the remaining space of the high-bandwidth memory being able to accommodate the pages to be processed, the pages to be processed can be identified as hot pages and stored in the high-bandwidth memory. In response to the remaining space being unable to accommodate the pages to be processed, the L pages with the lowest comprehensive scores can be identified from the hot pages stored in the high-bandwidth memory and stored in the low-bandwidth memory. The pages to be processed can also be identified as hot pages and stored in the high-bandwidth memory.

[0091] In some embodiments of this disclosure, in response to P target pages being located in low-bandwidth memory, the P target pages can be identified as pages to be processed, where P is a positive integer and less than or equal to the total number of target pages. In response to the remaining space in high-bandwidth memory being able to accommodate the pages to be processed, the pages to be processed can be identified as hot pages and stored in high-bandwidth memory. In response to the remaining space being unable to accommodate the pages to be processed, the P pages with the lowest comprehensive scores can be identified from the hot pages stored in high-bandwidth memory, the pages to be migrated can be stored in low-bandwidth memory, and the pages to be processed can be identified as hot pages and stored in high-bandwidth memory.

[0092] This allows for the "new replace old" approach to storing newly generated key-value pages from the inference engine in high-bandwidth memory, while simultaneously migrating the hottest pages with the lowest overall score to low-bandwidth memory. Conversely, cold pages accessed by the inference engine and located in low-bandwidth memory can be migrated to high-bandwidth memory, and the hottest pages with the lowest overall score can also be migrated to low-bandwidth memory. This enables dynamic migration of key-value pages between high-bandwidth and low-bandwidth memory. Given the limited capacity of high-bandwidth memory, this approach continuously retains frequently accessed, high-value key-value pages in high-bandwidth memory while relegating infrequently accessed, low-value key-value pages to low-bandwidth memory. This maintains a high hot-page hit rate over the long term, reduces access frequency to low-bandwidth memory, effectively lowers inference latency, and fully leverages the performance advantages of a two-tier storage architecture.

[0093] Based on the above introduction, Figure 3 This is a flowchart illustrating an embodiment of the method for obtaining the target page described in this disclosure. Figure 3 As shown, the specific implementation methods are as follows.

[0094] In step 301, it is determined whether all target pages are located in high-bandwidth memory. If so, step 302 is executed; otherwise, step 303 is executed.

[0095] In step 302, the target page is retrieved from high-bandwidth memory, and then the process ends.

[0096] In step 303, the target page is retrieved from low-bandwidth memory.

[0097] In this embodiment, to simplify the process, it is assumed that all target pages are located in low-bandwidth memory. However, in practical applications, some target pages may be located in high-bandwidth memory, while the remaining target pages may be located in low-bandwidth memory.

[0098] In step 304, the target page is identified as the page to be processed, and it is determined whether the remaining space of the high-bandwidth memory can accommodate the page to be processed. If so, step 305 is executed; otherwise, step 306 is executed.

[0099] In step 305, the page to be processed is identified as a hot page and stored in high-bandwidth memory, after which the process ends.

[0100] In step 306, based on the number of pages to be processed, the number of pages with the lowest comprehensive score are determined from the hot pages stored in high-bandwidth memory. The pages to be migrated are then stored in low-bandwidth memory, and the pages to be processed are determined as hot pages and stored in high-bandwidth memory. The process then ends.

[0101] In some embodiments of this disclosure, the method for obtaining the comprehensive score of any hot page may include: obtaining the latest access frequency of the hot page determined based on the exponential decay mechanism, obtaining the difference between the most recent access time of the hot page and the current time, and determining the comprehensive score based on the access frequency and the difference.

[0102] The access frequency is updated in each decoding step, and the specific update method is as follows:

[0103] fp(t)=ρ*fp(t−1)+(1−ρ)*1[The key-value page is accessed at time t]; (3)

[0104] Where fp(t) represents the access frequency of the current decoding step, fp(t−1) represents the access frequency of the previous decoding step, ρ represents the exponential decay coefficient, the specific value of which can be determined according to actual needs, usually between 0 and 1, 1[key page is accessed at time t] represents the indicator function, indicating whether the key page is accessed by the inference engine in the current decoding step. If so, 1[key page is accessed at time t] is 1, otherwise it is 0.

[0105] Using an exponential decay mechanism to determine the access frequency of key-value pages can mitigate the interference from long-standing historical access behavior, accurately reflect the recent access popularity of key-value pages, and avoid historically frequently accessed pages occupying high bandwidth memory for extended periods.

[0106] In addition, the difference between the most recent access time of a hot page and the current time can be obtained, and a comprehensive score can be determined based on the latest access frequency and the difference.

[0107] If possible:

[0108] Score=β*fp(t)+(1−β)*(1 / (Δtp+ϵ)); (4)

[0109] Wherein, Score represents the overall score, Δtp represents the difference, ϵ represents a small positive number to prevent division by zero, and β represents the weighting coefficient, which is used to balance access frequency and access timeliness. The specific value can be determined according to actual needs.

[0110] The above comprehensive scoring method can take into account both access frequency and access timeliness. It combines the Least Frequently Used (LFU) and Least Recently Used (LRU) strategies to maximize the retention of high-frequency hot pages, continuously improve the hit rate of hot pages, and reduce inference latency caused by cross-memory reads.

[0111] For key-value pages newly added to high-bandwidth memory, if the key-value page was previously located in low-bandwidth memory, its storage_tier field can be updated from COLD to HOT. If the key-value page is a newly generated key-value page by the inference engine, the storage_tier field can be directly set to HOT. In addition, in either case, the access_freq field can be initialized to a higher baseline value to prevent the key-value page from being migrated to low-bandwidth memory in a short period of time.

[0112] In some embodiments of this disclosure, when storing a page to be migrated in low-bandwidth memory, the page to be migrated can first be quantized and compressed, and then the compression result can be stored in low-bandwidth memory. Correspondingly, when a page to be processed is determined as a hot page and stored in high-bandwidth memory, in response to the page to be processed coming from low-bandwidth memory, the page to be processed can be decompressed (inverse quantized) and then stored in high-bandwidth memory.

[0113] By performing quantization compression, the storage space occupied by low-bandwidth memory can be reduced, and the reading time can be reduced. The quantization compression can be a page-level symmetric linear quantization. The quantization coefficient can be determined based on the maximum value of the absolute value of all values ​​of the key-value tensor of the page to be migrated. Then, quantization compression is completed based on the quantization coefficient. The compression result and the quantization coefficient can be stored in low-bandwidth memory at the same time. During decompression, the compression result can be restored to the key-value page using the quantization coefficient and stored in high-bandwidth memory.

[0114] In some embodiments of this disclosure, a quantized embedding matrix corresponding to the vocabulary of a large language model can also be generated. The large language model is a large language model mounted on the inference engine. The method of generating an inference task based on the target page may include: determining the target embedding matrix corresponding to the target request based on the quantized embedding matrix, and generating an inference task based on the target page and the target embedding matrix, thereby improving the accuracy of the subsequent inference results.

[0115] The vocabulary is a predefined set of all available words in a large language model. It contains all the smallest semantic units that the large language model can recognize and process, and each word has a unique number.

[0116] Specifically, in some embodiments of this disclosure, the method for generating the quantized embedding matrix corresponding to the vocabulary may include: obtaining a vocabulary tag mapping table, which includes: each word in the vocabulary and the vocabulary type corresponding to each word, the vocabulary type including highly sensitive words and low sensitive words, and generating a quantized embedding matrix based on the vocabulary tag mapping table.

[0117] Power dispatch texts typically exhibit significant lexical distribution characteristics: equipment numbers, key words in operation instructions (such as closing, opening, and tripping), and numerical parameters (such as 220kV and protection settings) have a decisive impact on the accuracy of inference results and are usually considered highly sensitive words. In contrast, general descriptive words (such as conjunctions, modal particles, and common nouns) have a smaller impact on the accuracy of inference results and are usually considered low-sensitive words. Traditional methods do not differentiate between these lexical types.

[0118] By adopting the above processing method, a vocabulary tag mapping table can be used to record the vocabulary type of different words, and a quantized embedding matrix can be generated based on the vocabulary tag mapping table, thus laying a good foundation for subsequent differentiated processing of different words.

[0119] In some embodiments of this disclosure, the method of generating a vocabulary tag mapping table may include: obtaining the sensitivity score of each word in the vocabulary list, determining words with a sensitivity score greater than or equal to a first threshold as highly sensitive words, and determining words with a sensitivity score less than the first threshold as low sensitive words; or, sorting each word in descending order of sensitivity score, determining the words in the first Q positions after sorting as highly sensitive words, and determining the remaining words as low sensitive words, where Q is a positive integer greater than 1.

[0120] The specific value of the first threshold can be determined according to actual needs. Using the above method, the classification of highly sensitive and low-sensitivity words can be completed quickly and accurately, thereby improving the generation efficiency of the quantization embedding matrix and the accuracy of the generated results.

[0121] In some embodiments of this disclosure, the method for obtaining the sensitivity score of each word in the vocabulary may include: obtaining a domain corpus corresponding to the target request domain, obtaining the word frequency feature value of each word in the domain corpus, extracting a portion of sample data from the domain corpus, performing forward inference on each sample data using a large language model, and extracting the attention weight matrix of the last R layer attention head of the large language model, where R is a positive integer, determining the attention contribution score (ACS) of each word based on the attention weight matrix, and combining the word frequency feature value and the attention contribution score to determine the sensitivity score of each word.

[0122] By adopting the above processing method, the sensitivity score can be calculated by combining word frequency feature values ​​and the attention contribution of the model. This allows the sensitivity score to reflect both the professionalism of each word in the domain text and the actual influence of each word in the reasoning process, thereby improving the accuracy of word classification results.

[0123] Based on the above introduction, Figure 4This is a flowchart illustrating an embodiment of the method for generating a vocabulary tag mapping table as described in this disclosure. Figure 4 As shown, the specific implementation methods are as follows.

[0124] In step 401, the vocabulary of the large language model mounted on the inference engine is obtained, and the domain corpus corresponding to the target request is obtained.

[0125] Taking power dispatching systems as an example, the domain corpus may include unstructured text such as equipment ledgers, dispatching procedures, operation instruction history records, and fault reports.

[0126] In step 402, the word frequency feature values ​​of each word in the vocabulary list are obtained in the domain corpus.

[0127] For example, the texts in the domain corpus can be segmented first. Then, for each word in the vocabulary, the following processing can be performed: obtain the term frequency-inverse document frequency (TF-IDF) of the word in different texts, take the average of each TF-IDF, and normalize the average to the range of 0 to 1 to obtain the term frequency feature value of the word.

[0128] In step 403, a portion of sample data is extracted from the domain corpus.

[0129] For example, a predetermined number of typical sample corpora can be extracted.

[0130] In step 404, the large language model is used to perform forward inference on each sample corpus, and the attention weight matrix of the last R layer attention head of the large language model is extracted. The attention contribution of each word is determined based on the attention weight matrix.

[0131] For example, during the forward inference process of each sample corpus using a large language model, the attention weight matrix of each attention head in the final R layer of the large language model can be extracted. Each element in the attention weight matrix represents the degree of attention a query position pays to a certain key position. For each word in the vocabulary, the position of the word in each sample corpus can be determined. For each layer in each attention head, the average of the attention weights of all query positions when the position of the word is used as the key can be obtained to get the attention score of the word in the sample corpus, the attention head, and the layer. Then, the average of the attention scores of the word in all sample corpora, all attention heads, and all layers can be taken to obtain the attention contribution of the word.

[0132] The attention contribution reflects the actual influence degree of a vocabulary on the final reasoning result during the reasoning process of a large language model, and a higher attention contribution indicates a greater influence degree.

[0133] In step 405, the sensitivity score of each vocabulary is determined by combining the word frequency feature value and the attention contribution.

[0134] For example, there may be:

[0135] Sz=λ1*S1+λ2*S2; (5)

[0136] Wherein, Sz represents the sensitivity score, S1 represents the word frequency feature value, S2 represents the attention contribution, λ1 and λ2 are weight hyperparameters, λ1+λ2=1, and the specific values can be determined according to actual needs.

[0137] In step 406, vocabularies with a sensitivity score greater than or equal to a first threshold are determined as high-sensitivity vocabularies, and vocabularies with a sensitivity score less than the first threshold are determined as low-sensitivity vocabularies.

[0138] For example, assume that the vocabulary table includes a total of 32000 vocabularies, wherein the word frequency feature value of the vocabulary "closing" is 0.85, the attention contribution is 0.91, and the sensitivity score is 0.88, which is greater than the first threshold 0.60, so this vocabulary can be determined as a high-sensitivity vocabulary; the word frequency feature value of the vocabulary "but" is 0.12, the attention contribution is 0.09, and the sensitivity score is 0.11, which is less than the first threshold 0.60, so this vocabulary can be determined as a low-sensitivity vocabulary; finally, a total of 2560 vocabularies are determined as high-sensitivity vocabularies.

[0139] In step 407, a vocabulary label mapping table is generated according to each vocabulary and the vocabulary type of each vocabulary.

[0140] A quantized embedding matrix can be generated according to the vocabulary label mapping table. In some embodiments of the present disclosure, for each vocabulary in the vocabulary label mapping table, the following processing may be performed respectively: in response to the vocabulary being a high-sensitivity vocabulary, the embedding vector of the vocabulary is stored in a high precision partition (HP-Partition) matrix in the form of a high-precision floating point number; in response to the vocabulary being a low-sensitivity vocabulary, the embedding vector of the vocabulary is quantized and compressed to obtain a low-precision integer vector, the low-precision integer vector is stored in a low precision partition (LP-Partition) matrix, and the high precision partition matrix and the low precision partition matrix are used to form the quantized embedding matrix.

[0141] Correspondingly, Figure 5 is the flowchart of the embodiment of the method for generating a quantized embedding matrix described in the present disclosure. As Figure 5 As shown, the specific implementation methods are as follows.

[0142] In step 501, each word in the word tag mapping table is processed according to the methods shown in steps 502 to 504.

[0143] In step 502, it is determined whether the word is a highly sensitive word. If so, step 503 is executed; otherwise, step 504 is executed.

[0144] In step 503, the embedding vector of the word is saved in high-precision floating-point form into a high-precision partitioning matrix, and then step 505 is executed.

[0145] The word embedding vectors can be fixed parameters that have already been learned by a large language model.

[0146] In step 504, the embedding vector of the word is quantized and compressed to obtain a low-precision integer vector, and the low-precision integer vector is saved into a low-precision partitioning matrix.

[0147] The quantization compression can be a word-level symmetric linear quantization, which can save both the low-precision integer vector and the quantization coefficients into a low-precision partition matrix.

[0148] In step 505, a quantization embedding matrix is ​​formed using a high-precision partitioning matrix and a low-precision partitioning matrix, and then the process ends.

[0149] By employing differentiated embedding vector storage methods for words of different types, a quantized embedding matrix containing both high-precision and low-precision partitioning matrices can be generated. High-sensitivity words retain high-precision floating-point embedding vectors, which can fully preserve the semantic features of the words and avoid precision loss of important information due to quantization compression, thus ensuring the inference accuracy of large language models. Low-sensitivity words are compressed into low-precision integer vectors using symmetric linear quantization, which can reduce the overall storage space of the quantized embedding matrix and the overhead of off-chip memory access.

[0150] For example, suppose the vocabulary contains 32,000 words, of which 2,560 are highly sensitive words and the rest are low-sensitive words. The embedding vectors of highly sensitive words can be stored in a high-precision partitioning matrix in high-precision floating-point form (about 512 bytes per word), while the embedding vectors of low-sensitive words can be stored in a low-precision partitioning matrix in low-precision integer form (about 256 bytes per word). This reduces the overall storage of the quantized embedding matrix by about 46%, significantly reduces off-chip access overhead, and preserves the semantic features of highly sensitive words completely.

[0151] Both the vocabulary tag mapping table and the quantization embedding matrix can be generated offline.

[0152] Based on the quantized embedding matrix, the target embedding matrix corresponding to the target request can be determined. In some embodiments of this disclosure, the input word sequence corresponding to the target request can be obtained, and the following processing can be performed on each word in the input word sequence: in response to the word being a high-sensitivity word, the corresponding embedding vector is obtained from the high-precision partitioning matrix; in response to the word being a low-sensitivity word, the corresponding low-precision integer vector is obtained from the low-precision partitioning matrix, and the embedding vector is obtained by decompression; the embedding vectors corresponding to each word in the input word sequence are concatenated in word order to obtain the target embedding matrix.

[0153] Accordingly, Figure 6 This is a flowchart illustrating an embodiment of the method for generating the target embedding matrix described in this disclosure. Figure 6 As shown, the specific implementation methods are as follows.

[0154] In step 601, the input word sequence corresponding to the target request is obtained.

[0155] For example, the original text of the target request can be preprocessed, such as removing extra spaces, special garbled characters, invalid punctuation and other interfering content. Then, the preprocessed original text can be segmented to obtain each word, and arranged according to the order of appearance of each word in the original text to generate the input word sequence corresponding to the target request.

[0156] In step 602, each word in the input word sequence is processed according to the methods shown in steps 603 to 605.

[0157] In step 603, it is determined whether the word is a highly sensitive word or a low-sensitivity word. If it is a highly sensitive word, step 604 is executed; if it is a low-sensitivity word, step 605 is executed.

[0158] In step 604, the corresponding embedding vector is obtained from the high-precision partitioning matrix, and then step 606 is executed.

[0159] In step 605, the corresponding low-precision integer vector is obtained from the low-precision partition matrix, and the embedding vector is obtained by decompression.

[0160] In step 606, the embedding vectors corresponding to each word in the input word sequence are concatenated in word order to obtain the target embedding matrix.

[0161] By adopting the above processing method, the embedding vectors can be obtained by relying on the pre-built partition matrix and combining the differences in vocabulary types. The target embedding matrix can be generated by concatenating them in order, so that the inference engine can directly use the pre-built target embedding matrix for calculation, reducing the data processing overhead in the online inference path and improving inference efficiency.

[0162] For example, suppose the input word sequence contains 14 words, among which "220kV", "main transformer", "differential protection", "action", "closing", and "protection setting" are 6 highly sensitive words. Then, the corresponding embedding vectors can be obtained from the high-precision partition matrix. The remaining 8 low-sensitivity words can be obtained from the low-precision partition matrix. The embedding vectors can be obtained by decompression. Then, the embedding vectors corresponding to each word can be concatenated in order to obtain the target embedding matrix. Furthermore, an inference task can be generated based on the target page and the target embedding matrix. For example, the target page and the target embedding matrix can be combined to encapsulate input parameters, model configuration, computing resource information, etc., and integrated into a standardized inference task, which is then sent to the inference engine for execution to generate inference results. The inference results can then be returned to the requester of the target request.

[0163] Based on the above introduction, taking a power dispatching system as an example, the overall implementation process of the scheme described in this disclosure will be further explained.

[0164] Offline phase:

[0165] It can acquire a domain corpus in the power industry and determine the sensitivity score of each word in the vocabulary list based on the domain corpus. Then, it can determine the word type of each word based on the sensitivity score, i.e., whether it is a high-sensitivity word or a low-sensitivity word. Furthermore, it can generate a word tag mapping table based on each word and its corresponding word type, and can persistently store the word tag mapping table.

[0166] In addition, a quantized embedding matrix can be generated based on the vocabulary tag mapping table. Specifically, for each vocabulary in the vocabulary tag mapping table, the following processing is performed: in response to the vocabulary being a high-sensitivity vocabulary, the embedding vector of the vocabulary is saved in the form of a high-precision floating-point number in the high-precision partitioning matrix; in response to the vocabulary being a low-sensitivity vocabulary, the embedding vector of the vocabulary is quantized and compressed to obtain a low-precision integer vector, which is then saved in the low-precision partitioning matrix. The high-precision partitioning matrix and the low-precision partitioning matrix are then used to form the quantized embedding matrix. Furthermore, the quantized embedding matrix can be preloaded into the inference engine.

[0167] Online reasoning stage:

[0168] When an inference request arrives, it can be determined whether it is a real-time request or an offline request. Real-time requests can be added to the real-time queue, and offline requests can be added to the offline queue. During scheduling, the real-time queue can be scheduled first. That is, in response to the real-time queue being non-empty, M inference requests are taken from the real-time queue and identified as target requests. In response to the real-time queue being empty and the offline queue being non-empty, N inference requests are taken from the offline queue and identified as target requests.

[0169] For a target request, the target page corresponding to the target request can be determined from the hot pages stored in high-bandwidth memory and / or the cold pages stored in low-bandwidth memory, following the principle of prioritizing querying high-bandwidth memory and querying low-bandwidth memory if not all pages are hit.

[0170] For a target request, the corresponding input word sequence can be determined, and the target embedding matrix can be determined based on the input word sequence and the quantized embedding matrix. Then, an inference task can be generated based on the target page and the target embedding matrix, sent to the inference engine for processing, and the inference result generated by the inference engine can be obtained and returned to the requester of the target request.

[0171] For example, in a provincial power dispatch center, a dispatcher is handling a differential protection action event of the main transformer of a 220kV substation and needs to quickly obtain the standard handling procedure. The dispatcher can send an inference request (real-time request): "220kV main transformer differential protection action, please provide the standard handling procedure."

[0172] Assuming the target request only includes this real-time request, the corresponding target pages can be determined. Let's assume the target pages are 12 hot pages stored in high-bandwidth memory. Further, the target embedding matrix corresponding to this real-time request can be obtained, and an inference task can be generated based on the target pages and the target embedding matrix, then sent to the inference engine.

[0173] Correspondingly, the reasoning results returned by the reasoning engine can be obtained and the reasoning results can be returned to the dispatcher. For example, the reasoning results may be: 220kV main transformer differential protection action handling steps: (1) Immediately notify the operation and maintenance personnel to rush to the site; (2) Check the cause of the differential protection action and determine whether it is an internal fault; (3) If it is confirmed that the internal fault is located, isolate the faulty equipment according to the procedure and take safety measures; (4) Report the handling situation to the dispatch director and record the fault work order...

[0174] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this disclosure. Secondly, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this disclosure. Furthermore, for parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0175] The above is an introduction to the method embodiments. The following describes the solution described in this disclosure further through device embodiments.

[0176] Figure 7This is a schematic diagram of the structural composition of the first embodiment 700 of the reasoning result generation device described in this disclosure. Figure 7 As shown, it includes: a request scheduling module 701, a hierarchical compression module 702, and a quantization execution module 703.

[0177] The request scheduling module 701 is used to determine the target request to be processed.

[0178] The layered compression module 702 is used to determine the target page corresponding to the target request from the hot pages stored in the high-bandwidth memory and / or the cold pages stored in the low-bandwidth memory, according to the principle of first querying the high-bandwidth memory of the target chip and then querying the low-bandwidth memory of the target chip if not all pages are hit. In this process, the key-value cache generated during the inference engine is divided into key-value pages, and each key-value page is determined as a hot page or a cold page according to the access frequency. The inference engine runs on the target chip.

[0179] The quantization execution module 703 is used to generate inference tasks based on the target page, send the inference tasks to the inference engine, and obtain the inference results generated by the inference engine.

[0180] In some embodiments of this disclosure, the request scheduling module 701 determines the target request to be processed in the following ways: in response to obtaining any inference request, identifying the request type of the inference request; in response to the request type being a real-time request, adding the inference request to a real-time queue; in response to the request type being an offline request, adding the inference request to an offline queue; in each scheduling cycle, in response to the real-time queue being non-empty, retrieving M inference requests from the real-time queue and determining them as target requests, where M is a positive integer; in response to the real-time queue being empty and the offline queue being non-empty, retrieving N inference requests from the offline queue and determining them as target requests, where N is a positive integer and M is less than N.

[0181] In some embodiments of this disclosure, the request scheduling module 701 may also perform one or any combination of the following: In response to the inference engine needing to process a real-time request while processing an offline request, a preemption event indication is sent to the inference engine to instruct it to pause processing the offline request and save the intermediate state of the key-value cache corresponding to the offline request. After the real-time request is processed, the offline request processing is resumed based on the intermediate state. In response to the target request being a real-time request and the inference engine failing to generate an inference result within a specified time, a degradation event indication is sent to the inference engine to instruct it to generate a degradation result according to a predetermined degradation strategy. The degradation result is the inference result with reduced content. In response to obtaining the inference result, an audit log corresponding to the target request is generated. The audit log includes: the request type of the target request, the number of target requests, the proportion of hot pages in the target page, whether a preemption event occurred during the processing of the target request, and whether a degradation event occurred during the processing of the target request.

[0182] In some embodiments of this disclosure, the layered compression module 702 can obtain L newly generated key-value pages during the process of the inference engine generating inference results and determine them as pages to be processed, where L is a positive integer. In response to the remaining space of the high-bandwidth memory being able to accommodate the pages to be processed, the pages to be processed can be determined as hot pages and stored in the high-bandwidth memory. In response to the remaining space being unable to accommodate the pages to be processed, the L pages with the lowest comprehensive scores can be determined from the hot pages stored in the high-bandwidth memory, the pages to be migrated can be stored in the low-bandwidth memory, and the pages to be processed can be determined as hot pages and stored in the high-bandwidth memory.

[0183] In some embodiments of this disclosure, the hierarchical compression module 702, in response to P target pages being located in low-bandwidth memory, can determine the P target pages as pages to be processed, where P is a positive integer less than or equal to the total number of target pages; in response to the remaining space in high-bandwidth memory being able to accommodate the pages to be processed, can determine the pages to be processed as hot pages and store them in high-bandwidth memory; in response to the remaining space being unable to accommodate the pages to be processed, can determine the P pages with the lowest comprehensive score from the hot pages stored in high-bandwidth memory, store the pages to be migrated in low-bandwidth memory, and determine the pages to be processed as hot pages and store them in high-bandwidth memory.

[0184] In some embodiments of this disclosure, the method by which the layered compression module 702 obtains the comprehensive score of any hot page may include: obtaining the latest access frequency of the hot page determined based on the exponential decay mechanism, obtaining the difference between the most recent access time of the hot page and the current time, and determining the comprehensive score based on the access frequency and the difference.

[0185] In addition, in some embodiments of this disclosure, the method by which the layered compression module 702 stores the page to be migrated in low-bandwidth memory may include: quantizing and compressing the page to be migrated and then storing it in low-bandwidth memory; the method by which the page to be processed is determined as a hot page and stored in high-bandwidth memory may include: in response to the page to be processed coming from low-bandwidth memory, decompressing the page to be processed and then storing it in high-bandwidth memory.

[0186] In some embodiments of this disclosure, the quantization execution module 703 may also generate a quantization embedding matrix corresponding to the vocabulary of the large language model. The large language model is the large language model mounted on the inference engine. Accordingly, the method of generating an inference task based on the target page may include: determining the target embedding matrix corresponding to the target request based on the quantization embedding matrix, and generating an inference task based on the target page and the target embedding matrix.

[0187] In some embodiments of this disclosure, the quantization execution module 703 may generate a quantization embedding matrix corresponding to a vocabulary list by: obtaining a vocabulary tag mapping table, which includes: each word in the vocabulary list and the vocabulary type corresponding to each word, including high-sensitivity words and low-sensitivity words; and generating a quantization embedding matrix based on the vocabulary tag mapping table.

[0188] Figure 8 This is a schematic diagram of the structural composition of the second embodiment 800 of the reasoning result generation device described in this disclosure. Figure 8 As shown, it includes: a request scheduling module 701, a hierarchical compression module 702, a quantization execution module 703, and a vocabulary analysis module 704.

[0189] The vocabulary analysis module 704 can be used to generate a vocabulary tag mapping table. Specifically, it can obtain the sensitivity score of each word in the vocabulary table, identify words with a sensitivity score greater than or equal to a first threshold as high-sensitivity words, and identify words with a sensitivity score less than the first threshold as low-sensitivity words. Alternatively, it can sort the words in descending order of sensitivity score, identify the words in the top Q positions after sorting as high-sensitivity words, and identify the remaining words as low-sensitivity words, where Q is a positive integer greater than 1.

[0190] In some embodiments of this disclosure, the vocabulary analysis module 704 may obtain the sensitivity scores of each word in the vocabulary list by: obtaining a domain corpus corresponding to the target request domain; obtaining the word frequency feature values ​​of each word in the domain corpus; extracting a portion of sample data from the domain corpus; performing forward inference on each sample data using a large language model; extracting the attention weight matrix of the last R layer attention head of the large language model, where R is a positive integer; determining the attention contribution of each word based on the attention weight matrix; and combining the word frequency feature values ​​and the attention contribution to determine the sensitivity score of each word.

[0191] In some embodiments of this disclosure, the quantization execution module 703 generates the quantization embedding matrix according to the vocabulary label mapping table in the following ways: for each vocabulary in the vocabulary label mapping table, the following processing is performed respectively: in response to the vocabulary being a high-sensitivity vocabulary, the embedding vector of the vocabulary is saved in the form of a high-precision floating-point number in a high-precision partition matrix; in response to the vocabulary being a low-sensitivity vocabulary, the embedding vector of the vocabulary is quantized and compressed to obtain a low-precision integer vector; the low-precision integer vector is saved in a low-precision partition matrix; and the high-precision partition matrix and the low-precision partition matrix are used to form the quantization embedding matrix.

[0192] In some embodiments of this disclosure, the quantization execution module 703 determines the target embedding matrix corresponding to the target request based on the quantization embedding matrix in the following ways: obtaining the input word sequence corresponding to the target request, and performing the following processing on each word in the input word sequence: in response to the word being a high-sensitivity word, obtaining the corresponding embedding vector from the high-precision partition matrix; in response to the word being a low-sensitivity word, obtaining the corresponding low-precision integer vector from the low-precision partition matrix, and obtaining the embedding vector by decompression; and concatenating the embedding vectors corresponding to each word in the input word sequence according to the word order to obtain the target embedding matrix.

[0193] The specific workflow of each of the above device embodiments can be found in the relevant descriptions in the foregoing method embodiments, and will not be repeated here.

[0194] In addition, taking the power dispatching system as an example, the above modules are independent of each other and have standardized interfaces. They can be upgraded or replaced individually according to business needs, which facilitates the connection with existing business systems and effectively reduces the risk of implementing information technology innovation transformation. At the same time, the entire chain computing and processing of the solution described in this disclosure is executed locally on the target chip. All business data only flows within the intranet and does not leave the boundary of the power company's intranet, which complies with the compliance specifications of information technology innovation and the data security management requirements of the power industry.

[0195] The solutions described in this disclosure can be applied to the field of artificial intelligence, particularly in areas such as large models, natural language processing, distributed storage, and intelligent cloud computing. Artificial intelligence is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It involves both hardware and software technologies. Artificial intelligence hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. Artificial intelligence software technologies mainly include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0196] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0197] Figure 9A schematic block diagram of an electronic device 900 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0198] like Figure 9 As shown, the electronic device 900 includes a computing unit 901, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. The RAM 903 may also store various programs and data required for the operation of the electronic device 900. The computing unit 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0199] Multiple components in electronic device 900 are connected to I / O interface 905, including: input unit 906, such as keyboard, mouse, etc.; output unit 907, such as various types of displays, speakers, etc.; storage unit 908, such as disk, optical disk, etc.; and communication unit 909, such as network card, modem, wireless transceiver, etc. Communication unit 909 allows electronic device 900 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0200] The computing unit 901 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above, such as those described in this disclosure. For example, in some embodiments, the methods described in this disclosure can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by the computing unit 901, one or more steps of the methods described in this disclosure can be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to perform the methods described herein by any other suitable means (e.g., by means of firmware).

[0201] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard parts (ASSPs), systems-on-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0202] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0203] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory (EPROM), flash memory, optical fiber, compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0204] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0205] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0206] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0207] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0208] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for generating reasoning results, comprising: Identify the target request to be processed; Following the principle of prioritizing querying the high-bandwidth memory of the target chip, and querying the low-bandwidth memory of the target chip if not all pages are hit, the target page corresponding to the target request is determined from the hot pages stored in the high-bandwidth memory and / or the cold pages stored in the low-bandwidth memory. In this process, the key-value cache generated by the inference engine is divided into key-value pages, and each key-value page is determined as the hot page or the cold page according to the access frequency. The inference engine runs on the target chip. A reasoning task is generated based on the target page, the reasoning task is sent to the reasoning engine, and the reasoning result generated by the reasoning engine is returned.

2. The method according to claim 1, wherein, The determination of the target request to be processed includes: In response to receiving any inference request, identify the request type of the inference request; In response to the request type being a real-time request, the inference request is added to the real-time queue; in response to the request type being an offline request, the inference request is added to the offline queue. In each scheduling cycle, in response to the real-time queue being non-empty, M inference requests are taken from the real-time queue and determined as the target requests, where M is a positive integer. In response to the real-time queue being empty and the offline queue being non-empty, N inference requests are taken from the offline queue and determined as the target requests, where N is a positive integer and M is less than N.

3. The method according to claim 2, further comprising one or any combination of the following: In response to the fact that the inference engine needs to process the real-time request while it is processing the offline request, a preemption event indication is sent to the inference engine to instruct the inference engine to pause the processing of the offline request and save the intermediate state of the key-value cache corresponding to the offline request. After the real-time request is processed, the processing of the offline request is resumed according to the intermediate state. In response to the target request being the real-time request and the inference engine failing to generate the inference result within the specified time, a degradation event indication is sent to the inference engine to instruct the inference engine to generate a degradation result according to a predetermined degradation strategy, wherein the degradation result is the inference result after content reduction. In response to obtaining the inference result, an audit log corresponding to the target request is generated. The audit log includes: the request type of the target request, the number of target requests, the proportion of hot pages in the target page, whether a preemption event occurs during the processing of the target request, and whether a degradation event occurs during the processing of the target request.

4. The method according to claim 1, further comprising: Obtain the L newly generated key-value pages during the process of the inference engine generating the inference result, and determine them as pages to be processed, where L is a positive integer; In response to the fact that the remaining space of the high-bandwidth memory can accommodate the page to be processed, the page to be processed is identified as the hot page and stored in the high-bandwidth memory; In response to the fact that the remaining space cannot accommodate the pages to be processed, L pages with the lowest comprehensive scores are determined from the hot pages stored in the high-bandwidth memory, the pages to be migrated are stored in the low-bandwidth memory, and the pages to be processed are determined as the hot pages and stored in the high-bandwidth memory.

5. The method according to claim 4, further comprising: In response to the presence of P target pages in the low-bandwidth memory, the P target pages are determined as the pages to be processed, where P is a positive integer and is less than or equal to the total number of target pages; In response to the fact that the remaining space of the high-bandwidth memory can accommodate the page to be processed, the page to be processed is identified as the hot page and stored in the high-bandwidth memory; In response to the fact that the remaining space cannot accommodate the pages to be processed, P pages with the lowest comprehensive scores are identified from the hot pages stored in the high-bandwidth memory, the pages to be migrated are stored in the low-bandwidth memory, and the pages to be processed are identified as the hot pages and stored in the high-bandwidth memory.

6. The method according to claim 5, wherein, The overall score for any trending page includes: Obtain the latest access frequency of the hot page determined based on the exponential decay mechanism; Obtain the difference between the most recent access time and the current time of the hot page; The comprehensive score is determined based on the access frequency and the difference.

7. The method according to claim 5, wherein, The step of storing the page to be migrated into the low-bandwidth memory includes: quantizing and compressing the page to be migrated, and then storing it into the low-bandwidth memory; The step of determining the page to be processed as the hot page and storing it in the high-bandwidth memory includes: in response to the page to be processed coming from the low-bandwidth memory, decompressing the page to be processed and storing it in the high-bandwidth memory.

8. The method according to claim 1, further comprising: Generate a quantized embedding matrix corresponding to the vocabulary of a large language model, wherein the large language model is the large language model mounted on the inference engine; The step of generating the inference task based on the target page includes: determining the target embedding matrix corresponding to the target request based on the quantized embedding matrix, and generating the inference task based on the target page and the target embedding matrix.

9. The method according to claim 8, wherein, The process of generating the quantized embedding matrix corresponding to the vocabulary includes: Obtain a vocabulary tag mapping table, which includes: each word in the vocabulary table and the vocabulary type corresponding to each word, wherein the vocabulary type includes highly sensitive words and low sensitive words; The quantization embedding matrix is ​​generated based on the vocabulary tag mapping table.

10. The method according to claim 9, wherein, The vocabulary tag mapping table includes: The sensitivity score of each word in the vocabulary is obtained respectively. Words with a sensitivity score greater than or equal to a first threshold are identified as high-sensitivity words, and words with a sensitivity score less than the first threshold are identified as low-sensitivity words. Alternatively, the words can be sorted in descending order of their sensitivity scores, and the words in the top Q positions after sorting can be identified as the highly sensitive words, while the remaining words can be identified as the low-sensitivity words, where Q is a positive integer greater than 1.

11. The method according to claim 10, wherein, The process of obtaining the sensitivity scores for each word in the vocabulary list includes: Obtain the domain corpus corresponding to the target request; Obtain the word frequency feature values ​​of each word in the domain corpus; A portion of sample data is extracted from the domain corpus, and forward reasoning is performed on each sample data using the large language model. The attention weight matrix of the last R layer attention head of the large language model is then extracted, where R is a positive integer. The attention contribution of each word is determined based on the attention weight matrix. By combining the word frequency feature value and the attention contribution, the sensitivity score of each word is determined.

12. The method according to claim 9, wherein, The step of generating the quantization embedding matrix based on the vocabulary tag mapping table includes: For each word in the vocabulary tag mapping table, the following processing is performed: in response to the word being a high-sensitivity word, the embedding vector of the word is saved in the form of a high-precision floating-point number in a high-precision partitioning matrix; in response to the word being a low-sensitivity word, the embedding vector is quantized and compressed to obtain a low-precision integer vector, and the low-precision integer vector is saved in the low-precision partitioning matrix. The quantization embedding matrix is ​​composed of the high-precision partitioning matrix and the low-precision partitioning matrix.

13. The method according to claim 12, wherein, The step of determining the target embedding matrix corresponding to the target request based on the quantization embedding matrix includes: Obtain the input word sequence corresponding to the target request; For each word in the input word sequence, the following processing is performed: in response to the word being a high-sensitivity word, the corresponding embedding vector is obtained from the high-precision partitioning matrix; in response to the word being a low-sensitivity word, the corresponding low-precision integer vector is obtained from the low-precision partitioning matrix, and the embedding vector is obtained by decompression. The embedding vectors corresponding to each word in the input word sequence are concatenated in word order to obtain the target embedding matrix.

14. A reasoning result generation device, comprising: The module consists of a request scheduling module, a hierarchical compression module, and a quantization execution module. The request scheduling module is used to determine the target request to be processed; The layered compression module is used to determine the target page corresponding to the target request from the hot pages stored in the high-bandwidth memory and / or the cold pages stored in the low-bandwidth memory, according to the principle of first querying the high-bandwidth memory of the target chip and then querying the low-bandwidth memory of the target chip if not all pages are hit. The key-value cache generated during the inference process of the inference engine is divided into key-value pages, and each key-value page is determined as the hot page or the cold page according to the access frequency. The inference engine runs on the target chip. The quantization execution module is used to generate an inference task based on the target page, send the inference task to the inference engine, and return the inference result generated by the inference engine.

15. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-13.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-13.

17. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the method of any one of claims 1-13.

Images