An ai intelligence computing platform reasoning acceleration method and system

Abstract

The application provides an AI intelligent calculation platform reasoning acceleration method and system, relates to the technical field of artificial intelligence and deep learning, and obtains a retrieval query request initiated by a large language model of an AI intelligent calculation platform and cache records of historical access data corresponding to a query paradigm of the retrieval query request in a data set to be retrieved in a historical period; a query logic tree is constructed according to the retrieval query request, and a cache hit probability of each vector data in the data set to be retrieved is obtained; according to the query complexity and the cache hit probability, the block granularity and the access step length of the vector data in the data set to be retrieved are determined; a mapping relationship between the query logic tree and a storage address is established, and a target address space distribution is obtained; according to the block granularity, a tensor layout mapping table suitable for the query logic tree is constructed; and based on the tensor layout mapping table and the access step length, reasoning acceleration is realized, and the reasoning throughput and real-time response efficiency of the large language model on the intelligent calculation platform are improved.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and deep learning technology, and in particular to an AI intelligent computing platform inference acceleration method and system. Background Technology

[0002] With the rapid development of generative artificial intelligence technology, AI computing platforms have become the core infrastructure supporting the efficient operation of large language models. In particular, when large language models perform retrieval enhancement generation tasks, matching and retrieving massive vector data from external knowledge bases has become a key factor determining the real-time response efficiency of intelligent decision support and large-scale knowledge base interaction systems.

[0003] Existing solutions typically focus on operator fusion and static computation graph optimization, and utilize general caching strategies to alleviate storage access pressure. When handling large-scale vector data retrieval tasks, mainstream technologies mostly rely on traditional linear prefetching mechanisms or fixed memory access scheduling patterns, attempting to maintain the throughput of the hardware platform through static resource pre-allocation.

[0004] However, existing solutions struggle to dynamically perceive the deep dependencies between complex retrieval logic and underlying memory access patterns in retrieval enhancement generation tasks, and cannot accurately predict the nonlinear access characteristics during the matching process of massive vector data. This leads to a severe mismatch between physical data transmission and logical processing. This memory access bottleneck significantly increases system latency and causes computational blocking when processing massive amounts of data concurrently, making it difficult for the hardware's bus bandwidth utilization and execution efficiency to meet the demands of real-time inference. Therefore, existing technologies suffer from low memory access efficiency and poor real-time response performance during large-scale vector data retrieval. Summary of the Invention

[0005] The purpose of this application is to provide an AI intelligent computing platform inference acceleration method and system to solve the technical problems of low memory access efficiency and poor real-time response performance when retrieving large-scale vector data in the prior art.

[0006] Firstly, this application provides a method for accelerating inference on an AI computing platform, including:

[0007] Get the retrieval query requests initiated by the large language model of the AI ​​computing platform, as well as the cached records of historical memory access data corresponding to the query paradigm of the retrieval query requests within the historical period in the dataset to be retrieved.

[0008] A query logic tree is constructed based on the retrieval query request, and the hierarchical position and number of branches of the logical nodes are extracted from the query logic tree to determine the query complexity of the retrieval query request. A Markov chain is used to perform state transition analysis on the cached records to obtain the cache hit probability of each vector data in the dataset to be retrieved.

[0009] Based on query complexity and cache hit probability, determine the block granularity and memory access step size of the vector data in the dataset to be retrieved;

[0010] Establish a mapping relationship between the query logic tree and the storage address, determine the initial address space distribution based on the mapping relationship, and filter the initial address space according to the cache hit probability to obtain the target address space distribution;

[0011] Based on the block granularity, the vector data corresponding to the target address space distribution is logically reorganized to construct a tensor layout mapping table that is compatible with the query logic tree;

[0012] Based on the tensor layout mapping table and memory access step size, vector data streams are extracted from the dataset to be retrieved, and the burst transmission length of a single memory access is adjusted according to the inference timing of the large language model to accelerate inference.

[0013] Optionally, a query logic tree is constructed based on the retrieval query request, and the hierarchical position and number of branches of the logical nodes are extracted from the query logic tree to determine the query complexity of the retrieval query request, including:

[0014] Identify the logical operators and search keywords in the search query request, and arrange the search keywords into a multi-level tree structure according to the execution order of the logical operators to obtain a query logic tree with multiple logical nodes;

[0015] By identifying the hierarchical position and number of branches of each logical node in the query logic tree, the hierarchical position and number of branches of each logical node are determined. Based on the number of computation nodes in the logical node, the total level of the query logic tree, and the preset operator complexity weights, the query complexity of the retrieval query request is calculated.

[0016] Optionally, a Markov chain is used to perform state transition analysis on the cached records to obtain the cache hit probability of each vector data in the dataset to be retrieved, including:

[0017] The vector data in the cache records are arranged into a historical sequence according to time sequence, and the transition probability of the Markov chain is determined based on the historical sequence.

[0018] Markov iterative mapping is performed on the last access state in the historical sequence based on the transition probability to determine the global probability distribution vector of the dataset to be retrieved.

[0019] The cache hit probability of each vector data in the dataset to be retrieved is obtained by weighted summation of the probability components corresponding to each vector data in the global probability distribution vector.

[0020] Optionally, based on query complexity and cache hit probability, the granularity of the vector data in the dataset to be retrieved and the memory access step size are determined, including:

[0021] The task load corresponding to each logical node in the query logic tree is determined based on the query complexity, and the task load is numerically transformed using the preset load mapping rules to obtain the logical block size of each vector data.

[0022] The access popularity level of each vector data is determined based on the cache hit probability of each vector data, and the access popularity level is spatially mapped using a preset popularity level table to obtain the address offset value of each vector data.

[0023] Dimension quantization is performed on each logical block size to obtain the block granularity of each vector data, and interval alignment is performed on each address offset value to obtain the memory access step size of each vector data.

[0024] Optionally, a mapping relationship between the query logic tree and the storage address is established. Based on this mapping relationship, the initial address space distribution is determined. Then, the initial address space is filtered according to the cache hit probability to obtain the target address space distribution, including:

[0025] Identify the vector identifiers corresponding to the logical nodes belonging to the computing nodes in the query logic tree, and associate the vector identifiers with the storage addresses in the dataset to be retrieved, so as to establish a mapping relationship between the query logic tree and the storage addresses;

[0026] Construct the initial address space distribution based on the storage address pointed to by each vector identifier in the mapping relationship;

[0027] The target address space distribution is obtained by extracting the storage addresses of vector data whose cache hit probability is greater than a preset filtering threshold from the initial address space distribution.

[0028] Optionally, the vector data corresponding to the target address space distribution is logically reorganized according to the block granularity to construct a tensor layout mapping table adapted to the query logic tree, including:

[0029] Based on the block granularity, the vector data within the target address space distribution is divided into regions to obtain multiple block units and the physical index of each block unit;

[0030] Based on the logical execution order of the logical nodes belonging to the computation nodes in the query logic tree, all block units are arranged to obtain the reorganized sequence;

[0031] Based on the correspondence between the physical index and the corresponding block units in the recombined sequence, a tensor layout mapping table adapted to the query logic tree is constructed.

[0032] Optionally, based on the tensor layout map and memory access step size, vector data streams are extracted from the dataset to be retrieved, and the burst transmission length of a single memory access is adjusted according to the inference timing of the large language model, including:

[0033] The extraction order of vector data is determined based on the tensor layout mapping table, and the addressing interval between different vector data in the dataset to be retrieved is determined based on the memory access step size.

[0034] Based on the extraction order and addressing interval, the dataset to be retrieved is accessed by address to obtain a vector data stream;

[0035] By identifying the execution sequence of each task node in the inference timeline of the large language model, multiple inference cycles are obtained, and the deadline of each inference cycle is determined based on the proportion of the task weight of each inference cycle in the total time of the inference timeline.

[0036] The vector data stream is segmented according to the bus width of the AI ​​computing platform to obtain multiple single memory access units. The burst transmission length of each single memory access unit is calculated according to the deadline, and the vector data stream is sent to the execution end according to the burst transmission length.

[0037] Secondly, this application provides an AI intelligent computing platform inference acceleration system, comprising:

[0038] The acquisition module is used to acquire the retrieval query requests initiated by the large language model of the AI ​​computing platform, as well as the cached records of historical memory access data corresponding to the query paradigm of the retrieval query requests within the historical period in the dataset to be retrieved.

[0039] The construction module is used to build a query logic tree based on the retrieval query request, and extract the hierarchical position and branch number of logical nodes from the query logic tree to determine the query complexity of the retrieval query request. It uses Markov chains to perform state transition analysis on cached records to obtain the cache hit probability of each vector data in the dataset to be retrieved.

[0040] The determination module is used to determine the block granularity and memory access step size of the vector data in the dataset to be retrieved based on the query complexity and cache hit probability.

[0041] The determination module is also used to establish the mapping relationship between the query logic tree and the storage address, determine the initial address space distribution based on the mapping relationship, and filter the initial address space based on the cache hit probability to obtain the target address space distribution;

[0042] The reorganization module is used to logically reorganize the vector data corresponding to the target address space distribution according to the block granularity, and construct a tensor layout mapping table that is compatible with the query logic tree.

[0043] The extraction module is used to extract vector data streams from the dataset to be retrieved based on the tensor layout mapping table and memory access step size, and adjust the burst transmission length of a single memory access according to the inference timing of the large language model to accelerate inference.

[0044] Thirdly, this application provides an electronic device, comprising:

[0045] Memory, used to store computer programs;

[0046] A processor, used to implement the steps of an AI computing platform inference acceleration method as described in the first aspect above when executing a computer program.

[0047] Fourthly, this application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the steps of the AI ​​computing platform inference acceleration method described in the first aspect above.

[0048] This application provides an AI computing platform inference acceleration method that offers complete software and hardware feature input by simultaneously collecting information from the high-level semantic layer and the low-level hardware layer. It solves the processing latency problem caused by inaccurate prediction of dynamic data flow in existing solutions when handling retrieval enhancement generation tasks. It achieves initial adaptation between logical computing requirements and storage resource allocation, laying a quantitative foundation for eliminating memory access bottlenecks. It reduces the invalid occupation of the bus on non-target data, improving the determinism of data access. It resolves the severe mismatch between physical data flow and logical processing. It eliminates computational blocking and maximizes hardware execution efficiency.

[0049] Furthermore, this application divides the vector data within the target address space into regions at the block level and obtains their physical indexes. Then, based on the logical execution order of the logical nodes belonging to the computation nodes in the query logic tree, the obtained block units are reorganized to determine the reorganization sequence. Finally, a correspondence is established between the physical index and each position in the reorganization sequence, constructing a tensor layout mapping table highly adapted to the inference computation process. This solves the storage wall bottleneck in the data transfer process and greatly optimizes the response speed of the external knowledge base retrieval stage. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1A flowchart illustrating an AI computing platform inference acceleration method provided in this application embodiment;

[0052] Figure 2 A flowchart illustrating a method for obtaining query complexity provided in an embodiment of this application;

[0053] Figure 3 A flowchart illustrating a method for constructing a tensor layout mapping table provided in an embodiment of this application;

[0054] Figure 4 This application provides a schematic diagram of the structure of an AI computing platform inference acceleration system as an embodiment of the present application.

[0055] Figure 5 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0056] When performing retrieval enhancement generation tasks on large language models, the existing memory access modes based on static pre-allocation and linear prefetching cannot perceive complex retrieval logic and are difficult to predict nonlinear access characteristics, which leads to a technical bottleneck of serious mismatch between physical transmission and logical processing.

[0057] This application constructs a query logic tree to deeply analyze the semantic logic of retrieval query requests and combines Markov chains to perform state transition analysis on cached records, overcoming the lack of awareness of dynamic execution logic in existing technologies. Based on this, it transforms high-level logical features into low-level block granularity and memory access step size, constructing a tensor layout mapping table highly adapted to the computation process to eliminate the storage wall bottleneck caused by random address access. Finally, by matching the burst transmission length at the physical layer with the inference sequence of large language models in real time, it solves the problems of memory access latency and computational blocking in the matching process of massive vector data, improving the end-to-end execution efficiency of the intelligent computing platform when performing complex retrieval tasks.

[0058] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0059] The core of this application is to provide a method for accelerating inference on an AI computing platform, and a flowchart of one specific implementation is shown below. Figure 1 As shown, the method includes:

[0060] Step 101: Obtain the retrieval query request initiated by the large language model of the AI ​​computing platform, as well as the cached records of historical access data corresponding to the query paradigm of the retrieval query request within the historical period in the dataset to be retrieved.

[0061] In this step, the AI ​​computing platform refers to the hardware infrastructure that provides high-bandwidth and low-latency computing power support for large-scale deep learning model inference. The large language model refers to a deep neural network structure with massive parameters that can understand natural language instructions and generate corresponding semantic processing logic. The retrieval query request refers to the vector search instructions generated by the large language model when performing retrieval enhancement generation tasks, used to obtain supplementary information from external knowledge bases. The query paradigm refers to a standardized query template that summarizes the logical structure of the retrieval query request into a specific category. Historical memory access data refers to the set of records of processor or computing unit access operations on the storage medium within a preset time period, representing the frequency of access, order of access, and retention status of a specific data block in the storage hierarchy over a past period. The dataset to be retrieved refers to a massive knowledge base data set stored in external storage media in vector form. Cache records refer to log information on the hit probability and access frequency of historical memory access data belonging to a specific query paradigm in the storage hierarchy within a preset historical period.

[0062] In this embodiment, the retrieval query requests issued in real time by the large language model are first intercepted through the inference interface. Next, the semantic classification of the request is parsed and matched to the corresponding query paradigm. The specific matching process is as follows: The request is extracted using a pre-defined semantic parsing engine. The operator combination features and parameter templates are compared with a pre-stored paradigm library to calculate cosine similarity, and the standardized query template with the highest similarity is selected as the target query paradigm. Simultaneously, the paradigm for obtaining and querying data is achieved by reading the hardware logs of the storage controller. Matching historical memory access data in the dataset to be retrieved The most recent cache record The retrieved search query request It possesses semantic feature vector attributes. The retrieved cached records. Represented as a set including cache hit counts Total memory access count The dynamic sequence.

[0063] For example, suppose a large language model is running on an AI computing platform. A retrieval was generated Search requests for domain-specific professional documents The query paradigm identified as belonging to the document matching category was determined. Then retrieve the dataset to be retrieved. Cache records generated within a historical period for historical memory access data corresponding to the query paradigm. The specific data obtained includes the number of successful prefetches of vector data under this paradigm and the timestamps of data being replaced from the cache.

[0064] Step 102: Construct a query logic tree based on the retrieval query request, and extract the hierarchical position and number of branches of the logical nodes from the query logic tree to determine the query complexity of the retrieval query request. Use a Markov chain to perform state transition analysis on the cached records to obtain the cache hit probability of each vector data in the dataset to be retrieved.

[0065] In this step, the query logic tree refers to a hierarchical computational model constructed based on the semantic logical relationships in the retrieval query request. A logical node is the basic unit in the query logic tree that performs a specific logical operation or retrieval intent. Query complexity is a quantitative indicator representing the computational load and resource consumption of the retrieval task. A Markov chain is a probabilistic mathematical model that describes the evolution of vector data over time in the storage hierarchy using a state transition matrix. Vector data refers to a high-dimensional set of numerical features representing knowledge base information. Cache hit probability refers to the prediction of the likelihood that specific vector data will exist in high-speed memory at a future time.

[0066] like Figure 2 As shown, Figure 2 This is a flowchart illustrating a method for obtaining query complexity, provided in an embodiment of this application.

[0067] Step 201: Identify the logical operators and search keywords in the search query request, and arrange the search keywords into a multi-level tree structure according to the execution order of the logical operators to obtain a query logic tree with multiple logical nodes.

[0068] In this step, logical operators refer to computational identifiers that guide the logical combination of search keywords. These can include logical instructions such as AND, OR, and NOT. Search keywords refer to text units with core semantic features extracted from the search query request. A multi-level tree structure refers to a topological graph constructed by using search keywords as leaf nodes and logical operators as non-leaf nodes, according to their computational priority.

[0069] In this embodiment, the received search query request is first analyzed lexically using a natural language processing tool to extract keywords. and keywords and logical operators For example, in scenarios involving retrieving professional documents based on large language models, keywords can be extracted. For tensor computation, keywords For operator fusion, logical operators And. Then, following the Boolean logic operator precedence, the keywords are... and keywords As a subordinate branch of a logical node. Logical operators... As the root node, the link relationships between parent and child nodes are established using a tree construction algorithm, resulting in a query logic tree with multiple logical nodes. .

[0070] Step 202: By identifying the hierarchical position and number of branches of each logical node in the query logic tree, determine the hierarchical position and number of branches of each logical node, and calculate the query complexity of the retrieval query request based on the number of computation nodes in the logical node, the total level of the query logic tree, and the preset operator complexity weight.

[0071] In this step, the hierarchical position refers to the vertical depth of a logical node relative to the root node of the query logic tree. The number of branches refers to the total number of lower-level nodes directly connected to a single logical node. A computation node refers to a logical node in the query logic tree responsible for performing vector similarity matching or logical judgment. The total hierarchy refers to the maximum depth of the query logic tree from the root node to the deepest leaf node. The operator complexity weight refers to the pre-defined quantization cost coefficient for different types of logical operations.

[0072] In this embodiment of the application, the constructed query logic tree is first traversed. Calculate the depth value of each node at each level. For example, if the root node is identified as being at a certain level... Keyword nodes are located in the hierarchy Next, the total number of computation nodes responsible for vector similarity matching in the tree is identified and counted. .

[0073] For example, the number of computing nodes in the current task There are 3. The total number of levels in the tree is also determined. It has two layers. Then, it retrieves the preset operator complexity weights. For example, setting weights for operations. The value is 1.5. The query complexity is calculated using the formula. : , in the formula, Indicates the first The operator complexity weights corresponding to each computation node This is a preset parallelism adjustment coefficient, representing the hardware's concurrent processing capability for nodes at different levels. This represents the total level of the query logic tree. The final quantified score is obtained by comprehensively considering both computational density and tree depth.

[0074] Step 211: Arrange the vector data in the cache records into a historical sequence according to the time sequence, and determine the transition probability of the Markov chain based on the historical sequence.

[0075] In this step, the history sequence refers to the access states of a set of vector data arranged chronologically along a timeline. The transition probability is the numerical value indicating the likelihood of transitioning from the current vector access state to the next specific vector access state.

[0076] In this embodiment, a training sample set is first obtained. The training sample set includes multiple training samples. Each training sample includes historical access sequence information and corresponding jump target information labels. For each training sample, the following steps are performed: The jump frequency between different vector data states in the historical access sequence information is counted. An initial transition probability matrix is ​​obtained. The prediction accuracy of the initial transition probability matrix is ​​verified based on the jump target information labels. The likelihood function loss value of the Markov chain model is determined. If the likelihood function loss value does not meet the training stopping condition, the parameter weights in the initial transition probability matrix are adjusted using the maximum likelihood estimation method. An updated Markov chain model is obtained. The historical access sequence information is then input into the Markov chain model. The predicted jump target is obtained. This process continues until the statistical convergence condition is met. A trained Markov chain is obtained.

[0077] Next, retrieve the cached records from the hardware logs. Extract the vector data from the past period. The access trajectories are then arranged to form a historical sequence. For example, the sequence can be represented as a vector. to vector Then to vectors Statistical state of a trained Markov chain. Transition to state The number of times. And the status. The total number of transitions to all possible states. The transition probability is calculated by dividing the total number of transitions by the total number of transitions. .

[0078] Step 212: Perform Markov iterative mapping on the last access state in the historical sequence based on the transition probability to determine the global probability distribution vector of the dataset to be retrieved.

[0079] In this step, the last access state refers to the last vector data access unit recorded in the historical sequence. The global probability distribution vector refers to the set of probabilities that each vector data in the dataset to be retrieved may be hit in the next moment.

[0080] In this embodiment of the application, the last access state in the historical sequence is first determined. For example, the last accessed element was a vector. And transform it into a unit probability vector. Next, a Markov iterative mapping is performed, which involves generating a unit probability vector representing the final visited state. With the transition probability matrix Perform iterative multiplication until the preset total number of prediction steps is reached. This allows us to deduce the probability distribution over multiple prediction steps. Specifically, assume there are four vectors in the dataset to be retrieved. The corresponding transition probability matrix... Represented as:

[0081]

[0082] In the matrix, arrive This represents the jump probability between different vectors. Calculation formula: Obtain the global probability distribution vector . In the formula, Represents the global probability distribution vector. Represents the initial vector. Represents the transition probability matrix. This indicates the number of prediction steps.

[0083] Step 213: By weighted summation of the probability components corresponding to each vector data in the global probability distribution vector, the cache hit probability of each vector data in the dataset to be retrieved is obtained.

[0084] In this step, a probability component refers to the probability value corresponding to a specific vector data in the global probability distribution vector. Weighted summation refers to a calculation method that uses a preset weight decay factor that decreases as the number of prediction steps increases to accumulate the probability components corresponding to each prediction step.

[0085] In this embodiment of the application, the global probability distribution vector is first used as an example. Extract specific vector data Each probability component For example, vectors. The probability component of being hit in the first step of the future The probability component is 0.6. The value is 0.2. Then, a preset weight decay factor that decreases with increasing prediction steps is introduced. For example, setting the weight decay factor in step 1. The weight decay factor in step 2 is 0.9. The value is 0.1. The final cache hit probability is calculated using formula (1). :

[0086] (1)

[0087] In the formula, Indicates the first The probability components predicted step by step. This represents the corresponding weight decay factor. This represents the total number of prediction steps. This method accumulates the probability components at each time step based on a preset decay factor corresponding to the number of prediction steps. The final result is the cache hit probability for each vector data.

[0088] Step 103: Determine the block granularity and memory access step size of the vector data in the dataset to be retrieved based on the query complexity and cache hit probability.

[0089] In this step, block granularity refers to the physical size of dividing vector data into independent processing units. Memory access step size refers to the physical address offset between two consecutive read operations within the memory space.

[0090] Step 301: Determine the task load corresponding to each logical node in the query logic tree based on the query complexity, and use the preset load mapping rules to perform numerical transformation on the task load to obtain the logical block size of each vector data.

[0091] In this step, task load refers to the expected hardware resource consumption allocated based on the computational density of logical nodes in the query logic tree and the dimensions of the vectors to be processed. Load mapping rules refer to the linear correspondence or lookup rules between preset task load value ranges and logical block sizes. Logical block size refers to the theoretical data partitioning size determined to meet the parallel computing efficiency requirements of specific logical nodes, without considering hardware physical boundary constraints.

[0092] In this embodiment of the application, the query complexity is first... Logical nodes are assigned to each node in the query logic tree according to their topological distribution. Then, the workload of each logical node is determined by counting the number of floating-point operations performed during vector similarity matching. Specifically, assuming that during the execution of the retrieval enhancement generation task, logical nodes... Matrix multiplication operations involving high-dimensional tensors are computationally extremely demanding, resulting in a defined workload. This is set to a high load level. Then, the task load is adjusted using preset load mapping rules. Perform numerical transformation to obtain the logical block size of each vector data. Table 1 below shows examples of preset load mapping rules:

[0093] Table 1: Preset Load Mapping Rules

[0094]

[0095] For example, when the task load... When identified as 8 GFLOPs, the logical block size of the vector data is obtained by looking up Table 1. It is 16KB.

[0096] Step 302: Determine the access popularity level of each vector data based on the cache hit probability of each vector data, and use the preset popularity level table to perform spatial mapping of the access popularity level to obtain the address offset value of each vector data.

[0097] In this step, access popularity level refers to the priority classification of vector data based on the cache hit probability. The popularity level table is a set of mapping configurations for preset cache hit probability ranges, access popularity levels, mapping space regions, and corresponding address offset value ranges. The address offset value refers to the physical displacement relative to the starting reference position of the storage, set to optimize storage layout.

[0098] In this embodiment, the cache hit probability of each vector data is first extracted. The access popularity level is determined by comparing it with a preset popularity threshold. Then, a preset popularity level table is used to spatially map the access popularity levels, obtaining the address offset value of each vector data. Table 2 shows the preset heat level table and its corresponding spatial mapping example:

[0099] Table 2: Preset Heat Levels

[0100]

[0101] For example, suppose the cache hit probability of a certain set of vector data on intelligent computing platform A is... The value is 0.9. By consulting Table 2, its access popularity level is determined to be core hot data. Subsequently, spatial mapping is performed, and its address offset value is calculated based on its dynamic distribution index within the hot data area. It is 10MB.

[0102] Step 303: Perform dimension quantization on each logical block size to obtain the block granularity of each vector data, and perform interval alignment on each address offset value to obtain the memory access step size of each vector data.

[0103] In this step, the block granularity refers to the smallest data block unit that the hardware actually performs memory accesses, determined after alignment. The memory access step size refers to the fixed-interval displacement by which the address pointer moves within the memory space during the execution of the memory access sequence.

[0104] In this embodiment of the application, the size of each logical block is first obtained. Considering that the cache line length of the intelligent computing platform hardware is fixed... Dimension quantization is performed on the size of each logical block to obtain the block granularity. The calculation formula is shown in formula (2):

[0105] (2)

[0106] In the formula, Indicates the granularity of the blocks. Indicates the logical block size. This represents the hardware cache line alignment reference value. In the formula... This indicates a rounding up operation, designed to ensure that the calculated logical block size, after dimensionality quantization, fully covers the logical data requirements. If the logical block size... 16KB and hardware aligned base value If the value is 128 bytes, then the block granularity is calculated. Still 16KB. Simultaneously, obtain the offset value for each address. The memory access step size is finally obtained by adjusting it to the minimum step boundary of bus addressing using a range alignment algorithm. For example, when the address offset value When the memory access size is 10MB, it is aligned according to the alignment requirements corresponding to the bus width to obtain the memory access step size of this set of vector data. .

[0107] Step 104: Establish the mapping relationship between the query logic tree and the storage address, determine the initial address space distribution based on the mapping relationship, and filter the initial address space according to the cache hit probability to obtain the target address space distribution.

[0108] In this step, the storage address refers to the unique identifier of the vector data in physical memory or video memory. The mapping relationship refers to the correspondence between nodes in the query logic tree and their physical storage locations. The initial address space distribution refers to the original data storage area determined based on the mapping relationship. The target address space distribution refers to the set of high-value data accesses retained after filtering.

[0109] Step 401: Identify the vector identifiers corresponding to the logical nodes belonging to the computation nodes in the query logic tree, and associate the vector identifiers with the storage addresses in the dataset to be retrieved, so as to establish a mapping relationship between the query logic tree and the storage addresses.

[0110] In this step, the vector identifier refers to the logical index number that uniquely identifies a specific high-dimensional data item in the dataset to be retrieved.

[0111] In this embodiment, the query logic tree is first accessed using a depth-first traversal algorithm. Identify the computation nodes responsible for performing cosine similarity calculations or Boolean logic checks. Extract the vector identifiers corresponding to these computation nodes. Next, the global page table index information of the storage controller is retrieved. Each vector is identified... Its starting storage address in physical memory Perform pairing. Establish a mapping relationship that reflects logical computing needs and corresponding physical storage locations.

[0112] For example, suppose a query logic tree For a specific document retrieval enhancement task, two key computational nodes were identified. Their vector identifiers were extracted as follows: and The corresponding physical storage addresses were obtained by querying the underlying index. and Examples of mapping relationships are shown in Table 3.

[0113] Table 3: Mapping Relationship Mapping Table

[0114]

[0115] Step 402: Construct the initial address space distribution based on the storage address pointed to by each vector identifier in the mapping relationship.

[0116] In this step, the initial address space distribution refers to the set of raw physical memory regions that have not been optimized for performance, which are aggregated according to the mapping relationship.

[0117] In this embodiment of the application, the storage address in the mapping table is first determined. Perform address clustering. Use an address continuity detection algorithm to identify address segments located in the same memory page or adjacent address spaces. Consolidate scattered address pointers into contiguous physical access windows. Obtain the initial address space distribution. .

[0118] For example, obtaining the address Located in memory block .address Located in memory block Then it will and The commonly defined physical region is recorded as the initial address space distribution. This process integrates fragmented address pointers, forming a raw access view to the underlying hardware.

[0119] Step 403: Extract the storage addresses of vector data with a cache hit probability greater than a preset filtering threshold from the initial address space distribution to obtain the target address space distribution.

[0120] In this step, the filtering threshold refers to the preset hit probability judgment benchmark value used to filter inefficient access data.

[0121] In this embodiment, a preset filtering threshold is first introduced. Next, retrieve the initial address space distribution. Each memory address Corresponding cache hit probability The validity of each address is determined by numerical comparison logic. If the condition is met... storage address Objects are marked as valid access objects. Otherwise, they are marked as cold data and excluded from the current access sequence. This yields the target address space distribution. .

[0122] Example. Setting a filtering threshold. It is 0.6. If an address is detected... Corresponding cache hit probability It is 0.85. Address Corresponding cache hit probability It is 0.35. Because... Greater than Reserved address The corresponding physical region. And Less than The address will be temporarily removed. The corresponding space. The final target address space distribution. Only physical regions with high hit expectations are included.

[0123] Step 105: Logically reorganize the vector data corresponding to the target address space distribution according to the block granularity, and construct a tensor layout mapping table that is compatible with the query logic tree.

[0124] In this step, the tensor layout map refers to the logical mapping relationship used to guide the execution end to access the reorganized vector data in a specific order.

[0125] like Figure 3 As shown, Figure 3 This is a flowchart illustrating a method for constructing a tensor layout mapping table, as provided in an embodiment of this application.

[0126] Step 501: Based on the block granularity, divide the vector data within the target address space distribution into regions to obtain multiple block units and the physical index of each block unit.

[0127] In this step, a block unit refers to an independent storage module obtained by dividing vector data within the target address space according to physical alignment requirements. A physical index refers to the address tag used to locate the original distribution location of the block unit in the physical storage medium.

[0128] In this embodiment of the application, the block granularity determined in the preceding steps is first obtained. Utilizing block granularity As the cutting step size for the distribution of the target address space The vector data is divided into multiple blocks by region partitioning. Specifically, the total number of blocks is calculated as follows: , in the formula This represents the total number of blocks obtained from the partitioning. Represents the target address space distribution The total length of the physical address range, This indicates the granularity of the blocks.

[0129] For example, when processing a cheat knowledge base retrieval task, the target address space distribution is obtained. The total length is 128 kilobytes and the block granularity is The total number of block units is 16 kilobytes, calculated using a formula. The value is 8. Next, each data block resulting from the partitioning is defined as a block unit. .in The value is an integer from 1 to 8. Finally, each block unit is extracted. The starting base address in physical memory serves as the physical index. .

[0130] Step 502: Arrange all block units according to the logical execution order of logical nodes belonging to the computation nodes in the query logic tree to obtain the reorganization sequence.

[0131] In this step, the logical execution order refers to the order in which tasks are executed, determined by topological sorting or depth-first traversal of the query logic tree based on the computational dependencies of the large language model.

[0132] In this embodiment of the application, the query logic tree is first extracted. The node information is marked as a computation node. The dependencies between computation nodes are resolved using a topological sorting algorithm to determine the logical execution order. Specifically, the query logic tree is identified. Computation nodes in The calculation result is the computation node The input is determined by the logic execution order, which is to process the first input. Post-processing Then, based on this order, the associated block unit of each computing node is located. And arrange them.

[0133] For example, computing nodes The corresponding block unit is located further back in physical position. And computing nodes The corresponding block unit is the one located earlier in the physical position. Arrange them according to the logical execution order as follows: Immediately afterwards A sequential view. The final recombinant sequence is obtained. .

[0134] Step 503: Based on the correspondence between the physical index and the corresponding block units in the recombined sequence, construct a tensor layout mapping table that is adapted to the query logic tree.

[0135] In this embodiment of the application, the recombinant sequence is first obtained. The logical arrangement of each block unit. Extract the corresponding physical index. Physical indexes are established through mapping modeling. With recombinant sequences The correspondence between the various block units in the sequence. For example, in the recombination sequence. The first segment in the middle row Corresponding physical index The second-ranked block unit Corresponding physical index Record this correspondence in the tensor layout mapping table. The details are shown in Table 4:

[0136] Table 4: Tensor Layout Mapping Table

[0137]

[0138] Finally, this table is used to perform memory address redirection. A tensor layout mapping table adapted to the query logic tree is then constructed. This mapping table can guide the hardware to bypass the original order of physical storage and directly perform data throughput according to the optimal logical path of tensor computation.

[0139] Step 106: Based on the tensor layout mapping table and memory access step size, extract the vector data stream from the dataset to be retrieved, and adjust the burst transmission length of a single memory access according to the inference timing of the large language model to accelerate inference.

[0140] In this step, vector data stream refers to a continuous, ordered sequence of data transmitted on the bus. Inference timing refers to the timeline arrangement of each computational stage during the generation of the large language model. Burst transmission length refers to the number of data cycles continuously transmitted on the bus under a single handshake protocol.

[0141] Step 601: Determine the extraction order of vector data according to the tensor layout mapping table, and determine the addressing interval between different vector data in the dataset to be retrieved according to the memory access step size.

[0142] In this step, the extraction order refers to the logical order in which each block unit, determined by the tensor layout mapping table, enters the inference terminal. Addressing spacing refers to the physical address jump distance required to locate adjacent vector data in the dataset to be retrieved.

[0143] In this embodiment of the application, the tensor layout mapping table is first read. Determine logical access sequence number 1 to The corresponding physical index identifier sequence. Then, based on the memory access step size obtained from the previous steps... Determine the physical span between different vector data in the dataset to be retrieved. Specifically, identify the tensor layout map. The physical index that ranks first in the middle is Then, based on the memory access step size The numerical setting of the addressing interval is 8 alignment units. In this way, the originally abstract logical mapping is transformed into specific hardware addressing step parameters, providing a precise jump basis for the underlying addressing logic.

[0144] Step 602: Based on the extraction order and addressing interval, access the address of the dataset to be retrieved to obtain a vector data stream.

[0145] In this embodiment, the addressing logic of the storage controller is first driven according to the determined extraction order. Then, non-contiguous address access is performed on the dataset to be retrieved, taking into account the addressing interval. By performing physical address jumps according to the path determined by the reassembly sequence, the vector blocks originally scattered across different physical sectors are aggregated into a vector data stream. Specifically, locating to the physical address. Extract the first block unit Then, based on the addressing interval, it jumps to the next logically related address point to extract the block unit. Ultimately, the originally discretely stored feature data is integrated into a vector data stream with computational continuity. It directly supplies tensor operation units to the inference end, eliminating the transportation delay caused by the scattered physical location of data.

[0146] Step 603: By identifying the execution sequence of each task node in the inference timeline of the large language model, multiple inference cycles are obtained, and the deadline of each inference cycle is determined according to the proportion of the task weight of each inference cycle in the total time of the inference timeline.

[0147] In this step, an inference cycle refers to an independent time slice allocated to the large language model for performing a single inference task. The deadline is the latest time point within each inference cycle at which data transfer must be completed.

[0148] In this embodiment, the execution order of each task node, such as prefetching, computation, and generation, in the inference timeline of the large language model is first identified. The overall inference process is then divided into multiple inference cycles using a time-axis segmentation algorithm. Next, the total time of the inference sequence is obtained. Based on the task weight of each inference cycle in the total time of inference sequence The proportion of each inference cycle is used to determine the deadline for each inference cycle. For example, setting the total reasoning time. The time is 50 milliseconds, used to identify the task weight of the current vector data retrieval stage within the total inference time. The proportion in this is 10%, and the deadline for this inference cycle is determined. It takes 5 milliseconds.

[0149] Step 604: Divide the vector data stream according to the bus width of the AI ​​computing platform to obtain multiple single memory access units, calculate the burst transmission length of each single memory access unit according to the deadline, and send the vector data stream to the execution end according to the burst transmission length.

[0150] In this step, bus width refers to the maximum data width that the hardware interface can transmit in parallel within a single clock cycle. A single memory access unit refers to the minimum transmission payload obtained after physically segmenting the data stream according to the bus width.

[0151] In this embodiment of the application, the bus width of the intelligent computing platform A is first obtained. Then, using the bus width... For vector data streams Physical partitioning is performed to obtain multiple single-access memory units. Then, combined with the deadline... Calculate the burst transfer length for each single memory access unit. Among them, burst transmission length The calculation rule is as follows: based on the current bus clock cycle frequency. Vector data stream Total number of bytes to be transmitted and bus width and remaining response time Calculate the number of consecutive transmission steps required to complete the transmission within the specified time. The specific calculation formula is shown in formula (3):

[0152] (3)

[0153] In the formula, Indicates the burst transmission length. Represents vector data stream Total number of bytes, Indicates the bus width. Indicates the bus operating frequency. Indicates the time from the current moment to the deadline. The remaining time. Specifically, if the remaining response time is identified. If it is shorter, the burst transmission length can be calculated using a formula. The dynamic range was increased from the standard 4-beat to 16-beat. Finally, the calculated burst transmission length was used. Drive bus controller to send vector data stream By dynamically increasing the throughput per unit time, the extracted data stream can be directly adapted to the computational frequency of large models, thereby achieving end-to-end response acceleration when performing retrieval-enhanced inference tasks.

[0154] This application provides complete software and hardware feature input by synchronously collecting information from the high-level semantic layer and the low-level hardware layer. It solves the processing latency problem caused by inaccurate prediction of dynamic data flow when handling retrieval enhancement generation tasks in existing solutions. It achieves initial adaptation between logical computation requirements and storage resource allocation, laying a quantitative foundation for eliminating memory access bottlenecks. It reduces the invalid occupation of the bus on non-target data and improves the determinism of data access. It solves the serious mismatch between physical transmission and logical processing of data flow. It eliminates computational blocking and maximizes hardware execution efficiency.

[0155] Figure 4 This is a schematic diagram illustrating a specific implementation of an AI computing platform inference acceleration system provided in this application. (Refer to...) Figure 4 The system may include:

[0156] The acquisition module 21 is used to acquire the retrieval query request initiated by the large language model of the AI ​​intelligent computing platform, as well as the cached records of the historical access data corresponding to the query paradigm of the retrieval query request in the historical period in the dataset to be retrieved.

[0157] Module 22 is used to construct a query logic tree based on the retrieval query request, and extract the hierarchical position and branch number of logical nodes from the query logic tree to determine the query complexity of the retrieval query request. Markov chains are used to perform state transition analysis on cached records to obtain the cache hit probability of each vector data in the dataset to be retrieved.

[0158] Module 23 is used to determine the block granularity and memory access step size of the vector data in the dataset to be retrieved based on the query complexity and cache hit probability.

[0159] The determination module 23 is also used to establish the mapping relationship between the query logic tree and the storage address, determine the initial address space distribution based on the mapping relationship, and filter the initial address space based on the cache hit probability to obtain the target address space distribution;

[0160] The reorganization module 24 is used to logically reorganize the vector data corresponding to the target address space distribution according to the block granularity, and construct a tensor layout mapping table that is compatible with the query logic tree.

[0161] Extraction module 25 is used to extract vector data streams from the dataset to be retrieved based on tensor layout mapping table and memory access step size, and adjust the burst transmission length of a single memory access according to the inference timing of the large language model to achieve inference acceleration.

[0162] An AI computing platform inference acceleration system according to an embodiment of this application is used to implement the aforementioned AI computing platform inference acceleration method. Therefore, the specific implementation of the AI ​​computing platform inference acceleration system can be found in the embodiment section of the AI ​​computing platform inference acceleration method above. The specific implementation can be referred to the description of the corresponding embodiments, which will not be repeated here.

[0163] Figure 5 A schematic diagram of the hardware structure of the electronic device provided in an embodiment of this application is shown.

[0164] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of an AI computing platform inference acceleration method as described above.

[0165] The electronic device may include a processor 510 and a memory 520 storing computer program instructions.

[0166] Specifically, the processor 510 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0167] Memory 520 may include mass storage for data or instructions. For example, and not limitingly, memory 520 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 520 may include removable or non-removable (or fixed) media. Where appropriate, memory 520 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 520 is non-volatile solid-state memory.

[0168] Memory may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, and electrical, optical, or other physical / tangible memory storage devices. Therefore, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to the first aspect of this disclosure.

[0169] The processor 510 reads and executes computer program instructions stored in the memory 520 to implement any of the AI ​​computing platform inference acceleration methods in the above embodiments.

[0170] In one example, the electronic device may also include a communication interface 530 and a bus 540. Wherein, such as Figure 5 As shown, the processor 510, memory 520, and communication interface 530 are connected through bus 540 and complete communication with each other.

[0171] The communication interface 530 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0172] Bus 540 includes hardware, software, or both, that couples components of an online data traffic metering device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 540 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, any suitable bus or interconnect is contemplated herein.

[0173] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the above-described AI computing platform inference acceleration methods.

[0174] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory, random access memory, portable hard drives, magnetic disks, or optical disks.

[0175] Embodiments of the present invention also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described embodiments of the AI ​​intelligent computing platform inference acceleration method.

[0176] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0177] The foregoing has provided a detailed description of the AI ​​computing platform inference acceleration method and system provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A method for accelerating inference on an AI computing platform, characterized in that, include: Obtain the retrieval query requests initiated by the large language model of the AI ​​computing platform, as well as the cached records of the query paradigms to which the retrieval query requests belong in the dataset to be retrieved within the historical period; A query logic tree is constructed based on the retrieval query request, and the hierarchical position and number of branches of the logical nodes are extracted from the query logic tree to determine the query complexity of the retrieval query request. A Markov chain is used to perform state transition analysis on the cached records to obtain the cache hit probability of each vector data in the dataset to be retrieved. Based on the query complexity and the cache hit probability, determine the block granularity and memory access step size of the vector data in the dataset to be retrieved; Establish a mapping relationship between the query logic tree and the storage address, determine the initial address space distribution based on the mapping relationship, and filter the initial address space based on the cache hit probability to obtain the target address space distribution; Based on the block granularity, the vector data corresponding to the target address space distribution is logically reorganized to construct a tensor layout mapping table that is compatible with the query logic tree; Based on the tensor layout mapping table and the memory access step size, vector data streams are extracted from the dataset to be retrieved, and the burst transmission length of a single memory access is adjusted according to the inference timing of the large language model to accelerate inference. Based on the query complexity and the cache hit probability, determine the block granularity and memory access step size of the vector data in the dataset to be retrieved, including: The task load corresponding to each logical node in the query logic tree is determined based on the query complexity, and the task load is numerically transformed using a preset load mapping rule to obtain the logical block size of each vector data. The access popularity level of each vector data is determined based on the cache hit probability of each vector data, and the access popularity level is spatially mapped using a preset popularity level table to obtain the address offset value of each vector data. Dimension quantization is performed on each logical block size to obtain the block granularity of each vector data, and interval alignment is performed on each address offset value to obtain the memory access step size of each vector data.

2. The method according to claim 1, characterized in that, A query logic tree is constructed based on the retrieval query request, and the hierarchical position and number of branches of the logical nodes are extracted from the query logic tree to determine the query complexity of the retrieval query request, including: Identify the logical operators and search keywords in the search query request, and arrange the search keywords into a multi-level tree structure according to the execution order of the logical operators to obtain a query logic tree with multiple logical nodes; By identifying the hierarchical position and number of branches of each logical node in the query logic tree, the hierarchical position and number of branches of each logical node are determined. Based on the number of computing nodes in the logical node, the total level of the query logic tree, and the preset operator complexity weight, the query complexity of the retrieval query request is calculated.

3. The method according to claim 2, characterized in that, Using Markov chains to perform state transition analysis on the cached records, the cache hit probability of each vector data in the dataset to be retrieved is obtained, including: The vector data in the cached records are arranged into a historical sequence according to time sequence, and the transition probability of the Markov chain is determined based on the historical sequence. Based on the transition probabilities, a Markov iterative mapping is performed on the last access states in the historical sequence to determine the global probability distribution vector of the dataset to be retrieved. The cache hit probability of each vector data in the dataset to be retrieved is obtained by weighted summation of the probability components corresponding to each vector data in the global probability distribution vector.

4. The method according to claim 1, characterized in that, Establish a mapping relationship between the query logic tree and the storage address, determine the initial address space distribution based on the mapping relationship, and filter the initial address space according to the cache hit probability to obtain the target address space distribution, including: Identify the vector identifiers corresponding to the logical nodes belonging to the computing nodes in the query logic tree, and associate the vector identifiers with the storage addresses in the dataset to be retrieved, so as to establish a mapping relationship between the query logic tree and the storage addresses; Based on the storage address pointed to by each vector identifier in the mapping relationship, construct the initial address space distribution; The target address space distribution is obtained by extracting the storage addresses of vector data whose cache hit probability is greater than a preset filtering threshold from the initial address space distribution.

5. The method according to claim 1, characterized in that, Based on the block granularity, the vector data corresponding to the target address space distribution is logically reorganized to construct a tensor layout mapping table adapted to the query logic tree, including: Based on the block granularity, the vector data within the target address space distribution is divided into regions to obtain multiple block units and the physical index of each block unit; Based on the logical execution order of the logical nodes belonging to the computing nodes in the query logic tree, all block units are arranged to obtain the recombined sequence; Based on the correspondence between the physical index and the corresponding block unit in the recombined sequence, a tensor layout mapping table adapted to the query logic tree is constructed.

6. The method according to claim 1, characterized in that, Based on the tensor layout mapping table and the memory access step size, vector data streams are extracted from the dataset to be retrieved, and the burst transmission length of a single memory access is adjusted according to the inference timing of the large language model, including: The extraction order of the vector data is determined according to the tensor layout mapping table, and the addressing interval between different vector data in the dataset to be retrieved is determined according to the memory access step size. According to the extraction order and the addressing interval, the dataset to be retrieved is accessed by address to obtain a vector data stream; By identifying the execution sequence of each task node in the inference timeline of the large language model, multiple inference cycles are obtained, and the deadline of each inference cycle is determined according to the proportion of the task weight of each inference cycle in the total time of the inference timeline. The vector data stream is segmented according to the bus width of the AI ​​computing platform to obtain multiple single memory access units. The burst transmission length of each single memory access unit is calculated according to the deadline, and the vector data stream is sent to the execution end according to the burst transmission length.

7. An AI intelligent computing platform inference acceleration system, characterized in that, include: The acquisition module is used to acquire the retrieval query requests initiated by the large language model of the AI ​​computing platform, as well as the cached records of the query paradigm to which the retrieval query requests belong in the dataset to be retrieved within the historical period. The construction module is used to construct a query logic tree based on the retrieval query request, and extract the hierarchical position and branch number of logical nodes from the query logic tree to determine the query complexity of the retrieval query request. The Markov chain is used to perform state transition analysis on the cached records to obtain the cache hit probability of each vector data in the dataset to be retrieved. The determination module is used to determine the block granularity and memory access step size of the vector data in the dataset to be retrieved based on the query complexity and the cache hit probability. Specifically, the determination module is used to determine the task load corresponding to each logical node in the query logic tree based on the query complexity, and to perform numerical transformation on the task load using a preset load mapping rule to obtain the logical block size of each vector data. The access popularity level of each vector data is determined based on the cache hit probability of each vector data, and the access popularity level is spatially mapped using a preset popularity level table to obtain the address offset value of each vector data. Dimension quantization is performed on each logical block size to obtain the block granularity of each vector data, and interval alignment is performed on each address offset value to obtain the memory access step size of each vector data. The determination module is also used to establish a mapping relationship between the query logic tree and the storage address, determine the initial address space distribution according to the mapping relationship, and filter the initial address space according to the cache hit probability to obtain the target address space distribution; The reorganization module is used to logically reorganize the vector data corresponding to the target address space distribution according to the block granularity, and construct a tensor layout mapping table that is compatible with the query logic tree; The extraction module is used to extract vector data streams from the dataset to be retrieved based on the tensor layout mapping table and the memory access step size, and adjust the burst transmission length of a single memory access according to the inference timing of the large language model to achieve inference acceleration.

8. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the AI ​​computing platform inference acceleration method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, enables an AI computing platform inference acceleration method as described in any one of claims 1 to 6.

Images