A token access method, device, storage medium and electronic equipment

Abstract

The application discloses a word element memory access method and device, a storage medium and an electronic equipment, and relates to the technical field of chips. The method comprises the following steps: obtaining memory access requirements of a target chip on a plurality of to-be-accessed word elements; generating total memory access configuration instructions of the plurality of to-be-accessed word elements according to the memory access requirements; analyzing the total memory access configuration instructions to obtain reading addresses corresponding to the plurality of to-be-accessed word elements respectively, and writing the plurality of reading addresses into a memory access queue in the target chip; and sequentially accessing the plurality of to-be-accessed word elements according to the writing order of the reading addresses in the memory access queue. The application generates the total memory access configuration instructions of the plurality of to-be-accessed word elements according to the memory access requirements, thereby reducing the cost of memory access resources. The reading addresses of the to-be-accessed word elements are obtained by analyzing the total memory access configuration instructions and written into the memory access queue, and the word element memory access is sequentially executed according to the writing order, thereby reducing the idle period of the pipeline, ensuring the continuity of the word element memory access, and further improving the overall inference speed and inference efficiency of the model.

Description

Technical Field

[0001] This application relates to the field of chip technology, and in particular to a word access method, apparatus, storage medium and electronic device. Background Technology

[0002] Large language model inference generates a large number of discrete token reading and writing tasks. The Direct Memory Access (DMA) module in the large language model chip can be used to handle the memory access of token-related data.

[0003] Currently, a unique configuration instruction is generated for each token and sent to the DMA module. The DMA module needs to read and parse the configuration instructions one by one in order to execute the token's memory access task.

[0004] However, using this method, the configuration instructions for the next token can only be read and parsed after the configuration instructions for the previous token have been parsed and the data access to memory has ended. During this process, there will be idle periods of waiting (pipeline bubbles), which will result in high memory access resource overhead, reduce the overall inference speed of the large language model, and thus affect the inference efficiency of the large language model. Summary of the Invention

[0005] In view of this, this application provides a token access method, apparatus, storage medium and electronic device. The main purpose is to improve the technical problem that in the prior art, the configuration instruction for the next token can only be read and parsed after the configuration instruction for the previous token has been parsed and the data access has ended. During this process, there will be idle periods (pipeline bubbles) that result in high memory access resource overhead, reduced overall inference speed of large language models and thus affect the inference efficiency of large language models.

[0006] Firstly, this application provides a lexical memory access method, including: Obtain the memory access requirements of the target chip for multiple unaccessed memory terms; Based on the memory access requirements, generate a total memory access configuration instruction for multiple of the terms to be accessed; The total memory access configuration instruction is parsed to obtain the read addresses corresponding to the multiple memory access terms, and the multiple read addresses are written into the memory access queue in the target chip. If there is free queue capacity in the memory access queue and there are unwritten remaining read addresses in the read address set, a supplementary read address with free queue capacity is selected from the remaining read addresses and written into the memory access queue. The read address set is a set of read addresses corresponding to the multiple memory access terms. According to the writing order of the read addresses in the memory access queue, the memory accesses are performed on multiple words to be accessed in sequence.

[0007] Optionally, parsing the total memory access configuration instruction to obtain read addresses corresponding to the plurality of memory access terms, and writing the plurality of read addresses into the memory access queue in the target chip, includes: The total memory access configuration instruction is parsed to obtain a set of read addresses for multiple memory access terms, and the set of read addresses includes the read addresses corresponding to each of the multiple memory access terms. Determine the memory access queue in the target chip, select the read address with the target queue capacity from the read address set, and write the selected read address into the memory access queue.

[0008] Optionally, when there is free queue capacity in the memory access queue and there are unwritten remaining read addresses in the read address set, selecting a supplementary read address with free queue capacity from the remaining read addresses and writing the supplementary read address into the memory access queue to achieve iterative writing of the read address includes: According to the clock cycle of the target chip, the free queue capacity and address write status of the memory access queue are monitored. If there is free queue capacity in the memory access queue and there are unwritten remaining read addresses in the read address set, a supplementary read address with free queue capacity is selected from the remaining read addresses. The supplementary read address is written into the memory access queue to achieve iterative writing of the read address.

[0009] Optionally, the step of sequentially accessing multiple terms to be accessed according to the write order of the read addresses in the memory access queue includes: The read addresses in the memory access queue are read according to the clock cycle of the target chip and the write order of the read addresses; Based on the read address, the locations of multiple metadata of the terms to be accessed are determined; The total memory access configuration instruction is parsed to obtain the memory access operation parameters corresponding to the multiple memory access terms to be accessed. Based on the memory access operation parameters, the metadata of the multiple terms to be accessed is accessed.

[0010] Optionally, generating a total memory access configuration instruction for multiple terms to be accessed based on the memory access requirements includes: The memory access request is parsed to obtain the memory access parameters corresponding to the multiple memory access terms to be accessed; Based on the memory access parameters, a total memory access configuration instruction is generated for multiple terms to be accessed, so that multiple memory access hardware threads in the target chip can access the multiple terms to be accessed according to the total memory access configuration instruction.

[0011] Optionally, the method further includes: If the number of the stored words to be visited meets the number threshold condition, determine the general-purpose registers in the target chip that are in an idle state; Based on the memory access operation parameters of the term to be accessed, the general-purpose register is configured as the read address storage area of ​​the term to be accessed, and the multiple read addresses obtained by parsing are stored in the general-purpose register; Multiple read addresses are read from the general-purpose register and written into the memory access queue in the target chip.

[0012] Secondly, this application provides a lexical memory access device, comprising: The acquisition module is configured to acquire the memory access requirements of the target chip for multiple memory access terms; The generation module is configured to generate a total memory access configuration instruction for multiple terms to be accessed based on the memory access requirements. The parsing module is configured to parse the total memory access configuration instruction to obtain the read addresses corresponding to the multiple memory access terms, and write the multiple read addresses into the memory access queue in the target chip. If there is free queue capacity in the memory access queue and there are unwritten remaining read addresses in the read address set, a supplementary read address with free queue capacity is selected from the remaining read addresses, and the supplementary read address is written into the memory access queue. The read address set is a set composed of the read addresses corresponding to the multiple memory access terms. The memory access module is configured to access multiple terms to be accessed sequentially according to the writing order of the read addresses in the memory access queue.

[0013] Thirdly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the lexical memory access method described in the first aspect.

[0014] Fourthly, this application provides an electronic device, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor executes the computer program to implement the lexical memory access method described in the first aspect.

[0015] By employing the above technical solution, this application provides a word access memory method, apparatus, storage medium, and electronic device, which obtains the memory access requirements of a target chip for multiple words to be accessed; generates a total memory access configuration instruction for multiple words to be accessed based on the memory access requirements; parses the total memory access configuration instruction to obtain the read addresses corresponding to the multiple words to be accessed, and writes the multiple read addresses into the memory access queue in the target chip; when there is free queue capacity in the memory access queue and there are unwritten remaining read addresses in the set of read addresses, selects a supplementary read address with free queue capacity from the remaining read addresses, and writes the supplementary read address into the memory access queue; and accesses the multiple words to be accessed sequentially according to the writing order of the read addresses in the memory access queue. Compared with existing technologies, this application reduces the overhead of memory access resources by generating a total memory access configuration instruction for multiple words to be accessed based on memory access requirements; by parsing the total memory access configuration instruction to obtain the read address of each word to be accessed and writing it into the memory access queue, and executing word access sequentially according to the order of the read address in the memory access queue, the pipeline idle cycle can be reduced, word access continuity can be guaranteed, and thus the overall inference speed and inference efficiency of the model can be improved. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating a lexical memory access method provided in an embodiment of this application is shown; Figure 2 A flowchart illustrating a lexical memory access method provided in an embodiment of this application is shown; Figure 3 This illustration shows a schematic diagram of a lexical memory access device provided in an embodiment of this application; Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0019] The embodiments of this application will now be described in more detail with reference to the accompanying drawings. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0020] To address the technical problem in current technologies where the configuration instructions for the next token cannot be read and parsed until the previous token's configuration instructions have been parsed and data access is complete, resulting in idle periods (pipeline bubbles) and high memory access resource overhead, thus reducing the overall inference speed of large language models and impacting their inference efficiency, this embodiment provides a lexical memory access method, such as... Figure 1 As shown, the method includes: Step 101: Obtain the memory access requirements of the target chip for multiple memory terms to be accessed.

[0021] In this embodiment, the target chip can be an AI acceleration chip designed for large-scale sparse computation scenarios, capable of handling inference and training computation tasks for large Transformer-type models such as Mixture of Experts (MoE) and sparse attention. For example, the target chip in this embodiment may specifically incorporate memory access acceleration hardware, which may include at least one of the following: multiple hardware threads, an address calculation unit, a prefetch buffer, a register multiplexing module, and an exception handling module.

[0022] Optionally, the memory access acceleration hardware can be a DMA hardware engine. This hardware can receive a single total memory access configuration command issued by the software, batch parse and generate read addresses for all words to be accessed, complete address iterative prefetching through the memory access queue, and rely on multiple independent memory access hardware threads to execute various sparse word memory access operations in parallel, including data aggregation read operations (Gather), data scattering write operations (Scatter), expert routing distribution operations (Dispatch), and remote memory write operations (RemoteWrite). It also supports dynamic reuse of storage address information in general-purpose registers in scenarios with a small number of words and monitors hardware anomalies throughout the entire memory access process in real time. This enables zero-bubble pipeline and low configuration overhead sparse word data transfer, solving the problems of high configuration overhead, pipeline bubbles, low register utilization, and high memory access latency caused by traditional DMA descriptor issuance per word.

[0023] In the embodiments of this application, the tokens to be accessed can be large model inputs or token vector data generated during intermediate computation. These tokens can possess locally continuous and globally discrete storage characteristics. Discrete storage characteristics can trigger sparse memory access operations such as Gather, Scatter, Dispatch, and Combine. For example, in the embodiments of this application, the tokens to be accessed can specifically be input tokens that need to be aggregated and read under the sparse attention mechanism, tokens distributed by MoE expert routing, or aggregated output tokens of expert computation results.

[0024] In this embodiment, the memory access request can be a service request sent from the upper-layer software to the target chip, and can include all original information related to the memory access behavior corresponding to the terms to be accessed. For example, the memory access request in this embodiment can specifically include the total number of terms to be accessed, the word metadata bit width, the memory access operation type, the original memory storage base address, and sparse index offset information.

[0025] In the embodiments of this application, this step can be executed by the upper-layer software scheduling module of the target chip. The software scheduling module monitors the model calculation pipeline in real time. When it detects that a sparse word read / write operation needs to be performed, it collects the parameter information corresponding to all the words to be accessed in the current batch, integrates them to generate a complete memory access requirement, and sends it to the instruction generation unit inside the chip to complete the acquisition of the memory access requirement.

[0026] Step 102: Based on the memory access requirements, generate a total memory access configuration instruction for multiple memory access terms to be accessed.

[0027] In this embodiment, the total memory access configuration instruction can be a single global configuration instruction, which can be used to carry the complete address configuration data stream for all words to be accessed in the entire batch, unlike the traditional one-to-one word descriptor instruction of DMA. For example, the total memory access configuration instruction in this embodiment can specifically integrate all memory access configuration information such as the memory access address, word metadata length, memory access operation type, and total number of words to be accessed, and a single issuance can cover all word memory access tasks in the current batch.

[0028] In the embodiments of this application, all parameters within the memory access requirement can be uniformly integrated and compiled, and only one total memory access configuration instruction is output and sent to the hardware thread context. This can reduce the memory access configuration overhead from the traditional linear complexity of O(n) to the constant complexity of O(1), where n can represent the total number of words to be accessed in a single batch.

[0029] In this embodiment, the memory access hardware thread can be the hardware execution entity of the target chip's memory access acceleration hardware. The memory access hardware thread can independently complete the entire process of address resolution, address prefetching, and data transmission. Different memory access hardware threads run completely independently and do not block each other. For example, the memory access hardware thread in this embodiment may specifically include a thread executing the Dispatch task and a thread executing the Gather task. A single hardware engine can also switch between different memory access operations such as Gather, Scatter, and RemoteWrite.

[0030] Step 103: Parse the total memory access configuration instruction to obtain the read addresses corresponding to multiple memory access terms, and write the multiple read addresses into the memory access queue in the target chip. If there is free queue capacity in the memory access queue and there are unwritten remaining read addresses in the read address set, select a supplementary read address with free queue capacity from the remaining read addresses, and write the supplementary read address into the memory access queue to realize the iterative writing of the read addresses.

[0031] The set of read addresses is a set of read addresses corresponding to multiple stored words to be visited.

[0032] In this embodiment, the read address can be the source starting address of the token to be accessed stored in off-chip main memory or on-chip cache, or it can be a basic component of the data stream configured for a software-generated token address. For example, in this embodiment, the read address can be specifically calculated and generated cycle-by-cycle by the address calculation unit according to a single chip clock cycle, with a set of read addresses output per cycle for initiating transmission.

[0033] In this embodiment, the memory access queue can be an internal First-In-First-Out (FIFO) buffer queue, which can be used to temporarily cache batch read addresses, decoupling address prefetching and token data transmission, continuously filling the buffer to ensure continuous memory access, and eliminating the bubble problem in traditional DMA pipelines. For example, the memory access queue in this embodiment can have a fixed storage depth, which can correspond to the target queue capacity, and the queue follows a first-in-first-out data read / write rule.

[0034] In this embodiment, after receiving the total memory access configuration instruction, the address calculation unit inside the chip parses all the token address information integrated in the instruction one by one, extracts the independent read address corresponding to each memory access term, integrates them to form a complete address set, and writes the read addresses in batches according to the maximum storage capacity of the memory access queue to complete the address pre-storage.

[0035] Step 104: According to the writing order of the read addresses in the memory access queue, perform memory access on multiple words to be accessed in sequence.

[0036] In this embodiment, the clock cycle can be the basic timing unit for the synchronous operation of the target chip hardware. It can serve as a unified scheduling rhythm for address replenishment, address reading, and data transmission. All hardware address read and write operations can be executed according to a per-cycle mechanism. For example, in this embodiment, each clock cycle can complete at least one of the following operations: one queue address read, one address replenishment write, and one token data transmission.

[0037] In the embodiments of this application, memory access operation parameters can be obtained by parsing the overall memory access configuration instructions, and can limit the read and write behavior rules corresponding to a single set of tokens. For example, the memory access operation parameters in the embodiments of this application may specifically include the word token read / write byte length, memory access read / write type, data transmission target storage area, transmission bandwidth threshold, etc.

[0038] In this embodiment, the memory access hardware thread follows the first-in-first-out write order of the memory access queue. Each clock cycle, it retrieves a read address from the head of the queue, locates the corresponding memory access term metadata in memory based on the read address, and initiates the corresponding read and write transmission in conjunction with the parsed memory access operation parameters. With the mechanism of continuously replenishing addresses in the memory access queue, the pipeline achieves zero bubbles.

[0039] Optionally, existing technology generates an independent configuration instruction for each token and sends it to the DMA module. The DMA module constructs the configuration instructions in the form of a linked list and executes them serially. Specifically, after the DMA module retrieves a configuration instruction from the instruction linked list and completes its parsing, it can automatically obtain the address of the next configuration instruction from the instruction linked list. While the hardware is executing the data transfer of the currently parsed instruction, it can prefetch the configuration of the next task based on the address of the next configuration instruction, thereby completing the discrete transfer tasks of multiple tokens. However, when the data volume of each token is small, the sum of the memory read latency and instruction parsing latency of a single configuration instruction will be significantly greater than the data transfer clock cycle of a single small token. Using this method (i.e., the hardware prefetching of configuration instructions), it is impossible to complete the execution before the current token transfer is completed. After the data transfer is completed, the hardware can only wait for the next configuration instruction to return, which cannot form an effective transfer overlap window and the pipeline bubble cannot be eliminated. This defect is widely present in small token computing scenarios such as sparse attention small-granularity key-value cache (KV) blocks and MoE lightweight expert input.

[0040] Optionally, the embodiments of this application can replace the traditional descriptor prefetching mechanism. The embodiments of this application generate a total memory access configuration instruction for multiple to-be-accessed terms based on memory access requirements, and implement direct sequential reading by configuring data streams with consecutive addresses, which greatly reduces the overhead of a single address acquisition. The read addresses of each to-be-accessed term are obtained by parsing the total memory access configuration instruction and written into the memory access queue. Multiple sets of token read addresses are pre-stored in batches before execution. Term memory access is executed sequentially according to the order in which the read addresses in the memory access queue are written. During transmission, empty buffer slots are automatically filled with new addresses, ensuring that the corresponding address is already cached and ready when any token is moved, without additional waiting. The embodiments of this application are not constrained by the granularity of token data and can achieve zero-bubble pipeline execution in various small-granularity term memory access scenarios such as sparse attention and MoE.

[0041] Compared with existing technologies, this application reduces the overhead of memory access resources by generating a total memory access configuration instruction for multiple words to be accessed based on memory access requirements; by parsing the total memory access configuration instruction to obtain the read address of each word to be accessed and writing it into the memory access queue, and executing word access sequentially according to the order of the read address in the memory access queue, the pipeline idle cycle can be reduced, word access continuity can be guaranteed, and thus the overall inference speed and inference efficiency of the model can be improved.

[0042] As an optional approach, when executing the command to "parse the total memory access configuration instruction to obtain the read addresses corresponding to multiple memory access terms and write the multiple read addresses into the memory access queue in the target chip", the following methods can be used, but are not limited to these: Figure 2 As shown, it includes: Step 201: Parse the total memory access configuration instruction to obtain a set of read addresses for multiple memory access terms. The set of read addresses includes the read addresses corresponding to each of the multiple memory access terms.

[0043] In this embodiment, the read address set can be an ordered set of read addresses for all memory-accessible tokens in the current batch, which can be stored in a temporary cache inside the chip as a data source for filling the memory access queue addresses. For example, the read address set in this embodiment can be specifically arranged in an ordered manner according to the token routing order and the sparse index order.

[0044] In the embodiments of this application, the address calculation unit completely parses all token address configuration data streams integrated within the total memory access configuration instruction, extracts the independent read address corresponding to each to-be-accessed memory token, summarizes and integrates all read addresses, and generates a complete set of read addresses, providing a complete address data source for subsequent batch writing to the memory access queue.

[0045] Step 202: Determine the memory access queue in the target chip, select a read address with the target queue capacity from the read address set, and write the selected read address into the memory access queue.

[0046] In this embodiment, the target queue capacity can be the storage depth of the memory access queue, representing the maximum number of read addresses that the memory access queue can store in a single operation. For example, the target queue capacity in this embodiment can be configured to different values ​​such as 8, 16, or 32 based on the chip hardware design, and can be dynamically switched according to the model load.

[0047] In the embodiments of this application, the currently idle memory access queue can be identified, the target queue capacity corresponding to the queue can be read, and read addresses matching the number of target queue capacities can be extracted from the starting position of the read address set and written in batches to the memory access queue to complete address pre-filling.

[0048] As an optional approach, when performing the action of "selecting a supplementary read address from the remaining read addresses with available queue capacity and writing the supplementary read address into the memory access queue when there is available queue capacity in the memory access queue and there are unwritten remaining read addresses in the read address set", the following method can be used, but is not limited to: monitoring the available queue capacity and address write status of the memory access queue according to the clock cycle of the target chip; selecting a supplementary read address from the remaining read addresses when there is available queue capacity in the memory access queue and there are unwritten remaining read addresses in the read address set; and writing the supplementary read address into the memory access queue to achieve iterative writing of read addresses.

[0049] In this embodiment, the free queue capacity can be the number of unoccupied storage slots in the memory access queue. The free queue capacity can dynamically change in real time as the address reads within the queue consume memory. For example, in this embodiment, the free queue capacity increments by one for each address read by the memory access queue. A free queue capacity greater than 0 indicates that there is writable space in the queue.

[0050] In the embodiments of this application, the chip hardware timing monitoring module continuously collects the occupancy status of the memory access queue and the number of remaining addresses in the read address set on a clock cycle basis. When the conditions of having empty slots in the queue and having unwritten addresses in the set are met simultaneously, the module automatically extracts supplementary read addresses that match the empty capacity and writes them to the tail of the queue. The module iteratively performs the address supplementation operation until all addresses in the read address set are written to the queue, thereby decoupling address prefetching from data transmission and ensuring that the memory access hardware can continuously and uninterruptedly initiate word transmission.

[0051] As an optional approach, when executing "accessing multiple memory-to-be-accessed terms sequentially according to the write order of the read addresses in the memory access queue", the following method can be used, but is not limited to: reading the read addresses in the memory access queue according to the clock cycle and the write order of the read addresses; locating the positions of multiple memory-to-be-accessed terms based on the read addresses; parsing the total memory access configuration instruction to obtain the memory access operation parameters corresponding to each of the multiple memory-to-be-accessed terms; and accessing the located memory-to-be-accessed terms based on the memory access operation parameters.

[0052] In this embodiment, the writing order can be the order in which read addresses are written to the memory access queue in batches, corresponding to the first-in-first-out (FIFO) hardware read / write rule of the memory access queue. For example, in this embodiment, read addresses written to the queue first will be read by the hardware thread and have their memory access operations performed first.

[0053] In the embodiments of this application, the memory access hardware thread reads a valid read address from the head of the memory access queue every clock cycle, calculates the physical storage location of the metadata to be accessed in memory based on the read address, and at the same time, it parses the total memory access configuration instruction to extract the memory access operation parameters of the corresponding batch, and performs Gather, Scatter or RemoteWrite memory access transmission on the located metadata according to the transmission length and read / write type defined by the parameters, and continues to loop until all addresses in the queue have been processed.

[0054] As an optional approach, when executing "generating a total memory access configuration instruction for multiple memory access terms based on memory access requirements", the following method can be used, but is not limited to: parsing the memory access requirements to obtain memory access parameters corresponding to the multiple memory access terms respectively; generating a total memory access configuration instruction for the multiple memory access terms based on the memory access parameters, so that the multiple memory access hardware threads in the target chip can access the memory of the multiple memory access terms according to the total memory access configuration instruction.

[0055] In this embodiment, the memory access parameters can be configuration information for a single token dimension extracted from the memory access requirement, and can be used to assemble complete global configuration instructions. For example, the memory access parameters in this embodiment may specifically include word metadata length, single token offset address, memory access operation type, and data transfer target cache address.

[0056] In the embodiments of this application, the instruction generation unit can decompose the memory access requirements, extract the memory access parameters corresponding to all the memory access terms to be accessed, integrate all the memory access parameters to generate a single total memory access configuration instruction, and send it to the multi-parallel memory access hardware threads in the chip. The multiple hardware threads can independently load the same total memory access configuration instruction and execute different sparse memory access tasks in parallel. The running states of the threads do not interfere with each other, thereby improving the concurrent processing capability of the chip hardware.

[0057] As an alternative approach, but not limited to the following method, the method includes: determining the general-purpose registers in the target chip that are in an idle state when the number of memory access words meets the quantity threshold condition; configuring the general-purpose registers as the read address storage area of ​​the memory access words according to the memory access operation parameters of the memory access words, and storing the multiple read addresses obtained by parsing in the general-purpose registers; reading the multiple read addresses from the general-purpose registers and writing the multiple read addresses into the memory access queue in the target chip.

[0058] In this embodiment, the quantity threshold condition can be a pre-set criterion for determining a small number of terms, which can be used to trigger register reuse optimization logic. For example, the quantity threshold condition in this embodiment can be specifically configured as the number of terms to be accessed being less than or equal to 16, and the threshold can be flexibly adjusted according to the amount of chip register resources.

[0059] In the embodiments of this application, the general-purpose register can be a dynamically reusable hardware register group inside the chip. Under normal conditions, it can be used to store intermediate calculation parameters, and when idle, it can be dynamically reinterpreted into a read address storage area according to the semantics of memory access operations. For example, in the embodiments of this application, when the number of tokens meets the threshold condition, the hardware does not need to access the off-chip memory to read the address information, but directly stores the parsed read address into the idle general-purpose register. The software can directly write the address configuration information to the register, saving memory access overhead and significantly reducing the end-to-end latency in sparse memory access scenarios with a small number of tokens.

[0060] In this embodiment, the register multiplexing module counts the total number of tokens to be accessed in the current batch in real time. When the number of tokens meets the preset threshold, it traverses the internal register resources of the chip to filter idle general-purpose registers that are not occupied by any task. It redefines the storage purpose of the registers in combination with the memory access operation parameters and stores all the parsed read addresses into the register group. Subsequently, the hardware thread directly reads all read addresses from the general-purpose registers and writes them in batches to the memory access queue to complete the subsequent pipelined memory access process. This can improve the overall utilization of the chip registers and solve the defects of fixed function and resource waste of traditional DMA registers.

[0061] Optionally, an exception handling module may also be included to detect various hardware exceptions in real time during the entire process of address parsing, address writing, and token memory access. Exception types may include memory access queue write overflow exceptions, total memory access configuration instruction parsing errors, and memory data transfer failures. For example, in this embodiment, after the hardware detects any exception, it can immediately pause all current memory access hardware threads, record the exception type and location, and report the exception information to the upper-layer software. The software can then choose to resume the current memory access task or directly terminate the current sparse memory access process based on the exception type, thereby improving the hardware reliability in sparse token memory access scenarios.

[0062] Optionally, this application embodiment also provides a memory access acceleration example, including the following steps: Step 1: Receive a single global memory access configuration instruction; the upper-layer software of the target chip generates only one global memory access configuration instruction for all memory access terms in a whole batch and sends it to the memory access module.

[0063] Step 2: The hardware register file sends the configuration instructions to the instruction parser to break down the multiple parallel tasks; the hardware resource scheduler allocates the multiple task instructions to multiple independent Thread Context threads (i.e., the memory access hardware threads in this embodiment).

[0064] Step 3: Each Thread Context calculates the token access address (i.e., the read address in this embodiment) cycle by cycle, while the iteration control unit monitors the iteration progress in real time, and the notification register reuses the scheduler to continuously prefetch the next batch of token addresses to supply the threads.

[0065] Step 4: Each thread outputs the calculated address to the memory access interface, and reads and writes the local L3 cache (i.e., Local L3) or remote memory (i.e., Remote Memory) according to the operation type to complete the sparse token data transfer.

[0066] Step 5: If any thread encounters an access error, it is reported separately to the exception handling unit. Only the faulty thread is suspended, while the other channels continue to operate normally.

[0067] Step 6: After the current batch of tokens has been processed, the iteration control unit continuously triggers address prefetching, and the pipeline is seamlessly connected until all tokens have been iterated.

[0068] Compared with existing technologies, the embodiments of this application generate a set of read addresses by parsing the total memory access configuration instruction and writing them into the memory access queue in batches according to the target queue capacity, thereby achieving batch pre-storage of read addresses; by monitoring the free capacity of the queue according to the clock cycle and iteratively supplementing the read addresses, the supply of effective addresses in the queue is continuously maintained, reducing memory access pauses; by parsing the memory access requirements to obtain memory access parameters and generating a total memory access configuration instruction for parallel processing by multiple memory access hardware threads, the concurrent processing capability of memory access can be improved; by reusing idle general-purpose registers to store read addresses when the number of tokens reaches the target, the additional memory access required for address reading is reduced, and the memory access latency in scenarios with a small number of tokens is reduced.

[0069] Furthermore, as Figure 1 and Figure 2 The specific implementation of the method shown in this embodiment provides a lexical memory access device, such as... Figure 3 As shown, the device includes: an acquisition module 31, a generation module 32, a parsing module 33, and a memory access module 34.

[0070] The acquisition module 31 is configured to acquire the memory access requirements of the target chip for multiple memory access terms; The generation module 32 is configured to generate a total memory access configuration instruction for multiple memory access terms based on memory access requirements; The parsing module 33 is configured to parse the total memory access configuration instruction to obtain the read addresses corresponding to multiple memory access terms, and write the multiple read addresses into the memory access queue in the target chip. If there is free queue capacity in the memory access queue and there are unwritten remaining read addresses in the read address set, a supplementary read address with free queue capacity is selected from the remaining read addresses, and the supplementary read address is written into the memory access queue. The read address set is a set composed of the read addresses corresponding to the multiple memory access terms. The memory access module 34 is configured to access multiple words to be accessed sequentially according to the writing order of the read addresses in the memory access queue.

[0071] In some examples of this embodiment, the parsing module 33 is specifically configured to parse the total memory access configuration instruction to obtain a set of read addresses for multiple memory access terms, the set of read addresses including the read addresses corresponding to the multiple memory access terms respectively; determine the memory access queue in the target chip; select a read address of the target queue capacity from the set of read addresses; and write the selected read address into the memory access queue.

[0072] In some examples of this embodiment, the parsing module 33 is further configured to monitor the free queue capacity and address write status of the memory access queue according to the clock cycle of the target chip; when there is free queue capacity in the memory access queue and there are unwritten remaining read addresses in the read address set, a supplementary read address with free queue capacity is selected from the remaining read addresses; the supplementary read address is written in the memory access queue to realize iterative writing of read addresses.

[0073] In some examples of this embodiment, the memory access module 34 is specifically configured to read the read address in the memory access queue according to the clock cycle and the write order of the read address; locate the position of multiple memory metadata to be accessed according to the read address; parse the total memory access configuration instruction to obtain the memory access operation parameters corresponding to the multiple memory metadata to be accessed; and access the memory of the located multiple memory metadata according to the memory access operation parameters.

[0074] In some examples of this embodiment, the generation module 32 is specifically configured to parse the memory access requirements to obtain memory access parameters corresponding to multiple memory access terms; and generate a total memory access configuration instruction for multiple memory access terms according to the memory access parameters, so that multiple memory access hardware threads in the target chip can access the memory of multiple memory access terms according to the total memory access configuration instruction.

[0075] In some examples of this embodiment, the memory access module 34 is further configured to: determine the general-purpose registers in the target chip that are in an idle state when the number of terms to be accessed meets the quantity threshold condition; configure the general-purpose registers as the read address storage area of ​​the terms to be accessed according to the memory access operation parameters of the terms to be accessed, and store the multiple read addresses obtained by parsing in the general-purpose registers; read the multiple read addresses from the general-purpose registers and write the multiple read addresses into the memory access queue in the target chip.

[0076] It should be noted that other corresponding descriptions of the functional units involved in the lexical memory access device provided in this embodiment can be found in [reference]. Figure 1 and Figure 2 The corresponding descriptions in [the document] will not be repeated here.

[0077] Based on the above, Figure 1 and Figure 2 Accordingly, this embodiment also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method. Figure 1 and Figure 2 The method shown.

[0078] Based on this understanding, the technical solution of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as CD-ROM, USB flash drive, mobile hard drive, etc.) and includes several instructions to cause a computer device (such as personal computer, server, or network device, etc.) to execute the methods of various implementation scenarios of this application.

[0079] like Figure 4 The diagram shown is a hardware structure schematic of an electronic device according to the present invention, comprising: At least one processor 401; and, A memory 402 connected to the processing of at least one processor 401 terminal device; wherein, The memory 402 stores instructions that can be executed by at least one processor, such that the at least one processor can perform the aforementioned lexical memory access method.

[0080] Figure 4 Take a processor 401 as an example.

[0081] The electronic device may also include an input device 403 and an output device 404.

[0082] The processor 401, memory 402, input device 403, and output device 404 can be connected via a bus or other means. Figure 4 Taking the example of a connection between China and Israel via a bus.

[0083] Memory 402, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the lexical memory access method in the embodiments of this application, for example, Figure 1 and Figure 2 The method flow is shown. The processor 401 executes various functional applications and terminal device processing by running non-volatile software programs, instructions, and modules stored in the memory 402, that is, it implements the lexical memory access method in the above embodiments.

[0084] Memory 402 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created according to the use of the lexical memory access method, etc. Furthermore, memory 402 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 402 may optionally include memory remotely located relative to processor 401, and these remote memories may be connected via a network to the apparatus performing the lexical memory access method. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, processing networks of mobile terminal devices, and combinations thereof.

[0085] Input device 403 can receive user clicks and generate signal inputs related to user settings and function control of the token access method. Output device 404 may include display devices such as a display screen.

[0086] One or more modules are stored in memory 402, and when run by one or more processors 401, the lexical memory access method in any of the above method embodiments is executed.

[0087] Optionally, the aforementioned physical devices may also include a user interface, a network interface, a camera, radio frequency (RF) circuitry, sensors, audio circuitry, a Wi-Fi module, etc. The user interface may include a display screen, input units such as a keyboard, etc., and optional user interfaces may also include USB interfaces, card reader interfaces, etc. The network interface may optionally include standard wired interfaces, wireless interfaces (such as Wi-Fi interfaces), etc.

[0088] Those skilled in the art will understand that the physical device structure provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or combine certain components, or have different component arrangements.

[0089] The storage medium may also include an operating system and a processing module for network terminal devices. The operating system is a program that manages the hardware and software resources of the aforementioned physical devices, supporting the operation of information processing programs and other software and / or programs. The processing module for network terminal devices is used to enable terminal device processing between the various components within the storage medium, as well as terminal device processing with other hardware and software within the information processing physical device.

[0090] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platform, or it can be implemented by hardware. Compared with existing technologies, this embodiment reduces memory access resource overhead by generating a total memory access configuration instruction for multiple terms to be accessed based on memory access requirements; it obtains the read addresses of each term to be accessed by parsing the total memory access configuration instruction and writes them into the memory access queue, and executes term access sequentially according to the order of the read addresses written in the memory access queue, which can reduce pipeline idle cycles, ensure the continuity of term access, and thus improve the overall inference speed and efficiency of the model; it generates a set of read addresses by parsing the total memory access configuration instruction and writes them into the memory access queue in batches according to the target queue capacity, realizing batch pre-storage of read addresses; it continuously maintains the supply of effective addresses in the queue by monitoring the free capacity of the queue according to the clock cycle and iteratively supplementing the read addresses, reducing memory access pauses; it obtains memory access parameters by parsing the memory access requirements and generates a total memory access configuration instruction for parallel processing by multiple memory access hardware threads, which can improve the concurrent processing capability of memory access; and it reuses idle general-purpose registers to store read addresses when the number of terms reaches the target, reducing the additional memory access required for address reading and reducing memory access latency in scenarios with a small number of terms.

[0091] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0092] The above are merely specific embodiments of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to these embodiments, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A word element access method, characterized in that, include: Obtain the memory access requirements of the target chip for multiple unaccessed memory terms; Based on the memory access requirements, generate a total memory access configuration instruction for multiple terms to be accessed; The total memory access configuration instruction is parsed to obtain the read addresses corresponding to the multiple memory access terms, and the multiple read addresses are written into the memory access queue in the target chip. The memory access queue is a first-in-first-out buffer queue with a fixed storage depth. When there is free queue capacity in the memory access queue and there are unwritten remaining read addresses in the read address set, a supplementary read address with free queue capacity is selected from the remaining read addresses and written into the memory access queue. The read address set is a set of read addresses corresponding to the multiple memory access terms. According to the writing order of the read addresses in the memory access queue, the multiple words to be accessed are accessed sequentially; When there is free queue capacity in the memory access queue and there are unwritten remaining read addresses in the read address set, selecting a supplementary read address with free queue capacity from the remaining read addresses and writing the supplementary read address into the memory access queue includes: According to the clock cycle of the target chip, the free queue capacity and address write status of the memory access queue are monitored. If there is free queue capacity in the memory access queue and there are unwritten remaining read addresses in the read address set, a supplementary read address with free queue capacity is selected from the remaining read addresses. The supplementary read address is written into the memory access queue to achieve iterative writing of the read address; The step of sequentially accessing multiple terms to be accessed according to the writing order of the read addresses in the access queue includes: The read addresses in the memory access queue are read according to the clock cycle of the target chip and the write order of the read addresses; Based on the read address, the locations of multiple metadata of the terms to be accessed are determined; The total memory access configuration instruction is parsed to obtain the memory access operation parameters corresponding to the multiple memory access terms to be accessed. Based on the memory access operation parameters, the metadata of the multiple terms to be accessed is accessed.

2. The method according to claim 1, characterized in that, The step of parsing the total memory access configuration instruction to obtain the read addresses corresponding to the multiple memory access terms, and writing the multiple read addresses into the memory access queue in the target chip, includes: The total memory access configuration instruction is parsed to obtain a set of read addresses for multiple memory access terms, and the set of read addresses includes the read addresses corresponding to each of the multiple memory access terms. Determine the memory access queue in the target chip, select the read address with the target queue capacity from the read address set, and write the selected read address into the memory access queue.

3. The method according to claim 1, characterized in that, The step of generating a total memory access configuration instruction for multiple terms to be accessed based on the memory access requirements includes: The memory access request is parsed to obtain the memory access parameters corresponding to the multiple memory access terms to be accessed; Based on the memory access parameters, a total memory access configuration instruction is generated for multiple terms to be accessed, so that multiple memory access hardware threads in the target chip can access the multiple terms to be accessed according to the total memory access configuration instruction.

4. The method according to any one of claims 1-3, characterized in that, The method further includes: If the number of the stored words to be visited meets the number threshold condition, determine the general-purpose registers in the target chip that are in an idle state; Based on the memory access operation parameters of the term to be accessed, the general-purpose register is configured as the read address storage area of ​​the term to be accessed, and the multiple read addresses obtained by parsing are stored in the general-purpose register; Multiple read addresses are read from the general-purpose register and written into the memory access queue in the target chip.

5. A word element memory access device, characterized in that, include: The acquisition module is configured to acquire the memory access requirements of the target chip for multiple memory access terms; The generation module is configured to generate a total memory access configuration instruction for multiple terms to be accessed based on the memory access requirements. The parsing module is configured to parse the total memory access configuration instruction to obtain the read addresses corresponding to the multiple memory access terms, and write the multiple read addresses into the memory access queue in the target chip. The memory access queue is a first-in-first-out buffer queue with a fixed storage depth. When there is free queue capacity in the memory access queue and there are unwritten remaining read addresses in the read address set, a supplementary read address with free queue capacity is selected from the remaining read addresses and written into the memory access queue. The read address set is a set composed of the read addresses corresponding to the multiple memory access terms. The memory access module is configured to access multiple terms to be accessed sequentially according to the writing order of the read addresses in the memory access queue. When there is free queue capacity in the memory access queue and there are unwritten remaining read addresses in the read address set, selecting a supplementary read address with free queue capacity from the remaining read addresses and writing the supplementary read address into the memory access queue includes: According to the clock cycle of the target chip, the free queue capacity and address write status of the memory access queue are monitored. If there is free queue capacity in the memory access queue and there are unwritten remaining read addresses in the read address set, a supplementary read address with free queue capacity is selected from the remaining read addresses. The supplementary read address is written into the memory access queue to achieve iterative writing of the read address; The step of sequentially accessing multiple terms to be accessed according to the writing order of the read addresses in the access queue includes: The read addresses in the memory access queue are read according to the clock cycle of the target chip and the write order of the read addresses; Based on the read address, the locations of multiple metadata of the terms to be accessed are determined; The total memory access configuration instruction is parsed to obtain the memory access operation parameters corresponding to the multiple memory access terms to be accessed. Based on the memory access operation parameters, the metadata of the multiple terms to be accessed is accessed.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 4.

7. An electronic device comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 4.

8. A computer program product, the computer program product comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 4.

Images