Data processing method and device of large model, equipment and medium

Abstract

This invention provides a data processing method, apparatus, device, and medium for large models, applied to a chip. The chip includes a PNM (Producer-Consumer Module), a SIP (System-in-Package), and a DMA (Digital-Distribution Module). A producer-consumer model is established between the SIP and the PNM, with both dynamically alternating between the roles of producer and consumer. This invention utilizes the DMA to load data from external memory into a shared cache. During the pre-filling and generation phases of the large model, computational tasks are allocated to both the PNM and the SIP. The PNM and SIP dynamically alternate between the producer and consumer roles. This collaborative working method fully utilizes the chip's computational resources, improving processing efficiency and throughput. Therefore, in edge applications of large models, the collaborative work between the PNM, SIP, and DMA can be effectively coordinated, solving storage bandwidth and capacity issues and improving model inference efficiency and performance.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus, device and medium for processing large-scale models. Background Technology

[0002] As large models evolve, their number of parameters and computational complexity continue to increase. Running large models requires frequent access to a large number of parameters (weights, biases, etc.) and data, but the total memory capacity of edge devices is far smaller than that of data center servers, making it difficult to accommodate all the parameters and data of large models. Therefore, there is an urgent need for an effective data processing method for large models to address the storage bandwidth and capacity issues faced by large models in edge applications. Summary of the Invention

[0003] In view of the above problems, the present invention is proposed to provide a data processing method, apparatus, device and medium for large models that overcomes or at least partially solves the above problems.

[0004] To achieve the above and other related objectives, this invention provides a large-scale model data processing method applied to a chip, the chip including a PNM, a SIP, and a DMA, wherein a producer-consumer model is set between the SIP and the PNM, and the two dynamically alternately act as producers and consumers, the method comprising:

[0005] DMA loads model parameters and initial data from external memory into a shared cache;

[0006] In the pre-filling phase, PNM obtains the query vector, key vector, and value vector corresponding to each token of the input sequence from HBM, and performs data type conversion on the key vector and the value vector to obtain the converted key vector and the converted value vector, and places the converted key vector and the converted value vector in HBM or the shared cache; SIP performs attention calculation on the query vector, the converted key vector, and the converted value vector corresponding to each token to generate the updated context representation of each token; SIP obtains the output layer weights from HBM, and performs matrix multiplication calculation on the output layer weights and the context representation to obtain the output representation of each token;

[0007] During the generation phase, PNM obtains the current query vector, current key vector, and current value vector corresponding to the current input token from HBM, and performs matrix multiplication on the current query vector and the current key vector to obtain an attention score; SIP performs rotational position encoding on the attention score to obtain an encoded attention score; PNM performs matrix multiplication on the encoded attention score and the current value vector to obtain the context vector of the current input token; SIP performs softmax calculation on the context vector of the current input token to obtain a probability weight vector; SIP obtains the output layer weights from HBM, and performs matrix multiplication on the output layer weights and the probability weight vector to obtain the inference result of the current input token.

[0008] Optionally, before the step of loading model parameters and data from external storage to the shared cache by the DMA, the method further includes:

[0009] Pre-configure the loop parameters of PNM and declare them outside the loop;

[0010] Pre-configure the loop parameters for DMA, and declare them either outside or inside the loop.

[0011] Optionally, before the step in the pre-filling stage where PNM obtains the query vector, key vector, and value vector corresponding to each token of the input sequence from HBM, the method further includes:

[0012] The prompts input by the user are segmented and vectorized to obtain the input sequence;

[0013] For each token in the input sequence, calculate its corresponding query vector, key vector, and value vector, and transmit the query vector, key vector, and value vector corresponding to each token to HBM.

[0014] Optionally, after the step of obtaining the query vector, key vector, and value vector corresponding to each token of the input sequence from the HBM, and performing transformation processing on the key vector and the value vector to obtain the transformed key vector and the transformed value vector, the method further includes:

[0015] PNM checks whether the task queue in the shared cache is full;

[0016] If the task queue is not full, PNM packages the transformed key vector and the transformed value vector into a new task and puts it into the task queue, so that SIP can retrieve the new task from the task queue and use the transformed key vector and value vector carried in the new task to perform subsequent attention calculations.

[0017] Optionally, after the step of the PNM checking whether the task queue in the shared cache is full, the method further includes:

[0018] If the task queue is full, PNM will conditionally wait until there is space in the task queue.

[0019] Optionally, after the step of performing rotational position encoding on the attention score to obtain the encoded attention score, the SIP further includes:

[0020] SIP checks whether the task queue in the shared cache is full;

[0021] If the task queue is not full, SIP packages the encoded attention score into a new task and puts it into the task queue, so that PNM can retrieve the new task from the task queue and use the encoded attention score carried in the new task to perform subsequent matrix multiplication calculations.

[0022] Optionally, after the step of SIP checking whether the task queue in the shared cache is full, the method further includes:

[0023] If the task queue is full, SIP will conditionally wait until there is space in the task queue.

[0024] Secondly, the present invention also provides a large-scale model data processing device applied to a chip, the chip including a PNM, a SIP, and a DMA, wherein a producer-consumer model is set between the SIP and the PNM, and the two dynamically alternately act as producers and consumers, the device comprising:

[0025] The loading module is used for DMA to load model parameters and initial data from external memory into the shared cache;

[0026] The pre-filling module is used in the pre-filling stage to: PNM obtains the query vector, key vector, and value vector corresponding to each token of the input sequence from HBM; performs data type conversion on the key vector and the value vector to obtain the converted key vector and the converted value vector; and places the converted key vector and the converted value vector in HBM or a shared cache. SIP performs attention calculation on the query vector, the converted key vector, and the converted value vector corresponding to each token to generate an updated context representation for each token. SIP obtains the output layer weights from HBM and performs matrix multiplication calculation on the output layer weights and the context representation to obtain the output representation for each token.

[0027] The generation module is used in the generation phase to: PNM obtains the current query vector, current key vector, and current value vector corresponding to the current input token from HBM, and performs matrix multiplication on the current query vector and the current key vector to obtain an attention score; SIP performs rotational position encoding on the attention score to obtain an encoded attention score; PNM performs matrix multiplication on the encoded attention score and the current value vector to obtain a context vector of the current input token; SIP performs softmax calculation on the context vector of the current input token to obtain a probability weight vector; and SIP obtains the output layer weights from HBM and performs matrix multiplication on the output layer weights and the probability weight vector to obtain the inference result of the current input token.

[0028] Thirdly, the present invention provides an electronic device comprising: a memory and a processor; the memory for storing a computer program; and the processor for executing the computer program stored in the memory to cause the electronic device to perform the steps of the large-scale data processing method described above.

[0029] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by an electronic device, implements the steps of the large model data processing method described above.

[0030] The above-described one or more technical solutions provided by this invention can have the following advantages or at least achieve the following technical effects:

[0031] This invention utilizes DMA to load data from external memory into a shared cache. During the pre-filling and generation phases of large models, computational tasks are distributed between two units: PNM and SIP. PNM and SIP dynamically alternate between acting as producer and consumer. This collaborative approach fully utilizes the chip's computing resources, improving processing efficiency and throughput. Therefore, in edge applications of large models, it effectively coordinates the collaborative work between PNM, SIP, and DMA, resolving storage bandwidth and capacity issues and improving model inference efficiency and performance. Attached Figure Description

[0032] Figure 1 The diagram shows a flowchart of a data processing method for a large model in one embodiment of the present invention.

[0033] Figure 2 This is a schematic diagram of the calculation process during pre-filling in one embodiment of the present invention;

[0034] Figure 3 This is a schematic diagram of the calculation process during generation in one embodiment of the present invention;

[0035] Figure 4 This diagram illustrates the generation of SIP task parameters using a producer-consumer model in one embodiment of the present invention.

[0036] Figure 5 Shown as an embodiment of the present invention, according to Figure 4 The producer-consumer model is illustrated using a scalar thread as a producer.

[0037] Figure 6 Shown as an embodiment of the present invention, according to Figure 4 The producer-consumer model is illustrated using the common thread as a producer.

[0038] Figure 7 This is a schematic diagram illustrating the generation of PNM task parameters using a producer-consumer model in one embodiment of the present invention.

[0039] Figure 8 Shown as an embodiment of the present invention, according to Figure 7 The producer-consumer model is illustrated using a scalar thread as a producer.

[0040] Figure 9 Shown as an embodiment of the present invention, according to Figure 7 The producer-consumer model is illustrated using the common thread as a producer.

[0041] Figure 10 This is a schematic diagram of the functional modules of a large model data processing device in one embodiment of the present invention.

[0042] Figure 11 The diagram shown is a schematic representation of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0043] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0044] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0045] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0046] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this disclosure described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.

[0047] Unless otherwise stated, the term "multiple" means two or more.

[0048] In this embodiment of the disclosure, the character " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B means: A or B.

[0049] The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0050] Before providing a further detailed description of the present invention, the nouns and terms used in the embodiments of the present invention are explained, and the nouns and terms used in the embodiments of the present invention are subject to the following interpretations:

[0051] <1> The producer-consumer model is used to solve the problem of data transfer and collaborative work between multiple threads (or processes). Its core idea is to use a "buffer" (or "message queue") as an intermediary, where "producer" threads are responsible for generating data and putting it into the buffer, and "consumer" threads are responsible for retrieving data from the buffer and processing it, thereby achieving decoupling of production and consumption and efficient collaboration.

[0052] Producer: Responsible for generating data (such as calculation results, user requests, task instructions, etc.) and placing the data into a buffer. When the buffer is full, the producer must wait (block) until there is free space in the buffer before continuing production.

[0053] Consumer: Responsible for retrieving data from the buffer and processing it (such as parsing requests, executing tasks, and storing results). When the buffer is empty, the consumer needs to wait (block) until new data is available in the buffer before continuing to consume data.

[0054] A buffer acts as an intermediary between producers and consumers, typically a finite-size queue (FIFO) or array. It balances the processing speed of producers and consumers (e.g., when producers generate data faster than consumers process it, the buffer temporarily stores the data; conversely, the consumer waits), achieving decoupling between the two (no direct interaction is required, communication is only through the buffer).

[0055] The technical solutions of the present invention will now be described in detail with reference to the accompanying drawings.

[0056] Please see Figure 1 An embodiment of the present invention provides a data processing method for a large model, applied to a chip, the chip including a PNM, a SIP, and a DMA, wherein a producer-consumer model is set between the SIP and the PNM, and the two dynamically alternately play the roles of producer and consumer. The method may include the following steps S10~S30:

[0057] In step S10, the DMA loads model parameters and initial data from external storage into the shared cache.

[0058] The chips include PNM (Processing Near Memory), SIP (System in Package), and DMA (Direct Memory Access).

[0059] The SIP and PNM employ a producer-consumer model, with SIP and PNM dynamically alternating between the producer and consumer roles. This alternation reflects the data dependencies between different operations in the pipeline: when an operation unit (such as PNM or SIP) completes a computational step and generates the data required for subsequent steps, it acts as a producer; when another operation unit needs to use this data for the next computation, it acts as a consumer.

[0060] In some operations, PNM is the producer and SIP is the consumer; while in other operations, SIP is the producer and PNM is the consumer.

[0061] DMA (Direct Memory Access) is used to load model parameters (such as Q, K, V projection matrices, and output layer weights) and data (such as token embeddings of the input sequence) from external memory (such as main memory) into a shared cache (such as L1) on the chip before large models begin execution, providing a foundation for subsequent SIP and PNM calculations. The use of DMA reduces the burden of memory access.

[0062] Model parameters can refer to the parameters required for the large model to run (such as the pre-filling stage or the generation stage), such as output layer weights, biases, and other possible parameters.

[0063] Shared cache can refer to various types of on-chip memory, such as L1 cache; please refer to [link / reference]. Figure 4 and Figure 7 L1 in the middle.

[0064] In its implementation, before the large model begins execution, DMA establishes an efficient data transfer path between relatively slow external memory (such as main memory) and high-speed shared cache (such as L1 cache) based on preset configuration parameters (such as source address, destination address, transfer length, etc.). This allows the parameters and initial input data required by the model to be moved to the shared cache in batches, reducing the number of times the slow main memory is accessed during subsequent model calculations (such as matrix multiplication, activation functions, etc.) (i.e. reducing the latency of accessing slow storage) and significantly improving computational performance.

[0065] In step S20, during the pre-filling stage, PNM obtains the query vector, key vector, and value vector corresponding to each token of the input sequence from HBM, and performs data type conversion processing on the key vector and the value vector to obtain the converted key vector and the converted value vector, and places the converted key vector and the converted value vector in HBM or the shared cache; SIP performs attention calculation on the query vector, the converted key vector, and the converted value vector corresponding to each token to generate the updated context representation of each token; SIP obtains the output layer weights from HBM, and performs matrix multiplication calculation on the output layer weights and the context representation to obtain the output representation of each token.

[0066] HBM (High Bandwidth Memory) is an advanced, stacked DRAM (Dynamic Random Access Memory) technology specifically designed for high-performance computing (HPC) and graphics processing (GPU). It addresses the bandwidth and energy efficiency bottlenecks encountered by traditional memory technologies such as GDDR. Please refer to [link / reference]. Figure 2 and Figure 3 L3 in the middle.

[0067] Please see Figure 2 Pre-fill stage:

[0068] PNM (Producer) is used to obtain the Q, K, and V vectors corresponding to each token in the input sequence from HBM; and to perform data type conversion processing on the K and V vectors (such as applying K and V projection matrices) to generate the converted K and V, which can then be written into HBM or shared cache.

[0069] SIP (consumer) retrieves the "Q vector and the transformed K and V vectors generated by PNM" from HBM or the shared cache, and then performs attention calculations on the Q vector and the transformed K and V vectors to generate the updated context representation of each token (usually corresponding to the result of Attention(Q, K, V)); then it can write the updated context representation of each token into HBM or the shared cache.

[0070] SIP (consumer) obtains the output layer weights from the shared cache, and then performs matrix multiplication on the updated context representation and output layer weights of each token to obtain the output representation of each token.

[0071] During the pre-filling phase, PNM first acts as a producer (processing the K and V vectors of each token), and then as a consumer (processing the output of SIP and ultimately computing the output representation of each token). SIP, on the other hand, acts as a consumer (processing the transformed K and V vectors of PNM's output), and its output is then consumed by SIP. This reflects a dynamic alternation of roles.

[0072] In its implementation, during the pre-filling phase, PNM obtains the query vector (Q), key vector (K), and value vector (V) corresponding to each token in the input sequence from HBM. It then performs data type conversion on the key and value vectors for each token, obtaining the converted key and value vectors, which are then written to HBM or a shared cache. SIP retrieves the query vector, converted key vector, and converted value vector for each token generated by PNM from HBM or the shared cache, performs attention calculations to generate an updated context representation for each token, and writes this updated context representation to the shared cache. SIP obtains the output layer weights from HBM or the shared cache, performs matrix multiplication on the output layer weights and context representation, obtaining the output representation for each token, and writes this output representation to HBM or the shared cache. Thus, through HBM or the shared cache, PNM and SIP can efficiently transfer and share data, enabling data to be accessed and processed quickly by PNM and SIP.

[0073] In step S30, during the generation phase, PNM obtains the current query vector, current key vector, and current value vector corresponding to the current input token from HBM, and performs matrix multiplication on the current query vector and the current key vector to obtain an attention score; SIP performs rotational position encoding on the attention score to obtain an encoded attention score; PNM performs matrix multiplication on the encoded attention score and the current value vector to obtain the context vector of the current input token; SIP performs softmax calculation on the context vector of the current input token to obtain a probability weight vector; SIP obtains the output layer weights from HBM, and performs matrix multiplication on the output layer weights and the probability weight vector to obtain the inference result of the current input token.

[0074] Please refer to Figure 3 Generation stage:

[0075] The PNM (producer) obtains the Q, K, and V vectors of the current input token from the HBM, performs matrix multiplication on the current Q and K vectors to calculate the attention score, and then writes the attention score of the current input token into the HBM or a shared cache for use in subsequent steps.

[0076] The SIP (consumer) obtains the "attention score" generated by PNM from HBM or a shared buffer, performs Rotation Position Encoding (RoPE) on the attention score to generate an encoded attention score, and then writes the encoded attention score into HBM or a shared buffer for use in subsequent steps. In this step, the SIP acts as a consumer, consuming the output of PNM and becoming a producer in the next step.

[0077] The PNM (consumer) retrieves the "encoded attention score" generated by the SIP from the shared cache, performs matrix multiplication on the encoded attention score and the current value vector to generate the context vector of the current input token, and then writes this context vector of the current input token into the HBM or the shared cache. At this point, the PNM acts as a consumer in this step, consuming the output of the SIP and becoming a producer in the next step.

[0078] The SIP (consumer) retrieves the "context vector of the current input token" generated by PNM from HBM or the shared cache, performs softmax calculation on the context vector of the current input token to generate a probability weight vector, and then writes this probability weight vector into HBM or the shared cache. At this point, the SIP acts as a consumer in this step, consuming the output of PNM and becoming a producer in the next step.

[0079] The SIP (consumer) retrieves the output layer vector from the HBM and performs matrix multiplication on the probability weight vector and the output layer weights to generate the inference result for the current input token. This inference result can then be written to the shared cache. In this step, the SIP acts as a consumer, consuming its output and completing the processing of the current token.

[0080] During the generation phase, the producer-consumer roles become more pronounced: PNM → SIP → PNM → SIP → SIP. Each step depends on the output of the previous step, and PNM and SIP alternate in computation and result transfer.

[0081] In its implementation, during the generation phase, PNM can obtain the current query vector, current key vector, and current value vector corresponding to the current input token from HBM. It can then perform matrix multiplication on the current query vector and current key vector to obtain the attention score, which is then written to HBM or a shared cache. SIP can obtain the attention score of the current input token from HBM or a shared cache, perform rotation and position encoding to obtain the encoded attention score, and write the encoded attention score to HBM or a shared cache. PNM obtains the encoded attention score from HBM or a shared cache, performs matrix multiplication on the encoded attention score and the current value vector to obtain the context vector of the current input token, and writes the context vector of the current input token to HBM or a shared cache. SIP can obtain the context vector of the current input token from HBM or a shared cache and perform softmax calculation to obtain the probability weight vector. Finally, SIP obtains the probability weight vector and output layer weights from HBM, performs matrix multiplication on the output layer weights and probability weight vectors to obtain the inference result for the current input token, and writes the inference result for the current input token into HBM or a shared cache. Thus, through HBM or a shared cache, PNM and SIP can efficiently transfer and share data, enabling data to be quickly accessed and processed by PNM and SIP.

[0082] Furthermore, in one embodiment, prior to step S10, the method may further include S40:

[0083] Step S40: Pre-configure the loop inner and outer parameters of PNM and declare them outside the loop;

[0084] Pre-configure the loop parameters for DMA, and declare them either outside or inside the loop.

[0085] The configuration parameters for DMA can include: channel address (specifying the source and destination addresses of DMA transfer), priority (setting the priority of DMA transfer to ensure that high-priority transfers are executed first), data transfer direction (specifying data transfer from peripheral to memory and from memory to peripheral), loop mode (configuring whether DMA automatically restarts after the transfer is completed), and data transfer amount (setting the amount of data transferred each time).

[0086] PNM configuration parameters can include: number of loops, loop conditions (setting the termination conditions of the loop, such as data processing completion or reaching a specific number of iterations), data buffer size, and data processing logic (defining the specific data processing logic of PNM inside the loop, such as data transformation and matrix operations).

[0087] In the specific implementation, before the DMA loads data from external storage into the shared cache, the SIP can initiate the configuration of the loop-inside and loop-outside parameters of PNM and DMA. The SIP pre-configures the DMA loop-inside and loop-outside parameters once, declaring them either outside or inside the loop. These parameters can include channel address, priority, data transfer direction, loop mode, and data transfer amount. After configuration, the SIP can trigger the DMA to start executing the data transfer task within the loop via a specific command or interface (e.g., the DMA loads model parameters and initial data from external storage into the shared cache according to the pre-configured parameters). Similarly, the SIP pre-configures the PNM loop-inside and loop-outside parameters once, declaring them outside the loop. These parameters can include loop count, loop condition, data buffer size, and data processing logic. After configuration, the SIP can trigger the PNM to start executing the data processing task within the loop via a specific command or interface (e.g., the PNM retrieves data from the shared cache and processes it according to the pre-configured parameters). Through the parameter configuration and triggering mechanism initiated by the SIP, the work of DMA and PNM can be fully scheduled and optimized, thereby adapting to different data processing needs and improving the inference effect and performance of large models.

[0088] Furthermore, in one embodiment, before step S20, the method may further include steps S50-S60:

[0089] Step S50: The prompt words input by the user are segmented and vectorized to obtain the input sequence;

[0090] Step S60: For each token in the input sequence, calculate its corresponding query vector, key vector, and value vector, and transmit the query vector, key vector, and value vector corresponding to each token to HBM.

[0091] The prompt words can refer to the text entered by the user into the large model, which is used to guide the large model to generate a specific type of response.

[0092] The input sequence refers to the sequence of user input, which is usually formed after tokenization and vectorization. This sequence is typically composed of a series of tokens (such as a word, a character, or other forms of text fragments).

[0093] In the specific implementation, after receiving the prompt words input by the user, a word segmenter can be used to segment the prompt words, and the segmented prompt words can be vectorized (e.g., embedded) to obtain the input sequence. For each token in the input sequence, its corresponding query vector, key vector, and value vector (linear transformation matrix) are calculated, and the calculated query vector, key vector, and value vector for each token are transmitted to HBM so that subsequent PNM and SIP can access and use this data. Thus, the large model can effectively process the prompt words input by the user and generate the corresponding output.

[0094] In this embodiment, DMA is responsible for loading data from external memory into the shared cache. During the pre-filling and generation phases of the large model, computational tasks are assigned to both the PNM and SIP units. The PNM and SIP dynamically alternate between acting as producer and consumer. This collaborative approach fully utilizes the chip's computing resources, improving processing efficiency and throughput. Therefore, in edge applications of large models, the collaborative work between the PNM, SIP, and DMA can be effectively coordinated, resolving storage bandwidth and capacity issues and improving model inference efficiency and performance.

[0095] Based on the foregoing embodiments, a second embodiment of the data processing method for the large model of the present invention is proposed. In this embodiment, after step S20, in which PNM obtains the query vector, key vector, and value vector corresponding to each token of the input sequence from HBM, and performs transformation processing on the key vector and the value vector to obtain the transformed key vector and the transformed value vector, the following steps A10~A20 may be included:

[0096] Step A10: PNM checks whether the task queue in the shared cache is full;

[0097] The shared cache serves as a buffer between PNM and SIP, storing tasks to be processed (e.g., containing the transformed key vector and the transformed value vector).

[0098] Please see Figures 4-6 , Figure 4 It demonstrates the relationship between scalar threads (producers) and common threads (consumers), as well as the management of task queues; Figure 5The scalar thread is described in detail as a producer in a loop, including checking if the task queue is full (check task: CFGSIP full) and generating new tasks (gen task: CFGSIP params). Figure 6 It describes the loop operations of the common thread as a consumer, including getting tasks (gettask: CFGSIP params) and running tasks (run task: attention).

[0099] from Figures 4-6 As can be seen, PNM is the producer, which is responsible for generating tasks and putting them into the task queue; SIP is the consumer, which is responsible for retrieving tasks from the task queue and executing them.

[0100] In the specific implementation, during the pre-filling stage, PNM (acting as the producer at this time) transforms the key vector and value vector to obtain the transformed key vector and value vector, and writes them into the shared cache. Then, before PNM generates a new task (such as gen task: CFGSIP params), it also checks whether the current task queue in the shared cache is full.

[0101] Step A20: If the task queue is not full, PNM packages the transformed key vector and the transformed value vector into a new task and puts it into the task queue, so that SIP can retrieve the new task from the task queue and use the transformed key vector and the transformed value vector carried in the new task to perform subsequent attention calculations.

[0102] In practical implementation, when the current task queue is not full, PNM can package the transformed key vector and value vector into a new task and put it into the task queue. Then, SIP can retrieve the new task from the shared cache task queue and use the transformed key vector and value vector carried in the new task to perform subsequent attention calculations. Thus, through the task queue, PNM and SIP in the producer-consumer model can work in parallel to a certain extent. PNM can prepare the next task while SIP processes the previous task, effectively balancing the processing speed between PNM and SIP and avoiding PNM wasting computing resources waiting for queue space.

[0103] Furthermore, in one embodiment, after step A10, the following step A30 may also be included:

[0104] Step A30: If the task queue is full, PNM will wait conditionally until there is space in the task queue.

[0105] In its implementation, when the current task queue is full, PNM conditionally waits until an empty slot becomes available in the queue. This avoids wasting computational resources.

[0106] In this embodiment, PNM checks whether the task queue in the shared cache is full. If the task queue is not full, PNM packages the transformed key vector and the transformed value vector into a new task and puts it into the task queue. SIP then retrieves the new task from the task queue and uses the transformed key vector and transformed value vector carried in the new task to perform subsequent attention calculations. Thus, through the task queue, an efficient and balanced collaborative relationship can be established between PNM and SIP, effectively balancing the processing speed between PNM and SIP and preventing PNM from wasting computational resources waiting for queue space.

[0107] Based on the foregoing embodiments, a third embodiment of the data processing method for the large model of the present invention is proposed. In this embodiment, after the step S20 in which SIP performs rotational position encoding processing on the attention score to obtain the encoded attention score, the following steps B10~B20 may be further included:

[0108] Step B10: SIP checks whether the task queue in the shared cache is full.

[0109] Please refer to Figures 7-9 , Figure 7 This demonstrates the relationship between the scalar thread and the common thread. The scalar thread (producer) is responsible for producing tasks, while the common thread is responsible for consuming tasks. Tasks are placed in states: valid slots, and specific tasks include CGFPNMA params, CGFPNMB params, and CGFPNMC params. Figure 8 The scalar thread is described in detail as a producer in a loop, including checking if the task queue is full (check task: CFGPNMA full) and generating new tasks (gentask: CFGPNMA params). Figure 9It describes the loop operations of the common thread as a consumer, including waiting for a task (CFGPNA params), running a task (PNMA with params), getting a task (CFGPNMA params), conditional waiting (CFGPNC params), and attention.

[0110] from Figures 7-9 As can be seen, SIP is the producer, responsible for generating tasks and putting them into the task queue; PNM is the consumer, responsible for retrieving tasks from the task queue and executing them.

[0111] In the specific implementation, SIP performs rotational position encoding on the attention score to obtain the encoded attention score, and writes it into the shared cache (at this time, SIP acts as the producer). Then, before SIP generates new tasks (such as CFPPNMAparams, CFPPNMB params, CFPPNMC params), it also checks whether the corresponding task queue (such as CFPPNMAparams, CFPPNMB params, CFPPNMC params) is full.

[0112] In step B20, if the task queue is not full, SIP packages the encoded attention score into a new task and puts it into the task queue so that PNM can retrieve the new task from the task queue and use the encoded attention score carried in the new task to perform subsequent matrix multiplication calculations.

[0113] In its implementation, if the task queue is not full, SIP packages the encoded attention score into a new task and adds it to the task queue. PNM can then retrieve the new task from the queue and use the encoded attention score carried in the new task to perform subsequent matrix multiplication calculations (e.g., run task: PNMA with params). This ensures the orderly generation and consumption of tasks, avoids task queue overflow, and guarantees efficient computation.

[0114] Furthermore, in one embodiment, after step B10, the following step B30 may also be included:

[0115] In step B30, if the task queue is full, SIP will conditionally wait until there is space in the task queue.

[0116] In this embodiment, if the task queue is full, SIP will conditionally wait until there is space in the task queue, thus avoiding the waste of computing resources.

[0117] In this embodiment, SIP checks whether the task queue in the shared cache is full. If the task queue is not full, SIP packages the encoded attention score into a new task and puts it into the task queue, so that PNM can retrieve the new task from the task queue and use the encoded attention score carried in the new task to perform subsequent matrix multiplication calculations. This ensures the orderly generation and consumption of tasks, avoids task queue overflow, and ensures efficient computation.

[0118] Based on the same inventive concept, the fourth embodiment of the present invention also provides a large-model data processing apparatus corresponding to the large-model data processing method of the foregoing embodiments. Since the principle by which the apparatus in the fourth embodiment solves the problem is similar to the large-model data processing method of the foregoing embodiments, the implementation of the apparatus can be referred to the implementation of the method, and repeated details will not be elaborated further. Please refer to... Figure 10 This invention discloses a large-scale data processing device for a chip, which includes a PNM (Producer-Consumer Model), a SIP (System-in-Package), and a DMA (Distribution-Directed Memory). A producer-consumer model is established between the SIP and the PNM, and the two dynamically alternate between acting as producers and consumers. The device may include:

[0119] Loading module 10 is used for DMA to load model parameters and initial data from external memory into a shared cache;

[0120] The pre-filling module 20 is used in the pre-filling stage to: PNM obtains the query vector, key vector, and value vector corresponding to each token of the input sequence from HBM; performs data type conversion on the key vector and the value vector to obtain the converted key vector and the converted value vector; and places the converted key vector and the converted value vector in HBM or a shared cache. SIP performs attention calculation on the query vector, the converted key vector, and the converted value vector corresponding to each token to generate an updated context representation for each token. SIP obtains the output layer weights from HBM and performs matrix multiplication calculation on the output layer weights and the context representation to obtain the output representation for each token.

[0121] The generation module 30 is configured to, during the generation phase, have the PNM obtain the current query vector, current key vector, and current value vector corresponding to the current input token from the HBM, and perform matrix multiplication on the current query vector and the current key vector to obtain an attention score; the SIP performs rotational position encoding on the attention score to obtain an encoded attention score; the PNM performs matrix multiplication on the encoded attention score and the current value vector to obtain a context vector of the current input token; the SIP performs softmax calculation on the context vector of the current input token to obtain a probability weight vector; and the SIP obtains the output layer weights from the HBM and performs matrix multiplication on the output layer weights and the probability weight vector to obtain the inference result of the current input token.

[0122] In addition, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described data processing method for large models.

[0123] Figure 11 This is a schematic block diagram of the electronic device provided in an embodiment of this application. Figure 11 As shown, the electronic device includes at least one processor 401, a memory 402, at least one network interface 403, and a user interface 405. The various components in the electronic device are coupled together via a bus system 404. It is understood that the bus system 404 is used to implement communication between these components. In addition to a data bus, the bus system 404 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 11 The general will label all buses as bus systems.

[0124] The user interface 405 may include a monitor, keyboard, mouse, trackball, clicker, button, touchpad, or touch screen.

[0125] It is understood that memory 402 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM) or programmable read-only memory (PROM), used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM) and synchronous static random access memory (SSRAM). The memories described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable categories of memory.

[0126] In this embodiment of the invention, the memory 402 is used to store various types of data to support the operation of the electronic device 400. Examples of this data include: any executable program for operation on the electronic device 400, such as the operating system 4021 and application programs 4022; the operating system 4021 contains various system programs, such as the framework layer, core library layer, driver layer, etc., for implementing various basic services and handling hardware-based tasks. The application program 4022 may contain various applications, such as a media player, browser, etc., for implementing various application services. The data processing method for the large model provided in this embodiment of the invention can be included in the application program 4022.

[0127] The methods disclosed in the above embodiments of the present invention can be applied to processor 401, or implemented by processor 401. Processor 401 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 401 or by instructions in the form of software. The processor 401 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 401 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. General-purpose processor 401 may be a microprocessor or any conventional processor, etc. The steps of the large-scale model data processing method provided in the embodiments of the present invention can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in memory. The processor reads the information in the memory and combines it with its hardware to complete the steps of the aforementioned method.

[0128] In an exemplary embodiment, the electronic device 400 may be used by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), or complex programmable logic devices (CPLDs) to perform the aforementioned method.

[0129] In summary, this invention utilizes DMA to load data from external memory into the shared cache; during the pre-filling and generation phases of large models, computational tasks are allocated to both the PNM and SIP units; the PNM and SIP dynamically alternate between acting as producer and consumer; this collaborative approach fully utilizes the chip's computational resources, improving processing efficiency and throughput. Therefore, in edge applications of large models, it effectively coordinates the collaborative work between the PNM, SIP, and DMA, resolving storage bandwidth and capacity issues, and improving model inference efficiency and performance.

[0130] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A data processing method for a large model, characterized in that, Applied to a chip, the chip including a PNM, a SIP, and a DMA, wherein a producer-consumer model is established between the SIP and the PNM, and the two dynamically alternate between acting as producers and consumers, the method includes: DMA loads model parameters and initial data from external memory into a shared cache; In the pre-filling phase, PNM obtains the query vector, key vector, and value vector corresponding to each token of the input sequence from HBM, and performs data type conversion on the key vector and the value vector to obtain the converted key vector and the converted value vector, and places the converted key vector and the converted value vector in HBM or the shared cache; SIP performs attention calculation on the query vector, the converted key vector, and the converted value vector corresponding to each token to generate the updated context representation of each token; SIP obtains the output layer weights from HBM, and performs matrix multiplication calculation on the output layer weights and the context representation to obtain the output representation of each token; During the generation phase, PNM obtains the current query vector, current key vector, and current value vector corresponding to the current input token from HBM, and performs matrix multiplication on the current query vector and the current key vector to obtain an attention score; SIP performs rotational position encoding on the attention score to obtain an encoded attention score; PNM performs matrix multiplication on the encoded attention score and the current value vector to obtain the context vector of the current input token; SIP performs softmax calculation on the context vector of the current input token to obtain a probability weight vector; SIP obtains the output layer weights from HBM, and performs matrix multiplication on the output layer weights and the probability weight vector to obtain the inference result of the current input token.

2. The method according to claim 1, characterized in that, Prior to the step of loading model parameters and data from external storage into the shared cache by the DMA, the method further includes: Pre-configure the loop parameters of PNM and declare them outside the loop; Pre-configure the loop parameters for DMA, and declare them either outside or inside the loop.

3. The method according to claim 1, characterized in that, Before the step in the pre-filling stage where PNM obtains the query vector, key vector, and value vector corresponding to each token of the input sequence from HBM, the method further includes: The prompts input by the user are segmented and vectorized to obtain the input sequence; For each token in the input sequence, calculate its corresponding query vector, key vector, and value vector, and transmit the query vector, key vector, and value vector corresponding to each token to HBM.

4. The method according to claim 1, characterized in that, After the step of obtaining the query vector, key vector, and value vector corresponding to each token of the input sequence from the HBM, and transforming the key vector and the value vector to obtain the transformed key vector and the transformed value vector, the method further includes: PNM checks whether the task queue in the shared cache is full; If the task queue is not full, PNM packages the transformed key vector and the transformed value vector into a new task and puts it into the task queue, so that SIP can retrieve the new task from the task queue and use the transformed key vector and the transformed value vector carried in the new task to perform subsequent attention calculations.

5. The method according to claim 4, characterized in that, After the step of PNM checking whether the task queue in the shared cache is full, the method further includes: If the task queue is full, PNM will conditionally wait until there is space in the task queue.

6. The method according to claim 1, characterized in that, After the step of SIP performing rotational position encoding on the attention score to obtain the encoded attention score, the method further includes: SIP checks whether the task queue in the shared cache is full; If the task queue is not full, SIP packages the encoded attention score into a new task and puts it into the task queue, so that PNM can retrieve the new task from the task queue and use the encoded attention score carried in the new task to perform subsequent matrix multiplication calculations.

7. The method according to claim 6, characterized in that, After the step of SIP checking whether the task queue in the shared cache is full, it further includes: If the task queue is full, SIP will conditionally wait until there is space in the task queue.

8. A data processing device for a large model, characterized in that, The device is applied to a chip, which includes a PNM (Producer-Consumer Module), a SIP (System-in-Package), and a DMA (Distribution-Directed Memory). A producer-consumer model is established between the SIP and the PNM. The PNM processes data using a single thread, while the SIP dynamically partitions and processes data using multiple threads. The device includes: The loading module is used for DMA to load model parameters and initial data from external memory into the shared cache; The pre-filling module is used in the pre-filling stage to: PNM obtains the query vector, key vector, and value vector corresponding to each token of the input sequence from HBM; performs data type conversion on the key vector and the value vector to obtain the converted key vector and the converted value vector; and places the converted key vector and the converted value vector in HBM or a shared cache. SIP performs attention calculation on the query vector, the converted key vector, and the converted value vector corresponding to each token to generate an updated context representation for each token. SIP obtains the output layer weights from HBM and performs matrix multiplication calculation on the output layer weights and the context representation to obtain the output representation for each token. The generation module is used in the generation phase to: PNM obtains the current query vector, current key vector, and current value vector corresponding to the current input token from HBM, and performs matrix multiplication on the current query vector and the current key vector to obtain an attention score; SIP performs rotational position encoding on the attention score to obtain an encoded attention score; PNM performs matrix multiplication on the encoded attention score and the current value vector to obtain a context vector of the current input token; SIP performs softmax calculation on the context vector of the current input token to obtain a probability weight vector; and SIP obtains the output layer weights from HBM and performs matrix multiplication on the output layer weights and the probability weight vector to obtain the inference result of the current input token.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to cause the processor to perform the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program that, when executed by a processor, implements the method of any one of claims 1 to 7.

Images