Method, device, equipment and medium for realizing causality mask in large model inference
By introducing the collaborative work of VMPE and SFU into the hardware architecture and utilizing preset bitwise operation masking operations, the problems of storage space and processing latency in the implementation of causal masking are solved, and efficient causal masking processing is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SIENGINE TECH CO LTD
- Filing Date
- 2026-04-24
- Publication Date
- 2026-05-29
AI Technical Summary
Existing causality masking implementations require pre-constructing a mask matrix, leading to increased storage space and processing delays.
It adopts a hardware-based implementation architecture and achieves causal masking by working together with VMPE and SFU without the need to pre-construct a mask matrix. It uses preset bitwise operation masking to process data in vector units.
It effectively saves storage space, reduces processing latency, and enables subsequent operations to be performed immediately after obtaining a vector input.
Smart Images
Figure CN122114012A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of self-attention mechanism technology, specifically to a method, apparatus, device, and medium for implementing causal masking in large model inference. Background Technology
[0002] A causal mask is a type of mask used in self-attention mechanisms to prevent the model from seeing only current and previous information, and thus preventing it from seeing future information. It is typically implemented as a two-dimensional matrix, where all elements below the diagonal are 1s and all elements above the diagonal are 0s. This requires a dot-matrix operation with the original output data. Alternatively, all elements below the diagonal can be 0s, and all elements above the diagonal can be negative infinity. This requires a dot-matrix addition operation with the original output data. Software implementations usually use the latter approach.
[0003] Therefore, existing methods for implementing causal masks require the mask matrix to be constructed in advance, which increases storage space, and the entire matrix must be calculated before operations can be performed on the mask matrix, which increases processing latency. Summary of the Invention
[0004] This application provides a method, apparatus, device, and medium for implementing causal masking in large model inference, which eliminates the need to pre-construct the mask matrix, thereby effectively saving storage space.
[0005] In a first aspect, embodiments of this application provide a method for implementing causality masking in large-scale model inference, based on a defined hardware implementation architecture. The method for implementing causality masking in large-scale model inference includes: After completing a vector-matrix operation, VMPE sends the result to SFU. SFU receives cmd commands sent by FW to obtain input data; SFU processes the acquired input data based on a preset bitwise mask operation and then performs a Softmax operation. The hardware implementation architecture includes a VMPE responsible for handling vector-matrix operations, an SFU responsible for handling other vector operations, and a FW running on a RISC-V CPU for controlling the DMA, VMPE, and SFU.
[0006] In conjunction with the first aspect, in one implementation method, The VMPE and SFU are connected via VMPE2SFU_FIFO and SFU2VMPE_FIFO, and a ShareRAM is also provided between the VMPE and SFU. The VMPE2SFU_FIFO is the channel through which VMPE sends data to SFU, and the SFU2VMPE_FIFO is the channel through which SFU sends data to VMPE.
[0007] In conjunction with the first aspect, in one implementation method, The DMA is located between the RISC-V CPU and ShareRAM; The RISC-V CPU is connected to VMPE and SFU.
[0008] In conjunction with the first aspect, in one implementation, after completing a vector-matrix operation, the VMPE sends the result to the SFU, specifically including: After completing a vector-matrix operation, VMPE sends the result to SFU via VMPE2SFU_FIFO or via ShareRAM.
[0009] In conjunction with the first aspect, in one implementation, the SFU receives a cmd command sent by the FW to obtain input data, specifically including: The FW sends a cmd command to the SFU, and the cmd command sent by the FW to the SFU specifies the input source with input data; SFU receives cmd commands sent by FW and obtains input data from the input source; The input source is ShareRAM or VMPE2SFU_FIFO, the input data is the Q / K matrix after position encoding, and the cmd command is a cmd format command related to causality masking.
[0010] In conjunction with the first aspect, in one implementation method, The two fields related to the causality mask are phase and token_position_base; The phase is either Prefill or Decode, and the causality mask is only valid in the Prefill phase; no causality mask is needed in the Decode phase. The token_position_base corresponds to the starting position of a set of input tokens, that is, the position number of the first token in the complete input, starting from 0.
[0011] In conjunction with the first aspect, in one implementation, the preset bitwise operation mask operation is specifically represented as: vector_out = vector_in&((1024'h1<<((token_position_base+1)<<3))-1) Where `token_position_base` represents the token position base address, counting from 0; `token_position_base+1` means incrementing `token_position_base` by 1; `(token_position_base+1)<<3` means shifting the result of `(token_position_base+1)` 3 bits to the left, equivalent to multiplying by 8; `1024'h1<<((token_position_base+1)<<3)` means shifting 1 to the left by the number of bits in the result of `((token_position_base+1)<<3)`, where `(1024'h1<<((token_position_base+1)<<3)` represents the position base address of the token, counting from 0; `token_position_base+1` means incrementing `token_position_base` by 1; `(1024'h1<<((token_position_base+1)<<3)` means shifting 1 to the left by the number of bits in the result of `((token_position_base+1)<<3)`. h1<<((token_position_base+1)<<3))-1 means subtracting 1 from the result of (1024'h1<<((token_position_base+1)<<3)). vector_in&((1024'h1<<((token_position_base+1)<<3))-1) means performing a bitwise AND operation between vector_in and the result of ((1024'h1<<((token_position_base+1)<<3))-1). vector_in represents the vector of input data, and vector_out represents the vector of the calculated result.
[0012] Secondly, embodiments of this application provide an apparatus for implementing causality masking in large-scale model inference, the apparatus comprising: The sending module is used to drive VMPE to send the result of a vector-matrix operation to SFU after completing the operation. The acquisition module is used to drive the SFU to receive cmd commands sent by the FW to obtain input data; The execution module is used to drive the SFU to process the acquired input data based on a preset bitwise operation mask operation and then perform a Softmax operation. The hardware implementation architecture includes a VMPE responsible for handling vector-matrix operations, an SFU responsible for handling other vector operations, and a FW running on a RISC-V CPU for controlling the DMA, VMPE, and SFU.
[0013] Thirdly, embodiments of this application provide an apparatus for implementing causal masking in large model inference. The apparatus for implementing causal masking in large model inference includes a processor, a memory, and a program for implementing causal masking in large model inference stored in the memory and executable by the processor. When the program for implementing causal masking in large model inference is executed by the processor, it implements the steps of the method for implementing causal masking in large model inference described above.
[0014] Fourthly, embodiments of this application provide a computer-readable storage medium storing a program for implementing causal masking in large model inference, wherein when the program for implementing causal masking in large model inference is executed by a processor, it implements the steps of the method for implementing causal masking in large model inference described above.
[0015] The beneficial effects of the technical solutions provided in this application include: There is no need to construct the mask matrix in advance, thus effectively saving storage space. Furthermore, subsequent processing is done in units of vectors. This application also processes causal masks in units of vectors. Once a vector input is obtained, subsequent operations can begin without waiting for all matrix-matrix multiplication operations to be completed before proceeding with subsequent operations. Attached Figure Description
[0016] Figure 1 This is a flowchart illustrating the method for implementing causality masking in large-scale model inference in this application; Figure 2 A schematic diagram illustrating the structure of the hardware implementation architecture; Figure 3 A schematic diagram of the functional modules of the device for implementing causal masking in large-scale model inference in this application; Figure 4 This is a schematic diagram of the hardware structure of the device for implementing causal masking in large-scale model inference in this application. Detailed Implementation
[0017] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.
[0018] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.
[0019] In a first aspect, embodiments of this application provide a method for implementing causal masking in large model inference. Based on a set hardware implementation architecture, causal masking can be implemented without pre-constructing a two-dimensional matrix or performing addition or multiplication operations on the matrix unit.
[0020] In one embodiment, reference is made to Figure 1 , Figure 1 This is a flowchart illustrating the method for implementing causal masking in large-scale model inference in this application. Figure 1 As shown, methods for implementing causal masks in large model inference include: S1: After completing a vector-matrix operation, VMPE sends the result to SFU; S2: SFU receives cmd commands sent by FW to obtain input data; S3: SFU processes the acquired input data based on a preset bitwise operation mask operation and then performs a Softmax operation; The hardware architecture includes a VMPE responsible for vector-matrix operations, an SFU responsible for other vector operations (including nonlinear functions such as Softmax and rmsnorm), and a FW running on a RISC-V CPU to control the DMA, VMPE, and SFU. The FW running on the RISC-V CPU controls the DMA, VMPE, and SFU through a command queue.
[0021] See Figure 2 The diagram shows the hardware implementation architecture of this application. The VMPE and SFU are connected via VMPE2SFU_FIFO and SFU2VMPE_FIFO, and a ShareRAM is also provided between the VMPE and SFU. VMPE2SFU_FIFO serves as the channel for the VMPE to send data to the SFU, and SFU2VMPE_FIFO serves as the channel for the SFU to send data to the VMPE. The DMA is located between the RISC-V CPU and the ShareRAM; the RISC-V CPU connects the VMPE and SFU.
[0022] Among them, VMPE stands for Vector-Matrix Processing Engine; SFU stands for Special Function Unit; ShareRAM stands for Shared SRAM; DMA stands for Direct Memory Access; RISC-V CPU stands for CPU based on the RISC-V instruction set; RISC-V stands for Reduced Instruction Set Computing 5; and FW stands for Firmware.
[0023] Furthermore, in one embodiment, after completing a vector-matrix operation, the VMPE sends the result to the SFU, specifically including: After completing a vector-matrix operation, VMPE sends the result to SFU via VMPE2SFU_FIFO or via ShareRAM.
[0024] Vector-matrix operations, which multiply a vector by a matrix, are operations where a vector (1, 64) represents a vector with 1 row and 64 columns, and (64, 64) represents a matrix with 64 rows and 64 columns. The vector-matrix multiplication is (1, 64) × (64, 64), and the output is also a vector of size (1, 64). The calculation process involves multiplying and adding each column of the vector (1, 64) to the matrix. For example, if the vector is represented by (a0, a1, …, a…), the matrix multiplication is performed on each column of the matrix. 63 ) represents the vector whose 0th column is represented by (b 0,0 , b 1,0 , …, b 63,0 The product result is (c0, c1, ..., c). 63 If c0 = a0 × b, then c0 = a0 × b 0,0 + a1×b 1,0 + … + a 63 ×b 63,0 The rest c1~c 63 analogy.
[0025] Furthermore, in one embodiment, the SFU receives a cmd command sent by the FW to obtain input data, specifically including: S201: The FW sends a cmd command to the SFU, and the cmd command sent by the FW to the SFU specifies the input source with input data; S202: The SFU receives the cmd command sent by the FW and obtains input data from the input source. Specifically, the cmd command sent by the FW to the SFU specifies the input source. After receiving the cmd command, the SFU obtains the input data from the input source and performs corresponding processing.
[0026] The input source is ShareRAM or VMPE2SFU_FIFO, the input data is the Q / K matrix after position encoding, and the cmd command is a cmd format command related to causality masking.
[0027] Specifically, the input source is explained as follows.
[0028] As shown in the command format in Table 1 below, in_sel[3:0] is used to indicate the source of input data (i.e., input data), and is defined as follows: 4'h0, ShareRAM; 4'h1, FIFO (i.e., through the VMPE2SFU_FIFO channel), that is, the input source of a certain cmd command is ShareRAM or VMPE2SFU_FIFO.
[0029] Table 1
[0030] In Table 1, `token_position_base` represents the base address of the token position, counting from 0; `o_sram_addr` represents the SRAM address where the output data will be stored; `reserved` indicates that the data is reserved; `i_vec_addr` represents the address of the input vector; `vec_data_type` represents the vector data type; `element_num` represents the number of elements in a vector; `vec_num` represents the number of vectors that the command will process; `valid` indicates that the command is invalid; `busy` indicates that the hardware is processing the command and is in a busy state; `done` indicates that the hardware has completed the processing of the command; `tags` represents tags / markers used to track commands sent by the software and completed by the hardware, similar to a command number; `phase` represents the phase, with large model inference divided into two phases: Prefill and Decode; `in_sel` represents the input data source selection; `out_sel` represents the output data destination selection; `opcode` represents the operation type encoding; and `Softmax` represents the activation function.
[0031] The input data refers to the data to be processed, such as the data after embedded encoding and various intermediate processing results; for the causal masking processing involved in this application, the input data refers to the data after the Q / K matrix has undergone position encoding processing.
[0032] SFU can handle multiple functions, such as normalization (RMSNorm), positional encoding (RoPE), activation function (Softmax), etc.; the causal masking processing involved in this application refers to Softmax + causal masking processing.
[0033] In this application, FW is a cmd format command sent to SFU and related to causality masking. SFU can handle multiple functions, and the command formats corresponding to these functions are the same or similar, as shown in Table 1. However, different functions are represented by different values of opcode[7:0]. After receiving a command, SFU first checks the opcode field, determines the function to be processed based on the field, and then parses the remaining fields of the command, that is, sends opcode[7:0]=Softmax, and sets the token_position_base and phase fields to process the specific command of causality masking.
[0034] In this application, there are two fields related to the causality mask: phase and token_position_base. The phase is either Prefill or Decode, and the causality mask is only valid in the Prefill phase. No causality mask is needed in the Decode phase. The token_position_base corresponds to the starting position of a set of input tokens, that is, the position number of the first token in the complete input, starting from 0.
[0035] In this application, after receiving the cmd command, the SFU first processes the acquired input data through a preset bitwise operation mask operation, and then performs the subsequent Softmax operation.
[0036] The default bitwise mask operation is specifically represented as follows: vector_out = vector_in&((1024'h1<<((token_position_base+1)<<3))-1) Where `token_position_base` represents the token position base address, counting from 0; `token_position_base+1` means incrementing `token_position_base` by 1; `(token_position_base+1)<<3` means shifting the result of `(token_position_base+1)` 3 bits to the left, equivalent to multiplying by 8; `1024'h1<<((token_position_base+1)<<3)` means shifting 1 to the left by the number of bits in the result of `((token_position_base+1)<<3)`, where `(1024'h1<<((token_position_base+1)<<3)` represents the position base address of the token, counting from 0; `token_position_base+1` means incrementing `token_position_base` by 1; `(1024'h1<<((token_position_base+1)<<3)` means shifting 1 to the left by the number of bits in the result of `((token_position_base+1)<<3)`. h1<<((token_position_base+1)<<3))-1 means subtracting 1 from the result of (1024'h1<<((token_position_base+1)<<3)). vector_in&((1024'h1<<((token_position_base+1)<<3))-1) means performing a bitwise AND operation between vector_in and the result of ((1024'h1<<((token_position_base+1)<<3))-1). vector_in represents the vector of input data, and vector_out represents the vector of the calculated result.
[0037] For example, the input data consists of 128 elements, each of which is an INT8 (8-bit) value, processed using a preset bitwise operation mask. In practical applications, the specific number of elements is specified by the `vec_num[7:0]` field of the `cmd` command. INT8 represents a data type, i.e., integer; 8-bit represents 8 bits. The output effect can be seen as follows: Position = 0→output only first element, [0]; others mask to 0 Position = 1→output two element, [1:0]; others mask to 0 Position = 2→output three element, [2:0]; others mask to 0 etc.
[0038] The following is a detailed explanation of the output effect.
[0039] Position refers to token_position_base. Substituting the values of Position into the preset bitwise mask operation, when Position=0, token_position_base+1=1, shifting left by 3 bits is equivalent to multiplying by 8, resulting in 8. Shifting 1024'h1 left by 8 bits gives 1024'h100. 1024'h100-1=1024'hFF. A bitwise AND operation is performed between vector_in and 1024'hFF. Since only the lower 8 bits (FF) are 1, and the rest are 0, the result of the bitwise AND is that only the lower 8 bits of vector_in are output, and the other bits become 0, i.e., output only the first element (each element is 8 bits).
[0040] When Position=1, token_position_base+1=2, shifting left by 3 bits is equivalent to multiplying by 8, resulting in 16. 1024'h1 shifted left by 8 bits becomes 1024'h10000. 1024'h10000-1=1024'hFFFF. A bitwise AND operation is performed between vector_in and 1024'hFFFF. Since only the lower 16 bits (FFFF) are 1 and the rest are 0, the result of the bitwise AND operation is that only the lower 16 bits of vector_in are output, and the other bits become 0, i.e., output two elements, only the first two elements [1:0] (each element is 8-bit).
[0041] The method for implementing causal masking in large model inference in this application embodiment does not require pre-constructing a mask matrix, thereby effectively saving storage space. Furthermore, subsequent processing is done in units of vectors. This application also processes causal masks in units of vectors. Once a vector input is obtained, subsequent operations can begin without waiting for all matrix-matrix multiplication operations to be completed before proceeding with subsequent operations.
[0042] Secondly, embodiments of this application also provide an apparatus for implementing causal masks in large-scale model inference.
[0043] In one embodiment, reference is made to Figure 3 , Figure 3 This is a schematic diagram of the functional modules of the device for implementing causal masking in large-scale model inference in this application. Figure 3 As shown, the device for implementing causal masking in large model inference includes: a sending module, an acquisition module, and an execution module.
[0044] The sending module is used to drive the VMPE to send the result of a vector-matrix operation to the SFU after completing the operation; the acquiring module is used to drive the SFU to receive the cmd command sent by the FW to obtain input data; the execution module is used to drive the SFU to process the obtained input data based on a preset bitwise operation mask operation and then perform a Softmax operation; wherein, the set hardware implementation architecture includes the VMPE responsible for processing vector-matrix operations, the SFU responsible for processing other vector operations, and the FW running on the RISC-V CPU for controlling the DMA, VMPE and SFU.
[0045] Thirdly, embodiments of this application provide a device for implementing causal masking in large model inference. The device for implementing causal masking in large model inference can be a personal computer (PC), a laptop computer, a server, or other device with data processing capabilities.
[0046] Reference Figure 4 , Figure 4 This is a schematic diagram of the hardware structure of a device for implementing causality masking in large model inference, as described in an embodiment of this application. In this embodiment, the device for implementing causality masking in large model inference may include a processor, a memory, a communication interface, and a communication bus.
[0047] The communication bus can be of any type and is used to interconnect the processor, memory, and communication interface.
[0048] Communication interfaces include input / output (I / O) interfaces, physical interfaces, and logical interfaces used for interconnecting internal devices within a device that implements causal masking in large model inference, as well as interfaces used for interconnecting the device implementing causal masking with other devices (such as other computing devices or user equipment). Physical interfaces can be Ethernet interfaces, fiber optic interfaces, ATM interfaces, etc.; user equipment can be displays, keyboards, etc.
[0049] Memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.
[0050] The processor can be a general-purpose processor, which can call a program stored in memory that implements causal masking in large model inference and execute the method for implementing causal masking in large model inference provided in the embodiments of this application. For example, the general-purpose processor can be a central processing unit (CPU). The method executed when the program for implementing causal masking in large model inference is called can refer to the various embodiments of the method for implementing causal masking in large model inference in this application, and will not be repeated here.
[0051] Those skilled in the art will understand that Figure 4 The hardware structure shown does not constitute a limitation of this application and may include more or fewer components than shown, or combine certain components, or have different component arrangements.
[0052] Fourthly, embodiments of this application also provide a computer-readable storage medium.
[0053] This application contains a computer-readable storage medium storing a program for implementing causal masking in large model inference, wherein when the program for implementing causal masking in large model inference is executed by a processor, it implements the steps of the method for implementing causal masking in large model inference as described above.
[0054] The method implemented when the program for implementing causal masking in large model inference is executed can be referred to in the various embodiments of the method for implementing causal masking in large model inference in this application, and will not be repeated here.
[0055] The terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus. The terms "first," "second," and "third," etc., are used to distinguish different objects, etc., and do not indicate a sequence, nor do they limit "first," "second," and "third" to different types.
[0056] In the description of the embodiments of this application, terms such as "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a concrete manner.
[0057] In the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of this application, "multiple" means two or more.
[0058] In some processes described in the embodiments of this application, multiple operations or steps are included in a specific order. However, it should be understood that these operations or steps may not be executed in the order they appear in the embodiments of this application, or they may be executed in parallel. The sequence number of the operation is only used to distinguish different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed sequentially or in parallel, and these operations or steps may be combined.
[0059] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device to execute the methods described in the various embodiments of this application.
[0060] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.
Claims
1. A method for implementing causal masking in large-scale model inference, based on a defined hardware implementation architecture, characterized in that, The methods for implementing causal masks in large model inference include: After completing a vector-matrix operation, the VMPE sends the result to the Special Function Unit (SFU). The Special Function Unit (SFU) receives cmd commands sent by the firmware (FW) to obtain input data; The Special Function Unit (SFU) processes the acquired input data based on a preset bitwise operation mask and then performs a Softmax operation. The hardware implementation architecture includes a VMPE responsible for handling vector-matrix operations, a Special Function Unit (SFU) responsible for handling other vector operations, and firmware (FW) running on a RISC-V CPU to control the operation of Direct Memory Access (DMA), VMPE, and SFU.
2. The method for implementing causal masking in large model inference as described in claim 1, characterized in that: The VMPE and the Special Function Unit (SFU) are connected through VMPE2SFU_FIFO and SFU2VMPE_FIFO, and a ShareRAM is also provided between the VMPE and the SFU. The VMPE2SFU_FIFO is the channel through which the VMPE sends data to the Special Function Unit (SFU), and the SFU2VMPE_FIFO is the channel through which the SFU sends data to the VMPE.
3. The method for implementing causal masking in large model inference as described in claim 2, characterized in that: The direct memory access DMA is located between the RISC-V CPU and ShareRAM; The RISC-V CPU is connected to VMPE and Special Function Unit (SFU).
4. The method for implementing causal masking in large model inference as described in claim 3, characterized in that, After completing a vector-matrix operation, the VMPE sends the result to the Special Function Unit (SFU), specifically including: After completing a vector-matrix operation, VMPE sends the result to the Special Function Unit (SFU) via VMPE2SFU_FIFO or via ShareRAM.
5. The method for implementing causal masking in large model inference as described in claim 3, characterized in that, The Special Function Unit (SFU) receives cmd commands sent by the firmware (FW) to obtain input data, specifically including: The firmware (FW) sends a cmd command to the special function unit (SFU), and the cmd command sent by the firmware (FW) to the special function unit (SFU) specifies the input source containing the input data. The Special Function Unit (SFU) receives cmd commands sent by the firmware (FW) and obtains input data from the input source. The input source is ShareRAM or VMPE2SFU_FIFO, the input data is the Q / K matrix after position encoding, and the cmd command is a cmd format command related to causality masking.
6. The method for implementing causal masking in large model inference as described in claim 5, characterized in that: The two fields related to the causality mask are phase and token_position_base; The phase is either Prefill or Decode, and the causality mask is only valid in the Prefill phase; no causality mask is needed in the Decode phase. The token_position_base corresponds to the starting position of a set of input tokens, that is, the position number of the first token in the complete input, starting from 0.
7. The method for implementing causal masking in large model inference as described in claim 1, characterized in that: The preset bitwise operation mask operation is specifically represented as follows: vector_out = vector_in&((1024'h1<<((token_position_base+1)<<3))-1) Where `token_position_base` represents the token position base address, counting from 0; `token_position_base+1` means incrementing `token_position_base` by 1; `(token_position_base+1)<<3` means shifting the result of `(token_position_base+1)` 3 bits to the left, equivalent to multiplying by 8; `1024'h1<<((token_position_base+1)<<3)` means shifting 1 to the left by the number of bits in the result of `((token_position_base+1)<<3)`, where `(1024'h1<<((token_position_base+1)<<3)` represents the position base address of the token, counting from 0; `token_position_base+1` means incrementing `token_position_base` by 1; `(1024'h1<<((token_position_base+1)<<3)` means shifting 1 to the left by the number of bits in the result of `((token_position_base+1)<<3)`. h1<<((token_position_base+1)<<3))-1 means subtracting 1 from the result of (1024'h1<<((token_position_base+1)<<3)). vector_in&((1024'h1<<((token_position_base+1)<<3))-1) means performing a bitwise AND operation between vector_in and the result of ((1024'h1<<((token_position_base+1)<<3))-1). vector_in represents the vector of input data, and vector_out represents the vector of the calculated result.
8. An apparatus for implementing causal masking in large-scale model inference, characterized in that, The apparatus for implementing causal masking in large model inference includes: The sending module is used to drive the VMPE to send the result of a vector-matrix operation to the special function unit SFU after completing the operation. The acquisition module is used to drive the special function unit (SFU) to receive cmd commands sent by the firmware (FW) to obtain input data. The execution module is used to drive the special function unit (SFU) to process the acquired input data based on a preset bitwise operation mask operation and then perform a Softmax operation. The hardware implementation architecture includes a VMPE responsible for handling vector-matrix operations, a Special Function Unit (SFU) responsible for handling other vector operations, and firmware (FW) running on a RISC-V CPU to control the operation of Direct Memory Access (DMA), VMPE, and SFU.
9. A device for implementing causal masking in large-scale model inference, characterized in that, The device for implementing causality masking in large model inference includes a processor, a memory, and a program for implementing causality masking in large model inference stored in the memory and executable by the processor, wherein when the program for implementing causality masking in large model inference is executed by the processor, it implements the steps of the method for implementing causality masking in large model inference as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program for implementing causal masking in large model inference, wherein when the program for implementing causal masking in large model inference is executed by a processor, it implements the steps of the method for implementing causal masking in large model inference as described in any one of claims 1 to 7.