A data layout bridging method and device for a band-sparse matrix and a medium

Abstract

The application discloses a data layout bridging processing method and device of a strip sparse matrix and a medium. A general matrix multiplication and addition operation unit is used to perform first matrix multiplication on a query matrix and a key matrix to generate a current intermediate accumulation result. In an offline stage, current sparse position indication information is determined in a metadata table, and current effective data is extracted from the intermediate accumulation result. In the loading of value matrix data, the path of the value matrix data is rearranged and folded into address conversion calculation, so that the value matrix data is arranged in a second data layout format, and the processing range is reduced by completing the extraction before nonlinear activation operation. According to the current effective data and the value matrix data, a hardware-level structured sparse arithmetic unit is used to perform second matrix multiplication to obtain a target layout bridging attention output result. The application solves the problem of additional register rearrangement and control logic overhead caused by layout mismatch, reduces memory access and hardware resource overhead, and improves runtime execution efficiency.

Description

Technical Field

[0001] This invention relates to the field of strip sparse matrix processing technology, and in particular to a data layout bridging processing method, apparatus and medium for strip sparse matrices. Background Technology

[0002] In sparse attention scenarios, even if the general strip pattern has been mapped to a structured sparse pattern at the algorithm level, a critical operator implementation problem still exists at runtime. The query matrix and key matrix calculation stages are usually executed by ordinary matrix multiplication and addition arithmetic logic units, and their accumulated results are in a normal type block layout. However, in the value matrix data calculation stage, if it is to be executed by sparse arithmetic logic units, the input must satisfy a different type of block layout, and metadata and selector information are also required.

[0003] In the process of developing this invention, the inventors discovered the following drawbacks in the prior art: If the prior art explicitly constructs the complete replacement result directly at runtime, it leads to additional register rearrangement, shared memory intermediate copies, explicit shuffling, and control logic overhead in both stages due to layout mismatch. Similarly, if metadata relies on online generation, it also significantly increases the runtime path weight. Summary of the Invention

[0004] This invention provides a data layout bridging processing method, apparatus and medium for striped sparse matrices, which reduces memory access and hardware resource overhead and improves runtime efficiency.

[0005] According to one aspect of the present invention, a data layout bridging processing method for a striped sparse matrix is ​​provided, comprising: The query matrix and the key matrix are multiplied by a general matrix multiplication and addition unit to generate the current intermediate accumulation result; wherein the current intermediate accumulation result is stored in a first data layout format. The current sparse position indication information is determined in the pre-built offline stage metadata table, and the current valid data is extracted from the current intermediate accumulation result according to the current sparse position indication information; wherein, the current valid data is rearranged in the second data layout format; During the loading of value matrix data, the path of the value matrix data is rearranged and folded into the preset address transformation calculation, so that the loaded value matrix data is arranged in the second data layout format, and the extraction is completed before the nonlinear activation operation to reduce its processing scope. Based on the current valid data and the value matrix data, a second matrix multiplication is performed through a preset hardware-level structured sparse arithmetic unit to obtain the target layout bridging attention output result.

[0006] According to another aspect of the present invention, a data layout bridging processing apparatus for a striped sparse matrix is ​​provided, comprising: The current intermediate accumulation result generation module is used to perform a first matrix multiplication on the query matrix and the key matrix through a general matrix multiplication and addition operation unit to generate the current intermediate accumulation result; wherein, the current intermediate accumulation result is stored in a first data layout format; The current valid data extraction module is used to determine the current sparse position indication information in the pre-built offline stage metadata table, and extract the current valid data from the current intermediate accumulation result according to the current sparse position indication information; wherein, the current valid data is rearranged in the second data layout format; The value matrix data layout format arrangement module is used to rearrange and fold the path of the value matrix data into the preset address transformation calculation during the loading process of the value matrix data, so that the loaded value matrix data is arranged in a second data layout format, and the extraction is completed before the nonlinear activation operation to reduce its processing range. The target layout bridging attention output result determination module is used to perform a second matrix multiplication through a preset hardware-level structured sparse arithmetic unit based on the current valid data and the value matrix data to obtain the target layout bridging attention output result.

[0007] According to another aspect of the present invention, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the data layout bridging processing method for a striped sparse matrix as described in any embodiment of the present invention.

[0008] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the data layout bridging processing method for a striped sparse matrix as described in any embodiment of the present invention.

[0009] The technical solution of this invention performs a first matrix multiplication on the query matrix and the key matrix using a general matrix multiplication and addition unit to generate the current intermediate accumulation result; determines the current sparse position indication information in a pre-built offline stage metadata table, and extracts the current valid data from the current intermediate accumulation result based on the current sparse position indication information; during the loading process of the value matrix data, the path of the value matrix data is rearranged and folded into a preset address transformation calculation, so that the loaded value matrix data is arranged in a second data layout format, and the extraction is completed before the nonlinear activation operation to reduce its processing range; based on the current valid data and the value matrix data, a second matrix multiplication is performed through a preset hardware-level structured sparse arithmetic unit to obtain the target layout bridging attention output result; this solves the problem of additional register rearrangement and control logic overhead caused by layout mismatch, reduces memory access and hardware resource overhead, can eliminate data layout conversion bottlenecks, achieve seamless operator fusion, and improve runtime execution efficiency.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart of a data layout bridging processing method for a striped sparse matrix according to Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of the structure of a data layout bridging processing device for a strip sparse matrix according to Embodiment 2 of the present invention; Figure 3 This is a schematic diagram of the structure of an electronic device provided according to Embodiment 3 of the present invention. Detailed Implementation

[0013] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0014] It should be noted that the terms "target," "current," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0015] It is worth noting that the information collected in the technical solution of this application is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data all comply with the relevant laws, regulations and standards of the relevant countries and regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse; if the user chooses to refuse, the process will proceed to the expert decision-making process.

[0016] Example 1 Figure 1 The flowchart of the data layout bridging processing method for a striped sparse matrix provided in Embodiment 1 of the present invention is applicable to scenarios where the layout of the attention calculation query matrix and key matrix calculation stages and the value matrix data calculation stages in a striped sparse matrix is ​​mismatched, and data layout bridging processing is performed. This method can be executed by a data layout bridging processing device for a striped sparse matrix, which can be implemented in hardware and / or software.

[0017] Correspondingly, such as Figure 1 As shown, the method includes: S110. Perform the first matrix multiplication on the query matrix and the key matrix through the general matrix multiplication and addition operation unit to generate the current intermediate accumulation result.

[0018] The current intermediate cumulative result is stored in the first data layout format.

[0019] The first data layout format is a layout format that satisfies the calculation of the general matrix multiplication and addition operation unit.

[0020] This embodiment primarily addresses the challenges faced by sparse attention mechanisms during operator implementation, which can be achieved based on layout bridging and low-transfer. For the system corresponding to the data layout bridging processing method, the overall input is generated by performing a first matrix multiplication on the query matrix and the key matrix using a general matrix multiplication and addition unit, generating the current intermediate accumulated result.

[0021] Specifically, a first matrix multiplication is performed on the query matrix and the key matrix to generate the current intermediate accumulated result, which is stored in a first data layout format. However, for computation with the subsequent value matrix data calculation stage, a second data layout format must be satisfied. This second data layout format is a layout format that satisfies the requirements of hardware-level structured sparse arithmetic units for computation.

[0022] Therefore, even if the algorithm layer has obtained a structured sparse pattern, the intermediate accumulation result cannot be directly used as the input of the hardware-level structured sparse arithmetic unit. A special layout bridge needs to be established between the two.

[0023] S120. Determine the current sparse position indication information in the pre-built offline stage metadata table, and extract the current valid data from the current intermediate accumulation result based on the current sparse position indication information.

[0024] The currently valid data is rearranged according to the second data layout format.

[0025] Among them, the current sparse location indication information is the indication information of the sparse location determined in the metadata table during the offline stage.

[0026] Optionally, before determining the current sparse location indication information in the pre-built offline stage metadata table and extracting the current valid data from the current intermediate accumulation result based on the current sparse location indication information, the method further includes: in the offline stage, calculating the corresponding reserved position and block type in each local group based on the strip mask range, block parameters and sparse mode offline encoding, determining them as sparse location indication information, and generating the offline stage metadata table.

[0027] In this embodiment, in order to address the issue that online metadata generation significantly increases the runtime path, it is also necessary to build an offline stage metadata table in advance.

[0028] Specifically, in the offline phase, based on the strip mask range, block parameters, and sparse mode offline encoding, the corresponding reserved position and block type in each local group are calculated, which are then determined as sparse position indication information, and an offline phase metadata table is generated.

[0029] In detail, during the offline phase, the selection information for the positions to be retained in the pre-acquired local strip groups, as well as the corresponding block types, can be pre-calculated based on information such as the strip mask range and block parameters. Next, the selection information needs to be encoded into packet metadata, which generates corresponding sparse position indication information and an offline phase metadata table.

[0030] Furthermore, by using the runtime kernel to look up the corresponding packaged metadata based on the chunk type, position, or iteration order, the runtime path can be simplified into a pipeline of layout transformation, metadata lookup, and sparse execution.

[0031] The advantage of this setup is that by generating the metadata table offline during the offline phase, the drawback of metadata relying on online generation, which would significantly increase the runtime path, is resolved. This simplifies the runtime control logic, reduces branch judgments and address calculation instructions, making the runtime path lighter and improving runtime execution efficiency.

[0032] Optionally, the step of extracting the current valid data from the current intermediate accumulation result based on the current sparse position indication information includes: obtaining and determining the current sparse position indication information in a pre-built offline stage metadata table based on the current retention position and the current block type; and performing an extraction operation in the output fragment layout corresponding to the current intermediate accumulation result based on the current sparse position indication information and the obtained current metadata requirements to obtain the current valid data.

[0033] Specifically, the rearrangement of the currently valid data in the second data layout format includes: the rearrangement of the currently valid data in the second data layout format is completed through in-thread register reorganization, so as to complete the layout transformation logic without introducing additional cross-thread communication.

[0034] In this embodiment, based on the current retention position and the current block type, the current sparse position indication information is determined in a pre-built offline stage metadata table. This allows the determination of the current sparse position indication information corresponding to the current intermediate accumulation result. Furthermore, the output fragment layout is extracted using the current sparse position indication information and the obtained current metadata requirements to obtain the currently valid data.

[0035] Furthermore, the current valid data is rearranged in the second data layout format, which is accomplished through in-thread register reorganization rather than through cross-thread data exchange, so as to complete the layout transformation logic without introducing additional cross-thread communication.

[0036] In this embodiment, the current valid data obtained in this way is rearranged in the second data layout format, so that the current valid data is converted into a consumable arrangement of input fragments of the hardware-level structured sparse arithmetic unit; and instead of generating the intermediate result after complete permutation first and then extracting, the valid value to be handed over to the hardware-level structured sparse arithmetic unit is directly extracted from the output fragment layout of the original intermediate accumulation result according to the target input fragment and metadata requirements.

[0037] Additionally, the hardware-level structured sparse arithmetic unit is a hardware arithmetic unit that supports structured sparse computation; the general matrix multiplication and addition unit is a standard matrix multiplication hardware arithmetic unit.

[0038] S130. During the loading of the value matrix data, the path of the value matrix data is rearranged and folded into the preset address transformation calculation, so that the loaded value matrix data is arranged in the second data layout format, and the extraction is completed before the nonlinear activation operation to reduce its processing range.

[0039] Optionally, the step of rearranging and folding the path of the value matrix data into a preset address translation calculation, so that the loaded value matrix data is arranged in a second data layout format, includes: rearranging and folding the path of the value matrix data into an address translation calculation from shared memory to registers, so that each thread obtains data in the order that the hardware-level structured sparse arithmetic unit can consume during the loading process of the value matrix data; wherein, the address translation calculation is implemented through address calculation of the matrix loading instruction; the address translation calculation is determined based on a preset address offset at compile time.

[0040] In this embodiment, for value matrix data, it is not necessary to generate an additional fully rearranged value matrix in shared memory, nor is it necessary to explicitly perform a complete data exchange in the register. Instead, the path rearrangement of the value matrix data is folded into the address translation calculation from shared memory to the register.

[0041] Specifically, address translation calculation is implemented through address calculation of matrix loading instructions, and the address translation calculation is determined based on compile-time address offsets. Furthermore, corresponding address translation operations can be performed using address offsets; this ensures that each thread receives data in a hardware-level structured sparse arithmetic unit's consuming order during the loading of value matrix data, reducing the pressure on thread hardware calls and bandwidth.

[0042] S140. Based on the current valid data and the value matrix data, perform a second matrix multiplication through a preset hardware-level structured sparse arithmetic unit to obtain the target layout bridging attention output result.

[0043] Optionally, the step of performing a second matrix multiplication using a preset hardware-level structured sparse arithmetic unit based on the current valid data and the value matrix data to obtain the target layout bridging attention output result includes: processing the current valid data with a normalization function to obtain current normalized valid data; performing a second matrix multiplication calculation on the current normalized valid data and the value matrix data using the hardware-level structured sparse arithmetic unit to obtain the layout bridging attention output result at the current time; and obtaining and summing the layout bridging attention output result from the previous time step with the layout bridging attention output result at the current time step to obtain the target layout bridging attention output result.

[0044] In this embodiment, the currently valid data can be set to It is necessary to check the currently valid data. After performing normalization function processing, the current normalized effective data is obtained, that is, through the formula... To calculate the current normalized valid data The extraction of currently valid data needs to be completed before the normalization function is applied, in order to narrow the processing range of the normalization function.

[0045] Furthermore, through a hardware-level structured sparse arithmetic unit, a second matrix multiplication is performed on the current normalized valid data and value matrix data. Since both the current normalized valid data and value matrix data are converted into the second data layout format, they can be directly used by the hardware-level structured sparse arithmetic unit. Therefore, the layout bridging attention output result at the current time step is obtained, assuming it is... .

[0046] Accordingly, the bridging attention output result is obtained and configured based on the previous time step. This is summed with the current layout bridging attention output to obtain the target layout bridging attention output, which is... .

[0047] The technical solution of this invention involves performing a first matrix multiplication on the query matrix and the key matrix using a general matrix multiplication and addition unit to generate the current intermediate accumulation result; determining the current sparse position indication information in a pre-built offline stage metadata table, and extracting the current valid data from the current intermediate accumulation result based on the current sparse position indication information; during the loading process of the value matrix data, the path of the value matrix data is rearranged and folded into a preset address transformation calculation, so that the loaded value matrix data is arranged in a second data layout format, and the extraction is completed before the nonlinear activation operation to reduce its processing scope; based on the current valid data... The value matrix data is used to perform a second matrix multiplication through a preset hardware-level structured sparse arithmetic unit to obtain the target layout bridging attention output result. This solves the problem of additional register rearrangement and control logic overhead caused by layout mismatch, and realizes low-overhead connection between the general matrix multiplication and addition unit and the hardware-level structured sparse arithmetic unit in the sparse attention computing kernel of the striped sparse matrix. The additional data movement overhead caused by explicit substitution is avoided by extracting fusion and address transformation. It reduces memory access, eliminates the data layout transformation bottleneck, realizes seamless operator fusion, and improves runtime execution efficiency.

[0048] Example 2 Figure 2 This is a schematic diagram of a data layout bridging processing device for a striped sparse matrix provided in Embodiment 2 of the present invention. The data layout bridging processing device for a striped sparse matrix provided in this embodiment can be implemented by software and / or hardware, and can be configured in a terminal device or server to implement a data layout bridging processing method for a striped sparse matrix according to an embodiment of the present invention. Figure 2 As shown, the device includes: a current intermediate accumulation result generation module 210, a current valid data extraction module 220, a value matrix data layout format arrangement module 230, and a target layout bridging attention output result determination module 240.

[0049] The current intermediate accumulation result generation module 210 is used to perform a first matrix multiplication on the query matrix and the key matrix through a general matrix multiplication and addition operation unit to generate the current intermediate accumulation result; wherein the current intermediate accumulation result is stored in a first data layout format; The current valid data extraction module 220 is used to determine the current sparse position indication information in the pre-built offline stage metadata table, and extract the current valid data from the current intermediate accumulation result according to the current sparse position indication information; wherein, the current valid data is rearranged in the second data layout format; The value matrix data layout format arrangement module 230 is used to rearrange and fold the path of the value matrix data into the preset address transformation calculation during the loading process of the value matrix data, so that the loaded value matrix data is arranged in the second data layout format, and the extraction is completed before the nonlinear activation operation to reduce its processing range. The target layout bridging attention output result determination module 240 is used to perform a second matrix multiplication through a preset hardware-level structured sparse arithmetic unit based on the current valid data and the value matrix data to obtain the target layout bridging attention output result.

[0050] The technical solution of this invention performs a first matrix multiplication on the query matrix and the key matrix using a general matrix multiplication and addition unit to generate the current intermediate accumulation result; determines the current sparse position indication information in a pre-built offline stage metadata table, and extracts the current valid data from the current intermediate accumulation result based on the current sparse position indication information; during the loading process of the value matrix data, the path of the value matrix data is rearranged and folded into a preset address transformation calculation, so that the loaded value matrix data is arranged in a second data layout format, and the extraction is completed before the nonlinear activation operation to reduce its processing range; based on the current valid data and the value matrix data, a second matrix multiplication is performed through a preset hardware-level structured sparse arithmetic unit to obtain the target layout bridging attention output result; this solves the problem of additional register rearrangement and control logic overhead caused by layout mismatch, reduces memory access and hardware resource overhead, can eliminate data layout conversion bottlenecks, achieve seamless operator fusion, and improve runtime execution efficiency.

[0051] Based on the above embodiments, an offline stage metadata table generation module is also included, which can be specifically used to: before determining the current sparse position indication information in the pre-built offline stage metadata table and extracting the current valid data from the current intermediate accumulation result according to the current sparse position indication information, in the offline stage, calculate the corresponding reserved position and block type in each local group according to the strip mask range, block parameters and sparse mode offline encoding, determine it as sparse position indication information, and generate the offline stage metadata table.

[0052] Based on the above embodiments, the current valid data extraction module 220 can be specifically used to: obtain and determine the current sparse position indication information in the pre-built offline stage metadata table according to the current retention position and the current block type; and perform extraction operations in the output fragment layout corresponding to the current intermediate accumulation result according to the current sparse position indication information and the obtained current metadata requirements to obtain the current valid data.

[0053] Based on the above embodiments, the value matrix data layout format arrangement module 230 can be specifically used to: rearrange and fold the path of the value matrix data into the address translation calculation from shared memory to register, so that each thread obtains the data in the order that the hardware-level structured sparse arithmetic unit can consume during the loading process of the value matrix data; wherein, the address translation calculation is implemented through the address calculation of the matrix loading instruction; the address translation calculation is determined based on the address offset preset at compile time.

[0054] Based on the above embodiments, the current valid data extraction module 220 can be specifically used to: rearrange the current valid data in the second data layout format, which is completed by reorganizing the registers within the thread, so as to complete the layout transformation logic without introducing additional cross-thread communication.

[0055] Based on the above embodiments, the target layout bridging attention output result determination module 240 can be specifically used to: perform normalization function processing on the current valid data to obtain the current normalized valid data; perform a second matrix multiplication calculation on the current normalized valid data and the value matrix data through the hardware-level structured sparse arithmetic unit to obtain the layout bridging attention output result at the current time; obtain and sum the layout bridging attention output result at the previous time with the layout bridging attention output result at the current time to obtain the target layout bridging attention output result.

[0056] Based on the above embodiments, it can be specifically used as follows: the hardware-level structured sparse arithmetic unit is a hardware arithmetic unit that supports structured sparse computation; the general matrix multiplication and addition unit is a standard matrix multiplication hardware arithmetic unit.

[0057] The data layout bridging processing device for striped sparse matrices provided in this embodiment of the invention can execute the data layout bridging processing method for striped sparse matrices provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method.

[0058] Example 3 Figure 3A schematic diagram of an electronic device 10, which can be used to implement Embodiment 4 of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0059] like Figure 3 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0060] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0061] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the data layout bridging processing method for striped sparse matrices.

[0062] In some embodiments, the data layout bridging processing method for a striped sparse matrix can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the data layout bridging processing method for a striped sparse matrix described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the data layout bridging processing method for a striped sparse matrix by any other suitable means (e.g., by means of firmware).

[0063] The method includes: performing a first matrix multiplication on a query matrix and a key matrix using a general matrix multiplication and addition unit to generate a current intermediate accumulation result; wherein the current intermediate accumulation result is stored in a first data layout format; determining current sparse position indication information in a pre-built offline stage metadata table, and extracting current valid data from the current intermediate accumulation result based on the current sparse position indication information; wherein the current valid data is rearranged in a second data layout format; during the loading of value matrix data, folding the path rearrangement of the value matrix data into a preset address transformation calculation, so that the loaded value matrix data is arranged in a second data layout format, and completing the extraction before the nonlinear activation operation to reduce its processing scope; and performing a second matrix multiplication using a preset hardware-level structured sparse arithmetic unit based on the current valid data and the value matrix data to obtain a target layout bridging attention output result.

[0064] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0065] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0066] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0067] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0068] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0069] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0070] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0071] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

[0072] Example 4 Embodiment 4 of the present invention also provides a computer-readable storage medium, wherein the computer-readable instructions, when executed by a computer processor, are used to perform a data layout bridging processing method for a striped sparse matrix. The method includes: performing a first matrix multiplication on a query matrix and a key matrix using a general matrix multiplication and addition unit to generate a current intermediate accumulation result; wherein the current intermediate accumulation result is stored in a first data layout format; determining current sparse position indication information in a pre-constructed offline stage metadata table, and extracting current valid data from the current intermediate accumulation result based on the current sparse position indication information; wherein the current valid data is rearranged in a second data layout format; during the loading process of the value matrix data, folding the path rearrangement of the value matrix data into a preset address transformation calculation, so that the loaded value matrix data is arranged in the second data layout format, and completing the extraction before the nonlinear activation operation to reduce its processing scope; and performing a second matrix multiplication using a preset hardware-level structured sparse arithmetic unit based on the current valid data and the value matrix data to obtain a target layout bridging attention output result.

[0073] Of course, the computer-executable instructions provided in the embodiments of the present invention, which include a computer-readable storage medium, are not limited to the method operations described above, but can also perform related operations in the data layout bridging processing method for striped sparse matrices provided in any embodiment of the present invention.

[0074] Based on the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0075] It is worth noting that in the embodiments of the above-mentioned data layout bridging processing method for striped sparse matrices, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the scope of protection of the present invention.

[0076] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for bridging the data layout of a striped sparse matrix, characterized in that, include: The query matrix and the key matrix are multiplied by a general matrix multiplication and addition unit to generate the current intermediate accumulation result; wherein the current intermediate accumulation result is stored in a first data layout format. The current sparse position indication information is determined in the pre-built offline stage metadata table, and the current valid data is extracted from the current intermediate accumulation result according to the current sparse position indication information; wherein, the current valid data is rearranged in the second data layout format; During the loading of value matrix data, the path of the value matrix data is rearranged and folded into the preset address transformation calculation, so that the loaded value matrix data is arranged in the second data layout format, and the extraction is completed before the nonlinear activation operation to reduce its processing scope. Based on the current valid data and the value matrix data, a second matrix multiplication is performed through a preset hardware-level structured sparse arithmetic unit to obtain the target layout bridging attention output result.

2. The method according to claim 1, characterized in that, Before determining the current sparse location indication information in the pre-built offline stage metadata table and extracting the current valid data from the current intermediate accumulation result based on the current sparse location indication information, the method further includes: In the offline phase, based on the strip mask range, block parameters, and sparse mode offline encoding, the corresponding reserved position and block type in each local group are calculated, which are then determined as sparse position indication information, and an offline phase metadata table is generated.

3. The method according to claim 2, characterized in that, The step of extracting currently valid data from the current intermediate accumulation result based on the current sparse position indication information includes: Obtain and determine the current sparse location indication information from the pre-built offline stage metadata table based on the current retention location and the current block type; In the output fragment layout corresponding to the current intermediate accumulation result, an extraction operation is performed based on the current sparse position indication information and the obtained current metadata requirements to obtain the current valid data.

4. The method according to claim 1, characterized in that, The step of rearranging and folding the path of the value matrix data into a preset address transformation calculation, so that the loaded value matrix data is arranged in a second data layout format, includes: The path rearrangement of the value matrix data is folded into the address translation calculation from shared memory to register, so that each thread obtains the data that can be consumed in the order of the hardware-level structured sparse arithmetic unit during the loading of the value matrix data. The address transformation calculation is implemented through address calculation of the matrix loading instruction; the address transformation calculation is determined based on the address offset preset at compile time.

5. The method according to claim 3, characterized in that, The currently valid data is rearranged according to the second data layout format, including: The current valid data is rearranged in the second data layout format, which is accomplished through in-thread register reorganization, so as to complete the layout transformation logic without introducing additional cross-thread communication.

6. The method according to claim 5 or 4, characterized in that, The step of performing a second matrix multiplication using a preset hardware-level structured sparse arithmetic unit based on the current valid data and the value matrix data to obtain the target layout bridging attention output result includes: The current valid data is processed by a normalization function to obtain the current normalized valid data; The hardware-level structured sparse arithmetic unit performs a second matrix multiplication on the current normalized valid data and the value matrix data to obtain the current time-state layout bridging attention output result. The layout bridging attention output result of the previous time step is obtained and summed with the layout bridging attention output result of the current time step to obtain the target layout bridging attention output result.

7. The method according to claim 1, 3, or 5, characterized in that, Also includes: The hardware-level structured sparse arithmetic unit is a hardware arithmetic unit that supports structured sparse computation. The general matrix multiplication and addition unit is a standard matrix multiplication hardware arithmetic unit.

8. A data layout bridging processing device for a striped sparse matrix, characterized in that, include: The current intermediate accumulation result generation module is used to perform a first matrix multiplication on the query matrix and the key matrix through a general matrix multiplication and addition operation unit to generate the current intermediate accumulation result; wherein, the current intermediate accumulation result is stored in a first data layout format; The current valid data extraction module is used to determine the current sparse position indication information in the pre-built offline stage metadata table, and extract the current valid data from the current intermediate accumulation result according to the current sparse position indication information; wherein, the current valid data is rearranged in the second data layout format; The value matrix data layout format arrangement module is used to rearrange and fold the path of the value matrix data into the preset address transformation calculation during the loading process of the value matrix data, so that the loaded value matrix data is arranged in a second data layout format, and the extraction is completed before the nonlinear activation operation to reduce its processing range. The target layout bridging attention output result determination module is used to perform a second matrix multiplication through a preset hardware-level structured sparse arithmetic unit based on the current valid data and the value matrix data to obtain the target layout bridging attention output result.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements a data layout bridging processing method for a striped sparse matrix as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute a data layout bridging processing method for a striped sparse matrix as described in any one of claims 1-7.

Images