Method and system for processing data in model and electronic equipment

Abstract

The invention discloses a method and system for processing data in a model and electronic equipment, and relates to the field of model technologies and machine learning. The method comprises the steps that parameter information of a model is read from an off-chip storage space, the parameter information is stored in a blocking mode in an unstructured sparse matrix format, and the model is obtained based on machine learning training; the parameter information of the model is decompressed and stored in a corresponding shared memory, and the decompressed parameter information is stored in a structured dense matrix format; based on the decompressed parameter information, output data matched with the input data is determined, and the output data is used for representing a dialogue result generated when the model processes the input data. The technical problem that the model parameter storage consumes much space is solved.

Description

Technical Field

[0001] The present application relates to the fields of large model technology and machine learning, and specifically, to a method, system and electronic device for processing data in a model. Background Art

[0002] At present, large machine learning models (abbreviated as large models) have become an important computing task on various computing platforms today, and have shown great value in scenarios such as dialogue systems. However, due to the huge number of parameters, large models often require huge storage space to store their parameter information. A single graphics processing unit (GPU) card often cannot store all model parameters. Therefore, large model parameter storage consumes a lot of space.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide a method, system and electronic device for processing data in a model, so as to at least solve the technical problem that the storage of model parameters consumes a lot of space.

[0005] According to one aspect of an embodiment of the present invention, a method for processing data in a large model is provided. The method may include: reading parameter information of the model from an off-chip storage space, wherein the parameter information is stored in blocks in an unstructured sparse matrix format, and the model is obtained based on machine learning training; decompressing the parameter information of the model and storing it in a corresponding shared memory, wherein the decompressed parameter information is stored in a structured dense matrix format; based on the decompressed parameter information, determining output data that matches the input data, wherein the output data is used to characterize a dialogue result generated by the model processing the input data.

[0006] According to one aspect of an embodiment of the present invention, another method for processing data in a large model is provided. The method may include: reading the parameter information of the model from the off-chip storage space by calling the first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter is the parameter information, the parameter information is stored in blocks in an unstructured sparse matrix format, and the model is obtained based on machine learning training; decompressing the parameter information of the model and storing it in a corresponding shared memory, wherein the decompressed parameter information is stored in a structured dense matrix format; based on the decompressed parameter information, determining output data matching the input data, wherein the output data is used to characterize the dialogue result generated by the model processing the input data; outputting the determined output data by calling the second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the output data.

[0007] According to one aspect of an embodiment of the present invention, another method for processing data in a large model is provided. The method may include: obtaining multimodal information input in an interactive interface, wherein the type of the multimodal information includes at least one of the following: text information containing character information, video frame information containing frame image information, and audio information; reading the parameter information of the model from an off-chip storage space, wherein the parameter information is stored in blocks in an unstructured sparse matrix format, and the model is obtained based on machine learning training; decompressing the parameter information of the model and storing it in a corresponding shared memory, wherein the decompressed parameter information is stored in a structured dense matrix format; based on the decompressed parameter information, determining reply information matching the multimodal information, wherein the reply information is used to characterize the reply content generated by the model processing the multimodal information, and the type of the reply information includes at least one of the following: text information, image information, and voice information; displaying the reply information in the interactive interface.

[0008] According to another aspect of an embodiment of the present invention, a system for processing data in a large model is also provided. The system may include: a thread block, which is used to read the parameter information of the model from an off-chip storage space and store the parameter information in a register, wherein the parameter information is stored in blocks in an unstructured sparse matrix format, and the model is obtained based on machine learning training; an on-chip memory, which decompresses and stores the parameter information of the model, wherein the decompressed parameter information is stored in a structured dense matrix format; a computing component, which is used to determine output data matching the input data based on the decompressed parameter information, wherein the output data is used to characterize the dialogue results generated by the model processing the input data.

[0009] According to another aspect of an embodiment of the present invention, an electronic device is also provided, which may include a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute computer-executable instructions. When the above-mentioned computer-executable instructions are executed by the processor, the above-mentioned method of any one of the above-mentioned items is implemented.

[0010] According to another aspect of an embodiment of the present invention, a processor is further provided, the processor being used to run a program, wherein any one of the above methods is executed when the program is running.

[0011] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium including a stored program, wherein when the program is executed, the device where the storage medium is located is controlled to execute any one of the above methods.

[0012] In an embodiment of the present invention, the parameter information of the model is read from an off-chip storage space, wherein the parameter information is stored in blocks in an unstructured sparse matrix format, and the model is obtained based on machine learning training; the parameter information of the model is decompressed and stored in a corresponding shared memory, wherein the decompressed parameter information is stored in a structured dense matrix format; based on the decompressed parameter information, output data matching the input data is determined, wherein the output data is used to characterize the dialogue results generated by the model processing the input data. That is, in this embodiment, the parameter information stored in an unstructured sparse matrix format is obtained, and after the parameter matrix is ​​decompressed and processed, the output data of the large model is obtained based on the decompressed parameter information and the input data. Since the memory space occupied by the matrix stored in an unstructured sparse manner is smaller, the technical effect of reducing the space consumed when storing the parameters of the large model is achieved, and the technical problem of the large space consumed by the model parameter storage is solved.

[0013] It is easy to notice that the above general description and the following detailed description are only for the purpose of exemplifying and explaining the present application, and do not constitute a limitation of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0015] Figure 1 is a schematic diagram of an application scenario of a method for processing data in a model according to an embodiment of the present application;

[0016] Figure 2 is a flow chart of a method for processing data in a model according to an embodiment of the present invention;

[0017] Figure 3 is a flowchart of a method for processing data in another model according to an embodiment of the present application;

[0018] Figure 4 is a schematic diagram of processing of data in a model by a computer device according to an embodiment of the present application;

[0019] FIG5( a ) is a flow chart of a method for processing data in another model according to an embodiment of the present application;

[0020] FIG5( b ) is a schematic diagram of a system for processing data in a model according to an embodiment of the present application;

[0021] Figure 6 is a flow chart of an end-to-end sparse matrix multiplication pipeline method based on a computing unit according to an embodiment of the present invention;

[0022] Figure 7 is a schematic diagram of block storage according to an embodiment of the present invention;

[0023] Figure 8 is a schematic diagram of a block offset array according to an embodiment of the present invention;

[0024] Fig. 9 is a schematic diagram of a non-zero element array according to an embodiment of the present invention;

[0025] Fig.10 is a schematic diagram of a tensor core calculation according to an embodiment of the present invention;

[0026] Fig.11 is a schematic diagram of a device for processing data in a model according to an embodiment of the present invention;

[0027] Fig.12 is a schematic diagram of a device for processing data in a model according to an embodiment of the present invention;

[0028] Fig.13 is a schematic diagram of a device for processing data in a model according to an embodiment of the present invention;

[0029] Fig.14 is a structural block diagram of a computer terminal according to an embodiment of the present invention;

[0030] Fig.15 The invention is a block diagram of an electronic device for a method for processing data in a model according to an embodiment of the invention. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0033] The technical solution provided in this application is mainly implemented by large-scale model technology. The large model here refers to a deep learning model with large-scale model parameters, which can usually contain hundreds of millions, tens of billions, hundreds of billions, trillions or even more than 10 trillion model parameters. The large model can also be called a cornerstone model / foundation model (Foundation Model). The large model is pre-trained through large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks, and the model has good generalization ability, such as large-scale language model (Large Language Model, LLM), multi-modal pre-training model (multi-modal pre-training model), etc.

[0034] It should be noted that when the large model is actually used, the pre-trained model can be fine-tuned through a small number of samples so that the large model can be applied to different tasks. For example, the large model can be widely used in natural language processing (Natural Language Processing, referred to as NLP), computer vision, speech processing and other fields, and can be specifically applied to computer vision tasks such as visual question answering (Visual Question Answering, referred to as VQA), image description (Image Caption, referred to as IC), image generation, etc., and can also be widely used in text-based sentiment classification, text summary generation, machine translation and other natural language processing tasks. Therefore, the main application scenarios of the large model include but are not limited to digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc. In the embodiment of the present application, data processing by machine learning model in a dialogue scenario is used as an example for explanation.

[0035] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following explanations:

[0036] Large machine learning models can be machine learning models with huge number of parameters;

[0037] Unstructured sparsity can refer to the fact that in machine learning sparse matrix calculations, the sparse data positions have no fixed structure;

[0038] Structured sparsity can refer to the fact that in machine learning sparse matrix calculations, the sparse data locations have a fixed structure;

[0039] Graphics Processing Unit (GPU) is a computing device used to process graphics and images. It can be widely used in image processing, scientific computing and machine learning.

[0040] Graphics Processing Unit kernel (GPU kernel for short) can be a function sent by the CPU to the GPU for execution;

[0041] Shared memory can refer to the on-chip memory that can be shared within a thread block on the GPU;

[0042] Global memory, which can refer to the GPU's main memory, is visible to all threads globally;

[0043] The hardware accelerator (Tensor Core) can be a component on the GPU used to accelerate matrix calculations. It can be used to perform matrix and convolution calculation tasks, and can be used to accelerate tasks involving tensor calculations in deep learning.

[0044] Example 1

[0045] According to an embodiment of the present application, a method for processing data in a model is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0046] Considering the huge number of model parameters of the large model and the limited computing resources of the mobile terminal, the data processing method in the above model provided in the embodiment of the present application can be applied to the following examples: Figure 1 The application scenarios shown are not limited to these. Figure 1 is a schematic diagram of an application scenario of a method for processing data in a model according to an embodiment of the present application. Figure 1In the application scenario shown, the large model is deployed in the server 10, and the server 10 can be connected to one or more client devices 20 through a local area network connection, a wide area network connection, an Internet connection, or other types of data networks. The client device 20 here may include but is not limited to: a smart phone, a tablet computer, a laptop computer, a PDA, a personal computer, a smart home device, a vehicle-mounted device, etc. The client device 20 can interact with the user through a graphical user interface to implement the call of the large model, thereby implementing the method provided in the embodiment of the present application.

[0047] In an embodiment of the present invention, a system composed of a client device and a server can perform the following steps: the client device obtains input data. The server executes step S101 to read the parameter information of the model from the off-chip storage space, wherein the parameter information is stored in blocks in an unstructured sparse matrix format; step S102 to decompress the parameter information of the model and store it in the corresponding shared memory, wherein the decompressed parameter information is stored in a structured dense matrix format, and the model is obtained based on machine learning training; step S103 to determine the output data that matches the input data based on the decompressed parameter information. The server 10 can transmit the output data to the client and display it in the graphical user interface of the client. It should be noted that when the operating resources of the client device can meet the deployment and operation conditions of the large model, the embodiment of the present invention can be performed in the client device.

[0048] Under the above operating environment, this application provides Figure 2 How data is processed in the model shown. Figure 2 FIG. 1 is a flow chart of a method for processing data in a model according to an embodiment of the present invention. Figure 2 As shown, the method may include the following steps:

[0049] Step S202, reading parameter information of the model from an off-chip storage space, wherein the parameter information is stored in blocks in an unstructured sparse matrix format, and the model is obtained based on machine learning training.

[0050] In the technical solution provided in the above step S202 of the present invention, the parameter information of the model can be stored in an off-chip storage space. When the model is to be used for data prediction or processing, the parameter information of the model can be read from the off-chip storage space. Among them, the off-chip storage space can be a global memory space (Global Memory, referred to as Gmem). The model can be a machine learning large model (referred to as a large model), a language model, etc., which can be obtained based on machine learning training and can be applied to a dialogue system. It should be noted that no specific restrictions are made on the use scenarios of the machine learning large model. The parameter information can be information stored in an unstructured sparse matrix format, which can be used to determine the parameters of the large model, and can include a parameter matrix or array. This is only an example, and no specific restrictions are made on the storage method of the parameter information. The sparse matrix format can be a format in which the number of elements with a value of 0 in the matrix state is far greater than the number of non-zero elements. For example, it can be (0, 1, 0, 0, 0). It should be noted that this is only an example, and no specific restrictions are made on the expression form and content of the sparse matrix format.

[0051] Optionally, the parameters of the model may be processed in blocks, and the block-wise parameters may be stored in an off-chip storage space in an unstructured sparse matrix format. When the model is used to process input data, the parameter information of the model may be read from the off-chip storage space.

[0052] For example, the parameter matrix of the large model can be divided into blocks, and the divided parameter matrix is ​​stored in an off-chip storage space in an unstructured sparse matrix format. When the large model is used, the parameter information of the large model can be read from the off-chip storage space.

[0053] Step S204, decompressing the parameter information of the model and storing it in the corresponding shared memory, wherein the decompressed parameter information is stored in a structured dense matrix format.

[0054] In the technical solution provided in the above step S204 of the present invention, the parameter information of the model can be decompressed, and the decompressed parameter information can be stored in the corresponding shared memory. Among them, the shared memory can be a memory area in the on-chip random access memory (Random Access Memory, referred to as RAM), for example, it can be a sharable on-chip memory (Shared memory, referred to as Smem) in the thread block, cache, etc. This is only an example, and the type of shared memory is not specifically limited. The dense matrix format can be a data structure and format for storing and representing dense matrices, which can include a format for storing elements of the matrix in the form of a two-dimensional array, for example, it can be (0, 1, 1, 1, 1). It should be noted that this is only an example, and the expression form and content of the dense matrix format are not specifically limited.

[0055] Optionally, the thread block of each image processor can read the parameter information of the large model from the off-chip storage space, and then store the parameter information in a distributed manner in registers corresponding to different threads in the thread block. The parameter information stored in the register can be decompressed and stored in the corresponding shared memory.

[0056] For example, for a GPU kernel that performs sparse matrix multiplication, each thread block in the GPU kernel reads the corresponding block-stored parameter information from the off-chip storage space, and then stores it in a distributed manner on registers corresponding to different threads in the thread block. The parameter information stored in the register can be decompressed in a dense matrix format and stored in a shared memory to obtain the parameters of a large model stored in a structured dense matrix format.

[0057] Step S206, based on the decompressed parameter information, determine output data that matches the input data, wherein the output data is used to characterize the dialogue result generated by the large model processing the input data.

[0058] In the technical solution provided in the above step S206 of the present invention, the output data matching the input data can be determined based on the decompressed parameter information. Among them, the input data can be data obtained through dialogue instructions, data input by the client, questions, conditions and other data of the client, text, images, content, etc., and no specific restrictions are made here on the form of expression of the input data. The output data can be used to characterize the dialogue results generated by the large model processing the input data, and can be data such as images, text, and voice. This is only for example, and no specific restrictions are made on the form of expression of the output data. The dialogue instruction can be information issued by the client for dialogue, for example, it can be a voice "open Xiao Ai classmate", it can be text, pictures, etc. This is only for example, and no specific restrictions are made on the form and content of the dialogue instruction.

[0059] Optionally, when the client wants to have a conversation through the dialogue system, it can issue a dialogue instruction. In response to receiving the dialogue instruction, the dialogue system obtains the client's input data, and can determine output data matching the input data based on the decompressed parameter information, and display it in the client's display interface.

[0060] For example, in a language model-based dialogue generation system (Chat GPT), a user can send a question to trigger the sending of a dialogue instruction. The dialogue generation system can obtain a dialogue instruction. When a dialogue instruction is received, the dialogue generation system can obtain the input data entered by the client, and based on the decompressed parameter information, it can determine the output data that matches the input data. The output data can be the dialogue result obtained by processing the question sent by the user, and the dialogue result can be displayed in the user's display interface. It should be noted that the data processing process in the above model is only an example. As long as it is a way of conducting a dialogue using a large model, it should be within the scope of protection of this application, and no specific restrictions are made here.

[0061] Through the above steps S202 to 206 of the present application, parameter information stored in an unstructured sparse matrix format is obtained. After the parameter matrix is ​​decompressed and processed, calculations are performed based on the decompressed parameter information and the input data to obtain the output data of the large model. Since the matrix with unstructured sparse storage occupies less memory space, the technical effect of reducing the space consumed when storing large model parameters is achieved, thereby solving the technical problem of excessive space consumption in model parameter storage.

[0062] The above method of this embodiment is further introduced below.

[0063] As an optional implementation, step S202, reading the parameter information of the model from the off-chip storage space, includes: reading from the off-chip storage space, the block offset array and the non-zero element array of the model corresponding to the thread block in the image processor.

[0064] In this embodiment, the block offset array and non-zero element array of the model corresponding to the thread block in the image processor can be read from the off-chip storage space. Among them, the block offset array can be used to determine the position of the element stored in the parameter information among all elements, which can be represented by Tile Offsets. The non-zero element array can be used to determine the value of the element in the parameter information, which can be represented by Non Zeros. The thread block can be located in the image processor core, is the smallest scheduling unit in the GPU, and can be composed of multiple threads, and the above threads can execute the same program code in parallel.

[0065] Optionally, the embodiment may process the model parameter matrix in a sparse matrix format to obtain at least one block of parameter information, wherein the parameter information may be composed of a block offset array and a non-zero element array. For an image processor kernel of sparse matrix multiplication, each thread block in the image processor kernel reads a block offset array and a non-zero element array corresponding to a corresponding block parameter matrix from an off-chip storage space to obtain the parameter information.

[0066] As an optional implementation, step S204, decompresses the parameter information of the model and stores it in a corresponding shared memory, including: determining the starting position of the non-zero elements in the parameter information in the non-zero element array based on the block offset array in the parameter information; determining the numerical value and position information of the non-zero elements in the non-zero element array based on the starting position; constructing the parameter information in a structured dense matrix format based on the numerical value and position information; and storing the parameter information in a structured dense matrix format in a corresponding shared memory.

[0067] In this embodiment, the parameter information of the model is read from the off-chip storage space. The starting position of the non-zero element in the parameter information in the non-zero element array is determined from the block offset array in the parameter information, and the numerical value and position information of the non-zero element in the non-zero element number can be determined based on the starting position. The parameter information of the structured dense matrix format can be constructed based on the numerical value and position information, and the parameter information of the structured dense matrix format can be stored in the corresponding shared memory. Wherein, the non-zero element can be data such as 1, 2, 3, and can be the parameter data in the model, which is only for example, and the size of the non-zero element is not specifically limited. The numerical value of the non-zero element in the parameter information and the position information corresponding to the non-zero element are stored in the non-zero element array. The position information can be used to determine the position of the non-zero element in the parameter matrix.

[0068] Optionally, the block offset array records the starting position of the non-zero elements in each block of parameter information in the non-zero element array. Therefore, through the block offset array, the starting position and ending position of the first non-zero element in each block of parameter information in the non-zero element array can be determined, thereby determining the numerical size of the non-zero elements in the parameter information.

[0069] For example, the block offset array stores 0, 10, and 16, where 0 indicates that the sum of non-zero elements in all parameter information before the first block of parameter information is 0; 10 indicates that the sum of non-zero elements in all parameter information before the second block of parameter information is 0, that is, the sum of non-zero elements in the first block of parameter information is 10; 16 indicates that the sum of all non-zero elements in the second block of parameter information and the first block of parameter information is 16. Assuming that the non-zero element array is (3, 2, 3, 1, 1, 10, 5), it can be determined that the first 5 digits in the non-zero element array are the non-zero elements in the first block of parameter information, and starting from the sixth digit, the value of the first non-zero element in the second block of parameter information. It should be noted that the above determination method is only an example, and the method of determining the value of non-zero elements in parameter information through the block offset array and the non-zero element array should be within the protection scope of the present application.

[0070] As an optional implementation, parameter information in a structured dense matrix format is constructed based on numerical value and position information, including: constructing parameter information in a structured sparse matrix format based on numerical value and position information; filling the parameter information in the structured sparse matrix format with target values ​​to obtain parameter information in a structured dense matrix format.

[0071] In this embodiment, parameter information in a structured sparse matrix format may be constructed based on the numerical value and position information of the non-zero elements, and the parameter information in a structured sparse matrix format may be filled with a target value to construct parameter information in a structured dense matrix format, wherein the target value may be 0.

[0072] Optionally, the parameter information stored in the sparse matrix format is read from the off-chip storage space to determine the value and position information of the non-zero elements in the parameter information. Based on the value and position information, the parameter information stored in the sparse matrix format is converted to obtain the parameter information in the structured sparse matrix format, and the parameter information in the structured sparse matrix format can be filled with data to obtain a parameter matrix in the structured dense matrix format.

[0073] As an optional implementation, step S206, based on the decompressed parameter information, determines the output data that matches the input data, including: extracting input features from the input data, wherein the input data is determined based on the dialogue instructions; iteratively processing the decompressed parameter information and the input features to determine the output features; and determining the output data based on the output features.

[0074] In this embodiment, input features can be extracted from the input data, and iterative processing is performed based on the decompressed parameter information and the input features to determine the output features. The output features can be converted to obtain output data. Among them, the decompressed parameter information can be a parameter matrix stored in a dense matrix format. The input features or output features can be in the form of vectors, matrices, etc., and can be an input feature matrix or an output feature matrix. For example, the input features can be represented by matrix B, which can also be called an input feature matrix. The output features can be represented by matrix C, which can also be called an output feature matrix. It should be noted that this is only an example, and no specific restrictions are made on the expression forms of input features and output features. The dialogue command can be information issued by the client for dialogue, for example, it can be a voice "Open Xiao Ai classmate", it can be text, pictures, etc. This is only an example, and no specific restrictions are made on the expression form and content of the dialogue command.

[0075] Optionally, the parameter information in the sparse matrix format stored in the register can be decompressed into a shared memory in a dense matrix format and stored. The output matrix can be divided into several blocks according to rows and columns, and each block of matrix C is obtained by multiplying the elements of the corresponding several rows of matrix A and the elements of the corresponding several columns of matrix B. The corresponding rows of matrix A and columns of matrix B can be divided into blocks, and a part of the multiplication results are calculated each time, and finally accumulated to obtain the final output matrix.

[0076] Optionally, the original parameter matrix is ​​divided into blocks and unstructured sparse processing is performed to obtain parameter information, and the parameter information is stored in an off-chip memory. The parameter information can be read from the off-chip and decompressed into the on-chip shared memory to be stored in a dense format, wherein the sparse data can be padded with 0s. Each GPU thread block calculates an output matrix, that is, iteratively calculates the corresponding parameter information and input matrix, and accumulates them to obtain input features.

[0077] For example, in a large model, the relationship between parameter information (e.g., parameter matrix), input features (e.g., input feature matrix) and output features (e.g., output feature matrix) can be: output features = parameter information × input features. Among them, the parameter information can be a weight matrix learned during the model training process, which can be obtained by dividing all the parameters required by the model into blocks. The input features are feature data obtained by the large model processing the input data, which can be used to characterize the input data. The output features can be the output of the large model, which can be used to determine the output data of the large model. Therefore, the large model can output the corresponding output feature matrix based on the parameter information and the given input features. After obtaining the decompressed parameter information, the decompressed parameter information and the input feature matrix can be iteratively calculated and cumulatively added to obtain the output feature matrix.

[0078] As an optional implementation, the method may further include: dividing the original parameter information of the model into at least one block of sub-parameter information; and converting the sub-parameter information into parameter information.

[0079] In this embodiment, the original parameter information of the model is obtained, and the original parameter information can be processed in blocks to obtain at least one piece of sub-parameter information, and the sub-parameter information can be converted into parameter information. The original parameter information can be a sparse matrix, which can be represented by a matrix A, and can be parameter information obtained after the model is trained, for example, it can be used to characterize the weight data of the model. This is only for example description, and no specific limitation is imposed on the form and content of the original parameter information.

[0080] Optionally, the embodiment may divide the original parameter information into blocks to obtain at least one block of sub-parameter information, each block of sub-parameter information storing a certain number of non-zero elements. To further save storage space during parameter storage, the sub-parameter information stored in structured sparse storage may be converted into an unstructured sparse matrix format for storage.

[0081] Since multiple GPU cards are used to store huge model parameters, and the resource consumption of the method of using multiple GPU cards to perform inference calculations of the model is too huge, it leads to huge deployment costs, and can only be accelerated for fixed structured coefficients, and cannot support arbitrary unstructured sparse matrix multiplication operations, resulting in the technical problem that model parameter storage consumes a lot of space. To solve the above problems, this embodiment performs unstructured sparse processing on the original parameter information of the model to obtain parameter information. This method reduces the amount of model parameters while retaining sufficient model accuracy, thereby reducing the required storage resources.

[0082] Among them, unstructured sparse storage is often more able to retain the accuracy of the model than structured sparse storage. The challenge faced by unstructured sparse on the GPU is that it is difficult to adapt to structured computing units (also known as operation units). Therefore, the parameter information can be stored and read in a sparse matrix format. When in use, the parameter information can be converted into a dense matrix format and calculated in a dense matrix format to effectively support unstructured sparse matrix multiplication operations on computing units (Tensor Cores). That is, in this application, by utilizing the Tensor Core in the GPU to accelerate the execution of the model after unstructured sparseness, in order to reduce resource consumption and improve computing efficiency, this method can make unstructured sparse matrix calculations run efficiently on the Tensor Core of the GPU, that is, effectively reduce the use of storage resources, and achieve the purpose of accelerating computing efficiency and reducing latency by reducing the amount of memory access.

[0083] As an optional implementation, step S202, converting the sub-parameter information into parameter information, includes: determining an initial index position corresponding to the sub-parameter information, and an array of non-zero elements corresponding to the sub-parameter information; based on the initial index position and the array of non-zero elements, determining the parameter information corresponding to the sub-parameter information.

[0084] In this embodiment, the initial index position corresponding to the sub-parameter information and the non-zero element array corresponding to the sub-parameter information can be determined, and the parameter information corresponding to the sub-parameter information can be determined based on the initial index position and the non-zero element array. The initial index position can be the sum of the non-zero elements in all sub-parameter information before the sub-parameter information.

[0085] Optionally, the matrix A can be divided into blocks T1, T2, T3, T4, T5, T6, T7, T8, ... A certain number of non-zero elements (A, B, C, D, E, F, G, H, I, J) are stored in the T1 block. Assuming that the sum of non-zero elements in TI is 10, the sum of non-zero elements in T2 is 16, the sum of non-zero elements in T3 is 21, ..., the initial index position corresponding to each block can be determined, and the initial index position corresponding to each block is stored in a block offset array named Tile Offsets. Based on the initial index position and the non-zero element array, the parameter information corresponding to the sub-parameter information can be determined.

[0086] As an optional implementation, step S202, determining the initial index position corresponding to the sub-parameter information, includes: determining the initial index position of the sub-parameter information based on the sum of non-zero elements in the sub-parameter information, wherein the initial index position is stored in a block offset array, and the length of the block offset array is the same as the number of blocks of the sub-parameter information in the original parameter information; determining the parameter information corresponding to the sub-parameter information based on the initial index position and the non-zero element array, includes: determining the block offset array and the non-zero element array storing the initial index position as the parameter information corresponding to the sub-parameter information.

[0087] In this embodiment, the initial index position of the sub-parameter information can be determined based on the sum of the non-zero elements in the sub-parameter information, and the initial index position of each block of sub-parameter information can be stored in an array to obtain a block offset array. The block offset array storing the initial index position and the non-zero element array can be determined as the parameter information corresponding to the sub-parameter information. The length of the block offset array is the same as the number of blocks of sub-parameter information in the original parameter information.

[0088] Optionally, the sum of non-zero elements in all sub-parameter information before the sub-parameter information may be determined, and the sum may be determined as the initial index position corresponding to the sub-parameter information, and the initial index position of each block of sub-parameter information may be stored in an array to obtain a block offset array. The block offset array and the corresponding non-zero element array may be determined as the parameter information corresponding to the sub-parameter information.

[0089] For example, the original parameter information is divided into three blocks of sub-parameter information. It is determined that the sum of non-zero elements of all parameter information before the first block of sub-parameter information is 0, and the initial index position corresponding to the first block of sub-parameter information is 0. It is determined that the sum of non-zero elements in the first block of sub-parameter information before the second block of sub-parameter information is 5, and the initial index position corresponding to the second block of sub-parameter information is 5. It is determined that the sum of non-zero elements in the first block of sub-parameter information and the second block of sub-parameter information before the third block of sub-parameter information is 15, and the initial index position corresponding to the third block of sub-parameter information is 15, and the block offset array (0, 5, 15) can be obtained. At the same time, the non-zero element array corresponding to each block of sub-parameter information is determined, and the block offset array and the non-zero element array corresponding to the sub-parameter information can be determined as the parameter information corresponding to the sub-parameter information.

[0090] As an optional implementation, determining the non-zero element array corresponding to the sub-parameter information includes: storing the non-zero elements in the sub-parameter information and the positions of the non-zero elements in the sub-parameter information to obtain the non-zero element array.

[0091] In this embodiment, the non-zero element corresponding to the sub-parameter information and the position of the non-zero element are determined, and the size of the non-zero element and the position of the non-zero element can be stored to obtain a non-zero element array. The position of the non-zero element can also be called the relative index of the non-zero element, which can be used to determine the distance between the position of the non-zero element and the first data content in the block of sub-parameter information.

[0092] Optionally, within each block, the original data type processed can be fp16, that is, a floating point type with a bit width of 16 bits. 32 bits can be used to store an fp16 data. Among them, in the non-zero element array (NonsZeros), the high 16 bits (VALUE<half,16bits> ) can be used to store the size of non-zero elements in the sub-parameter information. The low 16 bits (Location<short,16bits> ) can be used to store the relative index of the data in its corresponding block. Through such a data structure, the sparse matrix can be effectively stored in blocks, thereby achieving the purpose of reducing the storage space of the parameters.

[0093] As an optional implementation, step S202 stores the non-zero elements in the sub-parameter information and the positions of the non-zero elements in the sub-parameter information to obtain a non-zero element array, including: determining the position of the non-zero element in the sub-parameter information and the relative distance between the non-zero element and the first element in the sub-parameter information; storing the non-zero elements in the sub-parameter information and the relative distances corresponding to the non-zero elements to obtain a non-zero element array.

[0094] In this embodiment, the position of the non-zero element in the parameter information and the relative distance between the non-zero element and the first element in the sub-parameter information can be determined, and the value of the non-zero element and the relative distance corresponding to the non-zero element can be stored to obtain a non-zero element array.

[0095] For example, assuming the sub-parameter information is In the sub-parameter information, the position of the first non-zero element 2 in the sub-parameter information is 3, and the relative distance between the first element in the sub-parameter information is 3. The non-zero element 2 and the relative distance 3 can be stored. After polling all non-zero elements in turn in the above manner, the non-zero element array corresponding to the sub-parameter information is obtained.

[0096] As an optional implementation, in step S202, elements in the same sub-parameter information are continuously stored in the same off-chip storage space, and elements in different sub-parameter information are stored in different off-chip storage spaces.

[0097] In this embodiment, elements in the same block of sub-parameter information may be stored continuously in a storage space, and sub-parameter information of different blocks may be stored in storage spaces at different locations.

[0098] Optionally, the original parameter information is stored in blocks to obtain at least one block of sub-parameter information, the data in each block of sub-parameter information is continuously stored in a storage space, and the data in sub-parameter information of different blocks are stored in storage spaces at different locations.

[0099] This embodiment stores the parameters of the large model (i.e., original parameter information) through a sparse data storage method that combines block data and indexes. The sparse data (i.e., parameter information) can be stored in the global memory to achieve the purpose of improving data processing efficiency.

[0100] For example, in the sparse matrix multiplication of the machine learning model, for example, AxB=C. Among them, A can be a parameter matrix (weight), B is an input feature matrix (feature), C is an output feature matrix, matrix A is a sparse matrix, and B and C are dense matrices. Matrix A can be stored in blocks, and the data in each small block is continuously stored in a storage space, and the data in different blocks are stored in storage spaces at different locations. Each block stores a certain number of non-zero elements.

[0101] In this embodiment, parameter information stored in an unstructured sparse matrix format is obtained, the parameter matrix is ​​decompressed and processed, and calculations are performed based on the decompressed parameter information and input data to obtain output data of the large model. Since the unstructured sparsely stored matrix occupies less memory space, the technical effect of reducing the space consumed when storing large model parameters is achieved, thereby solving the technical problem of excessive space consumption in model parameter storage.

[0102] According to an embodiment of the present application, a method for processing data in a model is also provided based on the upper-layer software service side (Software-as-a-Service, abbreviated as SaaS) of the computing edge. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0103] Figure 3 FIG. 1 is a flowchart of a method for processing data in another model according to an embodiment of the present application. Figure 3 As shown, the method may include the following steps:

[0104] Step S302, reading the parameter information of the model from the off-chip storage space by calling the first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter is the parameter information, the parameter information is stored in blocks in an unstructured sparse matrix format, and the model is obtained based on machine learning training.

[0105] In the technical solution provided in the above step S302 of the present application, the first interface can be an interface for data interaction between the server and the client. The server can obtain the input data transmitted by the client, and call the first interface to read the parameter information of the model from the off-chip storage space as a first parameter of the first interface, thereby achieving the purpose of obtaining the parameter information.

[0106] Step S304, decompressing the parameter information of the model and storing it in the corresponding shared memory, wherein the decompressed parameter information is stored in a structured dense matrix format.

[0107] Step S306, based on the decompressed parameter information, determine output data that matches the input data, wherein the output data is used to represent the dialogue result generated by the model processing the input data.

[0108] Step S308: output the determined output data by calling a second interface, wherein the second interface includes a second parameter, and a parameter value of the second parameter is the output data.

[0109] In the technical solution provided in the above step S308 of the present application, the second interface can be an interface for data interaction between the server and the client. The server can pass the output data into the second interface as a parameter of the second interface to achieve the purpose of returning the output data to the client.

[0110] Figure 4 is a schematic diagram of processing of data in a model by a computer device according to an embodiment of the present application, such as Figure 4 As shown, the computer device can implement step S402, reading the parameter information of the model from the off-chip storage space by calling the first interface; step S404, decompressing the parameter information of the model and storing it in the corresponding shared memory; step S406, if a dialogue instruction is received, determining the output data matching the input data based on the decompressed parameter information; step S408, outputting the determined output data by calling the second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the output data.

[0111] According to an embodiment of the present application, in the scenario of text generation, a method for processing data in a model is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0112] FIG5(a) is a flow chart of another method for processing data in a model according to an embodiment of the present application. As shown in FIG5(a), the method may include the following steps:

[0113] Step S51, obtaining multimodal information input in the interactive interface, wherein the type of the multimodal information includes at least one of the following: text information containing character information, video frame information containing frame image information, and audio information.

[0114] In the technical solution provided in the above step S51 of the present application, multimodal information input in the interactive interface can be obtained. Among them, the interactive interface can be an interface for generating dialogues using a natural speech generation model, or it can be an operating interface of a mobile terminal, a client, etc. This is only an example, and no specific limitation is made to the type of interactive interface. The type of multimodal information may include at least one of the following: inquiry information containing character information, audio information, video frame information containing frame image information, images, etc. For example, it can be text, voice, etc. input by the user. It should be noted that this is only an example, and no specific limitation is made to the type of multimodal information.

[0115] Step S52, reading parameter information of the model from an off-chip storage space, wherein the parameter information is stored in blocks in an unstructured sparse matrix format, and the model is obtained based on machine learning training.

[0116] Step S53, decompressing the parameter information of the model and storing it in the corresponding shared memory, wherein the decompressed parameter information is stored in a structured dense matrix format.

[0117] Step S54, based on the decompressed parameter information, determine the reply information matching the multimodal information, wherein the reply information is used to characterize the reply content generated by the model processing the multimodal information, and the type of the reply information includes at least one of the following: text information, image information and voice information.

[0118] In the technical solution provided in the above step S54 of the present application, the multimodal information input in the interactive interface is obtained, and the parameter information of the model is read from the off-chip storage space; the parameter information of the model is decompressed and stored in the corresponding shared memory. Based on the decompressed parameter information, the reply information matching the obtained multimodal information can be determined.

[0119] Step S55, displaying the reply information in the interactive interface.

[0120] In the technical solution provided in the above step S55 of the present application, the reply information can be displayed in the interactive interface to conduct the next conversation with the user.

[0121] In an embodiment of the present application, parameter information of a model is read from an off-chip storage space by calling a first interface; the parameter information of the model is decompressed and stored in a corresponding shared memory; if a dialogue instruction is received, output data matching the input data is determined based on the decompressed parameter information; the determined output data is output by calling a second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the output data, thereby achieving the technical effect of reducing the space consumed by large model parameter storage and solving the technical problem of excessive space consumption of model parameter storage.

[0122] Example 2

[0123] According to an embodiment of the present application, an embodiment of a system for processing data in a model is also provided. Figure 5(b) is a schematic diagram of a system for processing data in a model according to an embodiment of the present application. As shown in Figure 5(b), the system 500 for processing data in a model may include: a thread block 502, an on-chip memory 504 and a computing component 506.

[0124] Thread block 502 is used to read parameter information of the model from an off-chip storage space and store the parameter information in a register, wherein the parameter information is stored in blocks in an unstructured sparse matrix format, and the model is obtained based on machine learning training.

[0125] In this embodiment, the thread block in each image processor can read the corresponding parameter information stored in the block from the off-chip storage space.

[0126] Optionally, the GPU thread block in each GPU kernel may read the corresponding block-stored parameter information from the global memory, and may store the acquired parameter information in a distributed manner in registers corresponding to different threads in the thread block.

[0127] The on-chip memory 504 decompresses and stores the parameter information of the model, wherein the decompressed parameter information is stored in a structured dense matrix format.

[0128] In this embodiment, the parameter information of the model may be decompressed, and the parameter information stored in the register may be decompressed in a dense matrix format and stored in the on-chip memory 504. The on-chip memory may be a shared memory.

[0129] The calculation component 506 is used to determine output data matching the input data based on the decompressed parameter information, wherein the output data is used to represent the dialogue result generated by the model processing the input data.

[0130] In this embodiment, the computing component 506 is used to determine the output data matching the input data based on the decompressed parameter information. The computing component may be a computing unit (Tensor Core), which may also be called a computing unit.

[0131] Optionally, the computing component 506 can read the corresponding data block (parameter information) from the on-chip memory 504 and perform matrix calculations. By reading off-chip sparse data and decompressing it on-chip in a dense matrix format for tensor core calculations, Tensor Core successfully supports the storage and calculation of off-chip sparse matrix data. In response to instructions to perform matrix multiplication and accumulation (MMA), the registers of each thread in the thread block read the corresponding decompressed parameter data from the shared memory and perform matrix calculations to obtain output data.

[0132] In this embodiment, a system for processing data in a model is provided. The system reads parameter information of a large model from an off-chip storage space through a thread block and stores the parameter information in a register; decompresses and stores the parameter information of the large model through an on-chip memory; if a dialogue instruction is received, output data matching the input data is determined based on the decompressed parameter information through a computing component, thereby achieving the technical effect of reducing the space consumed by storing large model parameters and solving the technical problem of excessive space consumption of model parameter storage.

[0133] Example 3

[0134] At present, large machine learning models have become an important computing task on various computing platforms today, and have demonstrated extremely high value in scenarios such as dialogue systems. The decoder-based model structure has become the mainstream structure of today's large models.

[0135] Due to the huge number of parameters, large models often require huge storage space to store their parameter information. A single GPU card is often difficult to store all model parameters. At the same time, the huge amount of memory access and calculation also brings high computing latency. Effectively reducing the storage space requirements and latency on the GPU has become an important demand.

[0136] In an optional embodiment, multiple GPU cards are usually used to store huge model parameters, and multiple GPU cards are used to perform model inference calculations. However, this method consumes too much resources, resulting in huge deployment costs, and can only accelerate fixed structured coefficients, and cannot support arbitrary unstructured sparse matrix multiplication operations. The resource consumption during the deployment of large models is high, and there is still a technical problem that the model parameter storage consumes a lot of space.

[0137] In view of the problem that the deployment of large models requires excessive storage resources, and at the same time, the huge memory access and computing volume of large models make it difficult to meet the demand for low latency in computing services, this application proposes a sparse matrix multiplication optimization method for machine learning models based on Tensor Core, which can reduce the amount of model parameters while retaining sufficient model accuracy by unstructured sparse processing of the original parameter information of the large model, thereby reducing the required storage resources. Among them, unstructured sparse storage is often more capable of retaining the accuracy of the model than structured sparse storage. The challenge faced by unstructured sparse on the GPU is that it is difficult to adapt to structured computing units (also called operation units). Therefore, the original parameter information can be stored and read in a sparse format, and calculated in a dense format to effectively support unstructured sparse matrix multiplication operations on the computing unit. That is, in this application, the execution of large models after unstructured sparseness is accelerated by utilizing the TensorCore in the GPU to reduce resource consumption and improve computing efficiency. This method can enable unstructured sparse matrix calculations to run efficiently on the Tensor Core of the GPU, which on the one hand effectively reduces the use of storage resources, and on the other hand accelerates computing efficiency and reduces latency by reducing memory access.

[0138] The following is a further introduction to a sparse matrix multiplication optimization method based on Tensor Core for a machine learning model proposed in an embodiment of the present invention.

[0139] Figure 6 is a flow chart of an end-to-end sparse matrix multiplication pipeline method based on a computing unit according to an embodiment of the present invention, such as Figure 6 As shown, the method may include the following steps.

[0140] Step S601, storing sparse data in global memory.

[0141] In this embodiment, the parameters of the large model can be stored through a sparse data storage method that merges block data and indexes, and the original parameter information (i.e., the original parameter matrix) can be converted into sparse data and stored in the global memory, where the sparse data can be an unstructured sparse matrix, which can also be called parameter information.

[0142] Optionally, a sparse data storage method of block data and index merging can be used to block and store the parameters of the large model to achieve the purpose of improving data processing efficiency.

[0143] For example, in the sparse matrix multiplication of the machine learning model, for example, AxB=C. Among them, A can be a parameter matrix (weight), B is an input feature matrix (feature), C is an output feature matrix, matrix A is a sparse matrix, and B and C are dense matrices. Matrix A can be stored in blocks, and the data in each small block is continuously stored in a storage space, and the data in different blocks are stored in storage spaces at different locations. Each block stores a certain number of non-zero elements.

[0144] Figure 7 is a schematic diagram of a block storage according to an embodiment of the present invention, such as Figure 7 As shown, the matrix A can be divided into blocks T1, T2, T3, T4, T5, T6, T7, T8, ... A certain number of non-zero elements (A, B, C, D, E, F, G, H, I, J) are stored in the T1 block. Figure 8 is a schematic diagram of a block offset array according to an embodiment of the present invention, such as Figure 8 As shown, the sum of the non-zero elements in all other blocks before each block can be stored in a block offset array named Tile Offsets to determine the initial index position of each block in memory, such as Figure 8 As shown, the sum of non-zero elements in TI is 10, the sum of non-zero elements in T2 is 16, the sum of non-zero elements in T3 is 21, and so on.

[0145] Optionally, within each block, the original data type processed may be fp16, that is, a floating point number type with a bit width of 16 bits. 32 bits may be used to store data of an fp16. Fig. 9 is a schematic diagram of a non-zero element array according to an embodiment of the present invention, such as Fig. 9 As shown, in the non-zero element array (NonsZeros), the high 16 bits (VALUE<half,16bits> ) can be used to store the original data content in the block (that is, the value of the non-zero element), and the low 16 bits (Location<short,16bits> ) can be used to store the relative index of the data in its corresponding block. Through such a data structure, the sparse matrix can be effectively stored in blocks.

[0146] Step S602, read the corresponding sparse data from the global memory and store it in the register.

[0147] In this embodiment, for sparse matrix multiplication, the GPU thread block in each GPU core reads the corresponding block-stored parameter information from the global memory, and the acquired parameter information can be distributedly stored in registers corresponding to different threads in the thread block.

[0148] Step S603, decompressing the parameter information stored in the register into the shared memory in a dense format.

[0149] In this embodiment, the parameter information in the sparse matrix format stored in the register may be decompressed into the shared memory in a dense format.

[0150] In this embodiment, a method is proposed to read off-chip sparse data and decompress it into a dense format on-chip for tensor core calculation. In this method, the sparse matrix can be calculated in a block-by-block manner. Fig.10 is a schematic diagram of a tensor core calculation according to an embodiment of the present invention, such as Fig.10 As shown, the output feature matrix (which can be referred to as the output matrix for short) can be divided into several blocks according to rows and columns, and each block of the output matrix is ​​obtained by multiplying the elements in the corresponding rows of the original parameter matrix and the elements in the corresponding columns of the input feature matrix. The rows of the corresponding original parameter matrix and the columns of the input feature matrix can be divided into blocks, and the multiplication results of a part are calculated each time, and finally accumulated to obtain the final output matrix.

[0151] Optionally, the original parameter matrix is ​​divided into blocks and unstructured sparse processing is performed to obtain parameter information, and the parameter information is stored in an off-chip memory. The parameter information can be read from the off-chip and decompressed into the on-chip shared memory to be stored in a dense format, wherein the sparse data can be padded with 0s. Each GPU thread block calculates a block of the output matrix, that is, iteratively calculates the corresponding parameter information and input matrix, and accumulates them.

[0152] Step S604, reading corresponding matrix data from the shared memory.

[0153] In this embodiment, the Tensor Core can read the corresponding data block from the shared memory and perform matrix calculations. By reading the off-chip parameter information and decompressing it on-chip in a dense matrix format for tensor core calculations, the Tensor Core successfully supports off-chip sparse matrix data storage and operations.

[0154] Optionally, in response to an instruction to perform a matrix multiplication and accumulation operation, a register of each thread in the thread block reads corresponding matrix data (ie, decompressed parameter information) from a shared memory.

[0155] Step S605, performing matrix operations through a computing unit.

[0156] In this embodiment, the Tensor Core reads data from these registers and performs matrix calculations.

[0157] In this embodiment, a sparse data storage method of block data and index merging is used to effectively store unstructured sparse matrix data in a limited data space; a method of reading off-chip sparse data and decompressing it on-chip in a dense matrix format for Tensor Core calculation is used to effectively enable the structured Tensor Core to support unstructured off-chip data input; an end-to-end Tensor Core-based sparse matrix multiplication pipeline design is used to efficiently utilize GPU hardware resources, thereby achieving support for arbitrary sparse structures, not limited to structured sparse structures such as 2:4; a method of sparsely reading and densely calculating is used to effectively support sparse matrix multiplication calculations on Tensor Cores, which is no longer limited to structured sparse structures, but can support unstructured sparse structures, thereby achieving the technical problem of reducing resource consumption during large model deployment and solving the technical problem of high resource consumption during large model deployment.

[0158] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0159] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0160] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0161] Example 4

[0162] According to an embodiment of the present invention, there is also provided a method for implementing the above Figure 2 The method for processing data in the model shown is a device for processing data in the model.

[0163] Fig.11 is a schematic diagram of a data processing device in a model according to an embodiment of the present invention, such as Fig.11 As shown, the device 1100 for processing data in the model may include: a first reading unit 1102 , a first storage unit 1104 and a first determining unit 1106 .

[0164] The first reading unit 1102 is used to read parameter information of the model from an off-chip storage space, wherein the parameter information is stored in blocks in an unstructured sparse matrix format, and the model is obtained based on machine learning training.

[0165] The first storage unit 1104 is used to decompress the parameter information of the model and store it in the corresponding shared memory, wherein the decompressed parameter information is stored in a structured dense matrix format.

[0166] The first determination unit 1106 is used to determine output data matching the input data based on the decompressed parameter information, wherein the output data is used to represent a dialog result generated by the model processing the input data.

[0167] It should be noted that the first reading unit 1102, the first storage unit 1104 and the first determination unit 1106 correspond to steps S202 to S206 in Example 1, and the three units and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in Example 1. It should be noted that the above units can be hardware components or software components stored in a memory and processed by one or more processors, and the above units can also be run in the computer terminal provided in Example 1 as part of the device.

[0168] According to an embodiment of the present invention, another method for implementing the above Figure 3 The method for processing data in the model shown is a device for processing data in the model.

[0169] Fig.12 is a schematic diagram of a data processing device in a model according to an embodiment of the present invention, such as Fig.12 As shown, the device 1200 for processing data in the model may include: a second reading unit 1202 , a second storage unit 1204 , a second determining unit 1206 and an output unit 1208 .

[0170] The second reading unit 1202 is used to read the parameter information of the model from the off-chip storage space by calling the first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter is the parameter information, the parameter information is stored in blocks in an unstructured sparse matrix format, and the model is obtained based on machine learning training.

[0171] The second storage unit 1204 is used to decompress the parameter information of the model and store it in the corresponding shared memory, wherein the decompressed parameter information is stored in a structured dense matrix format.

[0172] The second determining unit 1206 is used to determine output data matching the input data based on the decompressed parameter information if a dialogue instruction is received, wherein the output data is used to represent a dialogue result generated by the model processing the input data.

[0173] The output unit 1208 is configured to output the determined output data by calling a second interface, wherein the second interface includes a second parameter, and a parameter value of the second parameter is the output data.

[0174] It should be noted that the second reading unit 1202, the second storage unit 1204, the second determination unit 1206 and the output unit 1208 correspond to steps S302 to S308 in Example 1, and the four units and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in Example 1. It should be noted that the above units can be hardware components or software components stored in a memory and processed by one or more processors, and the above units can also be run in the computer terminal provided in Example 1 as part of the device.

[0175] According to an embodiment of the present invention, another device for processing data in a model for implementing the method for processing data in a model shown in FIG. 5( a ) is also provided.

[0176] Fig.13 is a schematic diagram of a data processing device in a model according to an embodiment of the present invention, such as Fig.13 As shown, the device 1300 for processing data in the model may include: an acquisition unit 1302 , a third reading unit 1304 , a processing unit 1306 , a third determination unit 1308 and a display unit 1310 .

[0177] The acquisition unit 1302 is used to acquire multimodal information input in the interactive interface, wherein the type of the multimodal information includes at least one of the following: text information including character information, video frame information including frame image information, and audio information.

[0178] The third reading unit 1304 is used to read parameter information of the model from the off-chip storage space, wherein the parameter information is stored in blocks in an unstructured sparse matrix format, and the model is obtained based on machine learning training.

[0179] The processing unit 1306 is used to decompress the parameter information of the model and store it in the corresponding shared memory, wherein the decompressed parameter information is stored in a structured dense matrix format.

[0180] The third determination unit 1308 is used to determine the reply information matching the multimodal information based on the decompressed parameter information, wherein the reply information is used to characterize the reply content generated by the model processing the multimodal information, and the type of the reply information includes at least one of the following: text information, image information and voice information.

[0181] The display unit 1310 is used to display the reply information in the interactive interface.

[0182] It should be noted that the acquisition unit 1302, the third reading unit 1304, the processing unit 1306, the third determination unit 1308 and the display unit 1310 correspond to steps S51 to S55 in Embodiment 1, and the five units and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in Embodiment 1. It should be noted that the above units can be hardware components or software components stored in a memory and processed by one or more processors, and the above units can also be run in the computer terminal provided in Embodiment 1 as part of the device.

[0183] In the above-mentioned device, parameter information stored in an unstructured sparse matrix format is obtained, the parameter matrix is ​​decompressed and processed, and calculations are performed based on the decompressed parameter information and input data to obtain the output data of the large model. Since the unstructured sparsely stored matrix occupies less memory space, the technical effect of reducing the space consumed when storing large model parameters is achieved, thereby solving the technical problem of excessive space consumption in model parameter storage.

[0184] Example 5

[0185] The embodiment of the present invention can provide a computer terminal, which can be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the computer terminal can also be replaced by a terminal device such as a mobile terminal.

[0186] Optionally, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of the computer network.

[0187] In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the method for processing data in the model: reading the parameter information of the model from the off-chip storage space, wherein the parameter information is stored in blocks in an unstructured sparse matrix format, and the model is obtained based on machine learning training; decompressing the parameter information of the model and storing it in the corresponding shared memory, wherein the decompressed parameter information is stored in a structured dense matrix format; based on the decompressed parameter information, determining output data that matches the input data, wherein the output data is used to characterize the dialogue results generated by the model processing the input data.

[0188] Optionally, Fig.14 is a structural block diagram of a computer terminal according to an embodiment of the present invention. Fig.14 As shown, the computer terminal A may include: one or more (only one is shown in the figure) processors 1402 , a memory 1404 and a transmission device 1406 .

[0189] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the method and device for processing data in the model in the embodiment of the present invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the method for processing data in the above-mentioned model. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer terminal A via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0190] The processor can call the information and application programs stored in the memory through the transmission device to perform the following steps: read the parameter information of the model from the off-chip storage space, wherein the parameter information is stored in blocks in an unstructured sparse matrix format, and the model is obtained based on machine learning training; decompress the parameter information of the model and store it in the corresponding shared memory, wherein the decompressed parameter information is stored in a structured dense matrix format; based on the decompressed parameter information, determine the output data that matches the input data, wherein the output data is used to characterize the dialogue results generated by the model processing the input data.

[0191] Optionally, the processor may also execute program code of the following steps: reading, from an off-chip storage space, a block offset array and a non-zero element array of a model corresponding to a thread block in the image processor.

[0192] Optionally, the processor may also execute the following program code steps: based on the block offset array in the parameter information, determine the starting position of the non-zero elements in the parameter information in the non-zero element array; based on the starting position, determine the numerical value and position information of the non-zero elements in the non-zero element array; based on the numerical value and position information, construct the parameter information in a structured dense matrix format; and store the parameter information in a structured dense matrix format in a corresponding shared memory.

[0193] Optionally, the processor may also execute the following program code: construct parameter information in a structured sparse matrix format based on numerical and positional information; fill the parameter information in a structured sparse matrix format with target values ​​to obtain parameter information in a structured dense matrix format.

[0194] Optionally, the processor may also execute program code of the following steps: extracting input features from input data, wherein the input data is determined based on dialogue instructions; iteratively processing the decompressed parameter information and the input features to determine output features; and determining output data based on the output features.

[0195] Optionally, the processor may also execute program codes of the following steps: dividing the original parameter information of the model into at least one block of sub-parameter information; and converting the sub-parameter information into parameter information.

[0196] Optionally, the processor may also execute the following program code: determining an initial index position corresponding to the sub-parameter information and an array of non-zero elements corresponding to the sub-parameter information; and determining parameter information corresponding to the sub-parameter information based on the initial index position and the array of non-zero elements.

[0197] Optionally, the processor may also execute program code of the following steps: determining an initial index position of the sub-parameter information based on the sum of non-zero elements in the sub-parameter information, wherein the initial index position is stored in a block offset array, and the length of the block offset array is the same as the number of blocks of the sub-parameter information in the original parameter information; determining parameter information corresponding to the sub-parameter information based on the initial index position and the non-zero element array, including: determining the block offset array and the non-zero element array storing the initial index position as the parameter information corresponding to the sub-parameter information.

[0198] Optionally, the processor may also execute program code of the following steps: storing non-zero elements in the sub-parameter information and positions of the non-zero elements in the sub-parameter information to obtain a non-zero element array.

[0199] Optionally, the processor may also execute the program code of the following steps: determining the position of the non-zero element in the sub-parameter information and the relative distance between the non-zero element and the first element in the sub-parameter information; storing the non-zero elements in the sub-parameter information and the relative distances corresponding to the non-zero elements to obtain an array of non-zero elements.

[0200] Optionally, the processor may further execute program code of the following steps: storing elements in the same sub-parameter information continuously in the same off-chip storage space, and storing elements in different sub-parameter information in different off-chip storage spaces.

[0201] As another optional example, the processor can call the information and application stored in the memory through the transmission device to perform the following steps: read the parameter information of the model from the off-chip storage space by calling the first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter is the parameter information, the parameter information is stored in blocks in an unstructured sparse matrix format, and the model is obtained based on machine learning training; decompress the parameter information of the model and store it in the corresponding shared memory, wherein the decompressed parameter information is stored in a structured dense matrix format; based on the decompressed parameter information, determine the output data that matches the input data, wherein the output data is used to characterize the dialogue results generated by the model processing the input data; output the determined output data by calling the second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the output data.

[0202] As another optional example, the processor may call the information and application stored in the memory through the transmission device to perform the following steps: obtaining multimodal information input in the interactive interface, wherein the type of the multimodal information includes at least one of the following: text information containing character information, video frame information containing frame image information, and audio information; reading parameter information of the model from an off-chip storage space, wherein the parameter information is stored in blocks in an unstructured sparse matrix format, and the model is obtained based on machine learning training; decompressing the parameter information of the model and storing it in a corresponding shared memory, wherein the decompressed parameter information is stored in a structured dense matrix format; determining reply information matching the multimodal information based on the decompressed parameter information, wherein the reply information is used to characterize the reply content generated by the model processing the multimodal information, and the type of the reply information includes at least one of the following: text information, image information, and voice information; and displaying the reply information in the interactive interface.

[0203] By adopting the embodiment of the present invention, parameter information stored in an unstructured sparse matrix format is obtained, the parameter matrix is ​​decompressed and processed, and then calculations are performed based on the decompressed parameter information and input data to obtain the output data of the model. Since the matrix stored in an unstructured sparse format occupies less memory space, the technical effect of reducing the space consumed when storing model parameters is achieved, thereby solving the technical problem of excessive space consumption when storing model parameters.

[0204] It can be understood by those skilled in the art that Fig.14 The structure shown is for illustration only, and the computer terminal A may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), a PAD, or other terminal devices. Fig.14 It does not limit the structure of the above-mentioned computer terminal A. For example, the computer terminal A may also include Fig.14 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Fig.14 Different configurations are shown.

[0205] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0206] Example 6

[0207] The embodiment of the present invention further provides a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store the program code executed by the method for processing data in the model provided in the first embodiment.

[0208] Optionally, in this embodiment, the computer-readable storage medium may be located in any one of the computer terminals in a computer terminal group in a computer network, or in any one of the mobile terminals in a mobile terminal group.

[0209] Optionally, in this embodiment, the computer-readable storage medium is configured to store program codes for executing the information stored in the memory and the application program executed by the processor through the transmission device.

[0210] In an embodiment of the present invention, parameter information stored in an unstructured sparse matrix format is obtained, the parameter matrix is ​​decompressed and processed, and calculations are performed based on the decompressed parameter information and input data to obtain output data of a large model. Since the unstructured sparsely stored matrix occupies less memory space, the technical effect of reducing the space consumed when storing parameters of a large model is achieved, thereby solving the technical problem of excessive space consumption in storing model parameters.

[0211] Example 7

[0212] An embodiment of the present invention may provide an electronic device, which may include a memory and a processor.

[0213] Fig.15 1 is a block diagram of an electronic device for a method of processing data in a model according to an embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0214] like Fig.15 As shown, the device 1500 includes a computing unit 1501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1502 or a computer program loaded from a storage unit 1508 to a random access memory (RAM) 1503. In RAM 1503, various programs and data required for the operation of the device 1500 can also be stored. The computing unit 1501, ROM 1502, and RAM 1503 are connected to each other via a bus 1504. An input / output (I / O) interface 1505 is also connected to the bus 1504.

[0215] A number of components in the device 1500 are connected to the I / O interface 1505, including: an input unit 1506, such as a keyboard, a mouse, etc.; an output unit 1504, such as various types of displays, speakers, etc.; a storage unit 1508, such as a disk, an optical disk, etc.; and a communication unit 15010, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 15010 allows the device 1500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0216] The computing unit 1501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 1501 performs the various methods and processes described above, such as a processing method for data in a model. For example, in some embodiments, the processing method for data in a model may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1508. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1500 via the ROM 1502 and / or the communication unit 15010. When the computer program is loaded into RAM 1503 and executed by the computing unit 1501, one or more steps of the method for processing data in the model described above may be performed. Alternatively, in other embodiments, the computing unit 1501 may be configured to execute the method for processing data in the model in any other appropriate manner (e.g., by means of firmware).

[0217] Various embodiments of the systems and techniques described above herein may be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard parts (ASSPs), system on chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0218] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0219] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0220] To provide interaction with a user, the systems and techniques described herein may be implemented on a computer having: a display device (e.g., a cathode ray tube or liquid crystal display, monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or path ball) through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0221] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local area networks, wide area networks, and the Internet.

[0222] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0223] It should be noted that the serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0224] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0225] In the several embodiments provided by the present invention, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0226] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0227] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0228] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, read-only memory random access memories, mobile hard disks, magnetic disks or optical disks.

[0229] The above are only preferred embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for processing data in a model, characterized in that: include: Reading parameter information of the model from an off-chip storage space, wherein the parameter information is stored in blocks in an unstructured sparse matrix format, and the model is obtained based on machine learning training; Decompressing the parameter information of the model and storing it in a corresponding shared memory, wherein the decompressed parameter information is stored in a structured dense matrix format; Based on the decompressed parameter information, output data matching the input data is determined, wherein the output data is used to represent a dialog result generated by the model processing the input data.

2. The method according to claim 1, characterized in that Reading the parameter information of the model from the off-chip storage space includes: The block offset array and the non-zero element array of the model corresponding to the thread block in the image processor are read from the off-chip storage space.

3. The method according to claim 1, characterized in that Decompressing the parameter information of the model and storing it in the corresponding shared memory includes: Based on the block offset array in the parameter information, determine the starting position of the non-zero elements in the parameter information in the non-zero element array; Based on the starting position, determining the value and position information of the non-zero element in the non-zero element array; Based on the numerical value and the position information, construct the parameter information in a structured dense matrix format; The parameter information in the structured dense matrix format is stored in the corresponding shared memory.

4. The method according to claim 3, characterized in that Based on the value and the position information, constructing the parameter information in a structured dense matrix format includes: Based on the numerical value and the position information, construct the parameter information in a structured sparse matrix format; The parameter information in the structured sparse matrix format is filled with a target value to obtain the parameter information in the structured dense matrix format.

5. The method according to claim 1, characterized in that Determining the output data matching the input data based on the decompressed parameter information includes: Extracting input features from the input data, wherein the input data is determined based on the dialogue instruction; Performing iterative processing based on the decompressed parameter information and the input features to determine output features; The output data is determined based on the output characteristics.

6. The method according to claim 1, characterized in that The method further comprises: Dividing the original parameter information of the model into at least one block of sub-parameter information; The sub-parameter information is converted into the parameter information.

7. The method according to claim 6, characterized in that Converting the sub-parameter information into the parameter information includes: Determine an initial index position corresponding to the sub-parameter information and an array of non-zero elements corresponding to the sub-parameter information; Based on the initial index position and the array of non-zero elements, the parameter information corresponding to the sub-parameter information is determined.

8. The method according to claim 7, characterized in that Determining the initial index position corresponding to the sub-parameter information includes: Determining the initial index position of the sub-parameter information based on a sum of non-zero elements in the sub-parameter information, wherein the initial index position is stored in a block offset array, and the length of the block offset array is the same as the number of blocks of the sub-parameter information in the original parameter information; Determining the parameter information corresponding to the sub-parameter information based on the initial index position and the non-zero element array includes: determining the block offset array and the non-zero element array storing the initial index position as the parameter information corresponding to the sub-parameter information.

9. The method according to claim 7, characterized in that: Determining the non-zero element array corresponding to the sub-parameter information includes: The non-zero elements in the sub-parameter information and the positions of the non-zero elements in the sub-parameter information are stored to obtain the non-zero element array.

10. The method according to claim 9, characterized in that The non-zero elements in the sub-parameter information and the positions of the non-zero elements in the sub-parameter information are stored to obtain the non-zero element array, including: Determine the position of the non-zero element in the sub-parameter information and the relative distance between the non-zero element and the first element in the sub-parameter information; The non-zero elements in the sub-parameter information and the relative distances corresponding to the non-zero elements are stored to obtain the non-zero element array.

11. The method according to claim 6, characterized in that The method further comprises: The elements in the same sub-parameter information are continuously stored in the same off-chip storage space, and the elements in different sub-parameter information are stored in different off-chip storage spaces.

12. A method for processing data in a model, characterized in that: include: Reading parameter information of the model from an off-chip storage space by calling a first interface, wherein the first interface includes a first parameter, a parameter value of the first parameter is the parameter information, the parameter information is stored in blocks in an unstructured sparse matrix format, and the model is obtained based on machine learning training; Decompressing the parameter information of the model and storing it in a corresponding shared memory, wherein the decompressed parameter information is stored in a structured dense matrix format; Based on the decompressed parameter information, determining output data matching the input data, wherein the output data is used to represent a dialog result generated by the model processing the input data; The determined output data is output by calling a second interface, wherein the second interface includes a second parameter, and a parameter value of the second parameter is the output data.

13. A method for processing data in a model, characterized in that: include: Acquire multimodal information input in the interactive interface, wherein the type of the multimodal information includes at least one of the following: text information including character information, video frame information including frame image information, and audio information; Reading parameter information of the model from an off-chip storage space, wherein the parameter information is stored in blocks in an unstructured sparse matrix format, and the model is obtained based on machine learning training; Decompressing the parameter information of the model and storing it in a corresponding shared memory, wherein the decompressed parameter information is stored in a structured dense matrix format; Determine, based on the decompressed parameter information, reply information matching the multimodal information, wherein the reply information is used to characterize reply content generated by the model processing the multimodal information, and the type of the reply information includes at least one of the following: text information, image information, and voice information; The reply information is displayed in the interactive interface.

14. A system for processing data in a model, characterized in that: include: A thread block is used to read parameter information of a model from an off-chip storage space and store the parameter information in a register, wherein the parameter information is stored in blocks in an unstructured sparse matrix format, and the model is obtained based on machine learning training; An on-chip memory decompresses and stores the parameter information of the model, wherein the decompressed parameter information is stored in a structured dense matrix format; A computing component is used to determine output data matching the input data based on the decompressed parameter information, wherein the output data is used to represent a dialog result generated by the model processing the input data.

15. An electronic device, characterized in that: include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of any one of methods 1-13 are implemented.

Images