Large-scale pre-training model compression and image data processing method and device

Abstract

The embodiment of the present application provides a large-scale pre-trained model compression and image data processing method and device, which constructs weight constraints, sparsity constraints and rank constraints according to the weights, preset sparsity and preset rank of the trained large-scale pre-trained model, and compresses the weights into low-rank parts and unstructured sparse parts using the constructed weight constraints, sparsity constraints and rank constraints to obtain a compressed large-scale pre-trained model. By combining unstructured sparsity and low-rank decomposition methods to compress the weights of large-scale pre-trained models, the amount of compression calculations is reduced and the model compression rate is improved.

Description

Technical Field

[0001] The present application relates to the technical field of neural network models, and in particular to methods and devices for large-scale pre-training model compression and image data processing. Background Art

[0002] The large scale of LPMs (Large scale Pre-trained Models) leads to huge costs in building such large scale pre-trained models, which makes it difficult to deploy them on some edge devices. Currently, the difficulty of deploying LPMs on edge devices can be reduced by compressing the LPMs model.

[0003] In the related art, compression of LPMs is achieved by pruning. Specifically, the fine-tuning rounds of iterative pruning and the required number of pruning are set; the importance of each parameter in the weight of the large-scale pre-trained model is efficiently evaluated on the data set; according to the importance of each parameter, a structured parameter group is constructed and the group importance is calculated, and the group importance threshold is set according to the pruning number, and the parameter group below the group importance threshold is pruned; the parameters of the large-scale pre-trained model after pruning the parameter group are efficiently fine-tuned on the data set; the above parameter evaluation and pruning steps are repeated until the compression rate corresponding to the pruning number is met. After the above steps, the pruned large-scale pre-trained model is obtained. However, the model compression effect achieved by the above scheme is limited, and the pruning algorithm has a large amount of computation. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a large-scale pre-training model compression and image data processing method and device to solve at least one of the above problems. The specific technical solution is as follows:

[0005] In a first aspect, an embodiment of the present application provides a large-scale pre-trained model compression method, the method comprising:

[0006] Obtain the weights, preset sparsity, and preset rank of the trained large-scale pre-trained model;

[0007] Constructing a weight constraint condition according to the weight, wherein the weight constraint condition is used to constrain an unstructured sparse part and a low-rank part;

[0008] Constructing a sparsity constraint condition according to the preset sparsity, wherein the sparsity constraint condition is used to constrain the sparsity of the unstructured sparse part;

[0009] Constructing a rank constraint condition according to the preset rank, wherein the rank constraint condition is used to constrain the rank of the low-rank part;

[0010] The weight constraint, the sparsity constraint and the rank constraint are used as constraints to compress the weight into a low-rank part and an unstructured sparse part to obtain a compressed large-scale pre-training model.

[0011] In a possible embodiment, the weight constraint condition is expressed as:

[0012] ;

[0013] Among them, X represents the input data of the large-scale pre-training model, W represents the weight, L represents the low-rank part, and S represents the unstructured sparse part.

[0014] In a possible embodiment, the sparsity constraint condition is expressed as:

[0015] stCard(S)≤k

[0016] Wherein, k represents the preset sparsity, and S represents the unstructured sparse part.

[0017] In a possible embodiment, the rank constraint condition is expressed as:

[0018] strank(L)≤r

[0019] Among them, r represents the preset rank, and L represents the low rank part.

[0020] In a possible embodiment, using the weight constraint, the sparsity constraint and the rank constraint as constraints to compress the weight into a low-rank part and an unstructured sparse part to obtain a compressed large-scale pre-training model includes:

[0021] Acquire a current value of the unstructured sparse part, wherein an initial value of the unstructured sparse part is a first preset value;

[0022] Determine the current value of the low-rank part according to the current value of the unstructured sparse part and the weight, using the weight constraint and the rank constraint;

[0023] According to the current value of the low-rank part and the weight, using the weight constraint and the sparsity constraint, updating the current value of the unstructured sparse part;

[0024] Return to the execution step: according to the current value of the unstructured sparse part and the weight, use the weight constraint and the rank constraint to determine the current value of the low-rank part until the preset number of returns is reached or the compression loss is less than a second preset value, and a compressed large-scale pre-trained model is obtained, wherein the compression loss represents the difference between the weight and the sum of the two parts, and the sum of the two parts is the sum of the low-rank part and the unstructured sparse part, and the weight of the compressed large-scale pre-trained model is represented by the low-rank part and the unstructured sparse part.

[0025] In a possible embodiment, determining the current value of the low-rank part according to the current value of the unstructured sparse part and the weight using the weight constraint and the rank constraint includes:

[0026] Acquire preset input data, calculate first data after the weight acts on the input data, and calculate second data after the current value of the unstructured sparse part acts on the input data;

[0027] According to the first data and the second data, third data is calculated using the weight constraint condition;

[0028] Performing singular value decomposition on the third data to obtain fourth data;

[0029] The fourth data is restored using the input data to obtain a current value of the low-rank part.

[0030] In a possible embodiment, updating the current value of the unstructured sparse part according to the current value of the low-rank part and the weight by using the weight constraint and the sparsity constraint includes:

[0031] According to the current value of the low-rank part and the weight, a current representation function of the weight constraint is obtained;

[0032] The current representation function is simplified using the Taylor expansion formula to obtain the current simplified function;

[0033] The sparsity constraint condition is used as a constraint, and the current simplification function is trimmed using a second-order Hessian matrix and a preset index to obtain a current value of the unstructured sparse part.

[0034] In a second aspect, an embodiment of the present application provides an image data processing method, the method comprising:

[0035] Obtaining image data to be processed and a compressed large-scale pre-trained model, wherein the compressed large-scale pre-trained model is obtained by compression processing using any large-scale pre-trained model compression method described in any of the embodiments of the present application;

[0036] The image data to be processed is input into the compressed large-scale pre-trained model to obtain an image processing result.

[0037] In a third aspect, an embodiment of the present application provides a large-scale pre-trained model compression device, the device comprising:

[0038] An acquisition module, used to obtain the weight, preset sparsity and preset rank of the trained large-scale pre-trained model;

[0039] A first construction module is used to construct a weight constraint condition according to the weight, wherein the weight constraint condition is used to constrain an unstructured sparse part and a low-rank part;

[0040] A second construction module is used to construct a sparsity constraint condition according to the preset sparsity, wherein the sparsity constraint condition is used to constrain the sparsity of the unstructured sparse part;

[0041] A third construction module is used to construct a rank constraint condition according to the preset rank, wherein the rank constraint condition is used to constrain the rank of the low-rank part;

[0042] A compression module is used to use the weight constraint, the sparsity constraint and the rank constraint as constraints to compress the weight into a low-rank part and an unstructured sparse part to obtain a compressed large-scale pre-trained model.

[0043] In a possible embodiment, the weight constraint condition is expressed as:

[0044] ;

[0045] Among them, X represents the input data of the large-scale pre-training model, W represents the weight, L represents the low-rank part, and S represents the unstructured sparse part.

[0046] In a possible embodiment, the sparsity constraint condition is expressed as:

[0047] stCard(S)≤k

[0048] Wherein, k represents the preset sparsity, and S represents the unstructured sparse part.

[0049] In a possible embodiment, the rank constraint condition is expressed as:

[0050] strank(L)≤r

[0051] Among them, r represents the preset rank, and L represents the low rank part.

[0052] In a possible embodiment, the compression module is used to:

[0053] Acquire a current value of the unstructured sparse part, wherein an initial value of the unstructured sparse part is a first preset value;

[0054] Determine the current value of the low-rank part according to the current value of the unstructured sparse part and the weight, using the weight constraint and the rank constraint;

[0055] According to the current value of the low-rank part and the weight, using the weight constraint and the sparsity constraint, updating the current value of the unstructured sparse part;

[0056] Return to the execution step: according to the current value of the unstructured sparse part and the weight, use the weight constraint and the rank constraint to determine the current value of the low-rank part until the preset number of returns is reached or the compression loss is less than a second preset value, and a compressed large-scale pre-trained model is obtained, wherein the compression loss represents the difference between the weight and the sum of the two parts, and the sum of the two parts is the sum of the low-rank part and the unstructured sparse part, and the weight of the compressed large-scale pre-trained model is represented by the low-rank part and the unstructured sparse part.

[0057] In a possible embodiment, the compression module is used to:

[0058] Acquire preset input data, calculate first data after the weight acts on the input data, and calculate second data after the current value of the unstructured sparse part acts on the input data;

[0059] According to the first data and the second data, third data is calculated using the weight constraint condition;

[0060] Performing singular value decomposition on the third data to obtain fourth data;

[0061] The fourth data is restored using the input data to obtain a current value of the low-rank part.

[0062] In a possible embodiment, the compression module is used to:

[0063] According to the current value of the low-rank part and the weight, a current representation function of the weight constraint is obtained;

[0064] The current representation function is simplified using the Taylor expansion formula to obtain the current simplified function;

[0065] The sparsity constraint condition is used as a constraint, and the current simplification function is trimmed using a second-order Hessian matrix and a preset index to obtain a current value of the unstructured sparse part.

[0066] In a fourth aspect, an embodiment of the present application provides an image data processing device, the device comprising:

[0067] An acquisition module, used to acquire the image data to be processed and the compressed large-scale pre-trained model, wherein the compressed large-scale pre-trained model is obtained based on the large-scale pre-trained model device in the embodiment of the present application;

[0068] An input module is used to input the image data to be processed into the compressed large-scale pre-trained model to obtain an image processing result.

[0069] In a fifth aspect, an embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the large-scale pre-trained model compression or image data processing method described in any one of the embodiments of the present application is implemented.

[0070] In a sixth aspect, an embodiment of the present application further provides an electronic device, comprising: a memory for storing computer programs; and a processor for executing the programs stored in the memory to implement any of the large-scale pre-trained model compression or image data processing methods described in the embodiments of the present application.

[0071] Beneficial effects of the embodiments of the present application: The large-scale pre-trained model compression and image data processing method and device provided by the embodiments of the present application construct weight constraints, sparsity constraints and rank constraints according to the weights, preset sparsity and preset rank of the trained large-scale pre-trained model, and use the constructed weight constraints, sparsity constraints and rank constraints to compress the weights into low-rank parts and unstructured sparse parts to obtain a compressed large-scale pre-trained model. By combining unstructured sparsity and low-rank decomposition methods to compress the weights of large-scale pre-trained models, the amount of compression calculations is reduced and the model compression rate is improved.

[0072] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0074] Figure 1 A schematic diagram of a large-scale pre-trained model compression method provided for related technologies;

[0075] Figure 2 A first schematic diagram of a large-scale pre-trained model compression method provided in an embodiment of the present application;

[0076] Figure 3 A schematic diagram of dividing the weight W into a low-rank part and an unstructured sparse part provided in an embodiment of the present application;

[0077] Figure 4 A second schematic diagram of the large-scale pre-trained model compression method provided in an embodiment of the present application;

[0078] Figure 5 A first schematic diagram of the image data processing method provided in an embodiment of the present application;

[0079] Figure 6 A first schematic diagram of an image data processing method provided in the related art;

[0080] Figure 7 A second schematic diagram of the image data processing method provided in an embodiment of the present application;

[0081] Figure 8 A second schematic diagram of an image data processing method provided in the related art;

[0082] Fig. 9 A third schematic diagram of the image data processing method provided in an embodiment of the present application;

[0083] Fig.10 A schematic diagram of a large-scale pre-trained model compression device provided in an embodiment of the present application;

[0084] Fig.11 A schematic diagram of an image data processing device provided in an embodiment of the present application;

[0085] Fig.12 A schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0086] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0087] Below, the terms appearing in this application are explained:

[0088] Weight sparse technology: is an optimization technology that speeds up the reasoning process by increasing the proportion of zero elements in the weight.

[0089] Low-rank decomposition technology: It is a machine learning technology used to adapt the model to specific tasks or fields while maintaining its original performance. The core idea is to introduce auxiliary features into the feature space of the model. These auxiliary features are used to capture information about specific tasks or fields. These auxiliary features are combined with the original features through low-rank decomposition to achieve fine-tuning of the model.

[0090] Low-rank decomposition efficient fine-tuning technology: Freeze the weights of large-scale pre-trained models and inject the trainable rank decomposition matrix into each weight of the Transformer layer in the large-scale pre-trained model, greatly reducing the number of trainable parameters for downstream tasks.

[0091] LPMs have demonstrated excellent performance and have shown outstanding capabilities in many tasks. For example, LPMs are used in video surveillance application scenarios for abnormal behavior detection. For example, LPMs are used in the intelligent recognition system of the kitchen to monitor anomalies in real time, such as whether the chef's movements are standardized, whether the chef is wearing a chef's hat, whether the trash can is overflowing, and whether there are animals such as cockroaches and rats in the kitchen. Of course, LPMs can also be applied in application scenarios such as customer conversations, machine learning, and automatic video generation. However, due to the large scale of LPMs, the construction of LMPS requires huge costs, which makes it difficult to deploy it on some edge devices. At present, the difficulty of deploying LPMs on edge devices can be reduced by compressing the LPMs model.

[0092] like Figure 1 As shown, in the related technology, the fine-tuning rounds of iterative pruning and the target pruning number are set; the importance of each parameter in the large-scale pre-trained model is efficiently evaluated on the data set; according to the importance of each parameter, a structured parameter group is constructed and the group importance is calculated, and the group importance threshold is set according to the target pruning number, and the parameter group below the group importance threshold is pruned; the parameters of the large-scale pre-trained model after pruning the parameter group are efficiently fine-tuned on the data set; it is determined whether the target pruning number is met, and if so, the pruned large-scale pre-trained model is obtained; if not, the above parameter evaluation and pruning steps are repeated until the target pruning number is met, and the pruned large-scale pre-trained model is obtained. However, the model compression effect achieved by the above scheme is limited. Usually, the amount of data of the pruned LPMs is about 70% of the original LPMs, and the pruning algorithm is very computationally intensive. Therefore, a more efficient large-scale pre-trained model compression method is urgently needed.

[0093] In order to solve the technical problems existing in the related technologies, this application proposes a large-scale pre-training model compression method, such as Figure 2 As shown, the method specifically comprises the following steps:

[0094] S201: Obtain the weight, preset sparsity, and preset rank of the trained large-scale pre-trained model.

[0095] Large-scale pre-trained models have become one of the most commonly used technologies in deep learning applications. Although they have performed well in fields such as computer vision, natural language processing, and speech processing, deploying them in production environments still faces many challenges. For example, large-scale pre-trained models are usually very large and may contain hundreds of billions of parameters. They may face difficulties in storage and transmission when deployed; the model inference speed is slow; large-scale pre-trained models require a lot of computing resources for inference, which may lead to computing bottlenecks and performance problems when deployed. Therefore, it is necessary to compress large-scale pre-trained models to reduce the difficulty of deploying large-scale pre-trained models on edge devices.

[0096] The weights of the large-scale pre-trained model after training are the uncompressed weights, which are represented by W. The preset sparsity and the preset rank are set by the user according to actual needs. The preset sparsity can be represented by k, and the preset rank can be represented by r.

[0097] S202: Constructing weight constraints according to the weights, wherein the weight constraints are used to constrain the unstructured sparse part and the low-rank part.

[0098] When compressing a large-scale pre-trained model, a weight constraint condition can be constructed according to the weight, and then the weight of the large-scale pre-trained model can be compressed based on the weight constraint condition. The weight constraint condition is used to constrain the unstructured sparse part and the low-rank part, that is, the weight is represented by the unstructured sparse part and the low-rank part, so the unstructured sparse part and the low-rank part can be constrained according to the weight.

[0099] In a possible embodiment, the weight constraint condition is expressed as:

[0100] ;

[0101] Where X represents the input data of the large-scale pre-trained model, such as image data or speech data, and X is usually in the form of a feature matrix or a feature vector; W represents the weight, L represents the low-rank part, and S represents the unstructured sparse part. Represents the minimum zero norm.

[0102] The embodiment of the present application realizes model compression through the model unstructured sparse method and low-rank decomposition method. The specific principle is to express the weight W into two parts, the low-rank part and the unstructured sparse part. That is, . Figure 3 A schematic diagram of dividing the weight W into a low-rank part and an unstructured sparse part provided in an embodiment of the present application. Figure 3 As shown in FIG. 1 , the weight W is divided into a low-rank part L and an unstructured sparse part S, and the low-rank part includes an upsampling matrix V and a downsampling matrix U. Input X to a large-scale pre-trained model with weight W to obtain Y. Input X to a large-scale pre-trained model with weights including a low-rank part L and an unstructured sparse part S to obtain Y. That is, compressing the large-scale pre-trained model does not affect its output results.

[0103] S203: Constructing a sparsity constraint condition according to a preset sparsity, wherein the sparsity constraint condition is used to constrain the sparsity of the unstructured sparse part.

[0104] When compressing a large-scale pre-trained model, a sparsity constraint condition can be constructed according to a preset sparsity, and then the weight of the large-scale pre-trained model can be compressed based on the sparsity constraint condition.

[0105] In a possible embodiment, the sparsity constraint condition is expressed as:

[0106] stCard(S)≤k

[0107] Among them, k represents the preset sparsity, S represents the unstructured sparse part, st is used to indicate subject to constraints, and Card() is used to indicate the number of elements in a finite set.

[0108] S204: Construct a rank constraint condition according to a preset rank, wherein the rank constraint condition is used to constrain the rank of the low-rank part.

[0109] When compressing a large-scale pre-trained model, a rank constraint condition can be constructed according to a preset rank, and then the rank constraint condition is used to compress the weights of the large-scale pre-trained model.

[0110] In a possible embodiment, the rank constraint is expressed as:

[0111] strank(L)≤r

[0112] Among them, r represents the preset rank, L represents the low rank part, st is used to represent the constraint, and rank() is used to represent the rank of the matrix.

[0113] S205: Using weight constraints, sparsity constraints and rank constraints as constraints, the weights are compressed into a low-rank part and an unstructured sparse part to obtain a compressed large-scale pre-training model.

[0114] In a possible embodiment, weight constraints, sparsity constraints, and rank constraints are used as constraints to compress weights into low-rank parts and unstructured sparse parts to obtain a compressed large-scale pre-trained model, including:

[0115] Obtain the current value of the unstructured sparse part, wherein the initial value of the unstructured sparse part is a first preset value; determine the current value of the low-rank part according to the current value and weight of the unstructured sparse part using weight constraints and rank constraints; update the current value of the unstructured sparse part according to the current value and weight of the low-rank part using weight constraints and sparsity constraints; return to the execution step: determine the current value of the low-rank part according to the current value and weight of the unstructured sparse part using weight constraints and rank constraints, until the preset number of returns is reached or the compression loss is less than a second preset value, and obtain a compressed large-scale pre-trained model, wherein the compression loss represents the difference between the weight and the sum of the two parts, the sum of the two parts is the sum of the low-rank part and the unstructured sparse part, and the weight of the compressed large-scale pre-trained model is represented by the low-rank part and the unstructured sparse part.

[0116] For example, the initial value of the unstructured sparse part may be 0 or other values, which is not limited in this application.

[0117] A possible embodiment, according to the current value and weight of the unstructured sparse part, uses weight constraints and rank constraints to determine the current value of the low-rank part, including: obtaining preset input data, calculating the first data after the weight acts on the input data, and calculating the second data after the current value of the unstructured sparse part acts on the input data; according to the first data and the second data, using the weight constraints to calculate the third data; performing singular value decomposition on the third data to obtain fourth data; and using the input data to restore the fourth data to obtain the current value of the low-rank part.

[0118] It should be noted that the preset input data is obtained based on the specific task to be performed by the large-scale pre-trained model. That is, the acquisition of the preset input data depends on the application scenario of the large-scale pre-trained model.

[0119] For example, when the application scenario of the large-scale pre-trained model is an abnormal behavior detection application scenario, the preset input data is a real-time scene image and a user-defined requirement. The user-defined requirement can be called a prompt word.

[0120] In large-scale pre-trained model ensembles, prompts are text or instructions that provide input to the model to guide the model to generate a specific type of response. Prompts can be a question, a description, a task description, or even a portion of a conversation history. The structure of prompts can be simple or complex, and can include keywords, constraints, example inputs and outputs, etc.

[0121] For example, when the application scenario of the large-scale pre-trained model is a customer service dialogue application scenario, the preset input data is user input and tokenization. Tokens refer to symbols used in large-scale pre-trained models to represent Chinese characters, English words, or Chinese and English phrases. Tokens can be single characters or sequences of multiple characters.

[0122] In one example, according to the current value and weight of the unstructured sparse part, the current value of the low-rank part is determined by using the weight constraint and the rank constraint, including: obtaining the preset input data X, calculating the first data WX after the weight acts on the input data, and calculating the second data SX after the current value of the unstructured sparse part acts on the input data; according to the first data WX and the second data SX, using the weight constraint The third data is obtained by calculation; the third data is subjected to singular value decomposition to obtain the fourth data; the fourth data is restored using the input data to obtain the current value L of the low-rank part.

[0123] For example, if the weight W is a matrix with m rows and n columns, then S is also a matrix with m rows and n columns. During initialization, the current value of S can be set to 0. By performing singular value decomposition on the WS part, a matrix with m rows and r columns and a matrix with r rows and n columns can be obtained. By solving the matrix with m rows and r columns and the matrix with r rows and n columns, the current value L of the low-rank part can be obtained.

[0124] A possible implementation method is to update the current value of the unstructured sparse part according to the current value and weight of the low-rank part, using weight constraints and sparsity constraints, including: obtaining the current representation function of the weight constraint according to the current value and weight of the low-rank part; simplifying the current representation function using the Taylor expansion formula to obtain the current simplified function; using the sparsity constraint as a constraint, using the second-order Hessian matrix and preset indicators to trim the current simplified function to obtain the current value of the unstructured sparse part.

[0125] As an example, the current representation function of the weight constraint is:

[0126] ;

[0127] Among them, W represents weight, X represents preset input data, represents the current value of the low-rank part of the current iteration number, t represents the current iteration number, S represents the unstructured sparse part, represents the two-norm.

[0128] For example, the specific expression of the preset indicator is:

[0129] ;

[0130] Among them, W represents the weight, represents the current value of the low-rank part of the current iteration number, t represents the current iteration number, X represents the preset input data, diag() represents the diagonal matrix function, X T represents the transpose of X, represents a third preset value, and I represents a unit matrix.

[0131] For example, if the weight W is a matrix with m rows and n columns, the corresponding preset indicators are calculated for each element in the W matrix, and the preset indicators are sorted in order of size, and the preset indicators arranged within the previous preset sparsity range are retained to obtain the current value of the unstructured sparse part.

[0132] For an example, see Figure 4 , Figure 4 A schematic diagram of a large-scale pre-training model compression method provided in an embodiment of the present application, W represents the weight, t represents the current iteration number, S_t-1 represents the current value of the unstructured sparse part obtained in the previous iteration before the current iteration number, represents a minus sign, the first large dashed rectangle and the second large dashed rectangle are used to separate the process of obtaining the low-rank part from the process of obtaining the unstructured sparse part, the small dashed rectangle is used to indicate that the elements in W are trimmed in units of channels, D_low indicates that the rank in its channel is less than the original matrix, V_t represents the upsampling matrix, U_t is used to represent the downsampling matrix, × is the multiplication sign, the dashed ellipse represents the cycle, ΔL is the compression loss, S_t represents the current value of the unstructured sparse part obtained at the current iteration number, ||X_j|| is the norm form of the input;

[0133] Input preset input data, for example, ||X_j||, calculate W×||X_j|| to obtain the first data, calculate ||X_j||×S_t-1 to obtain the second data, subtract the second data from the first data to obtain the third data; perform singular value decomposition on the third data based on the scaling factor to obtain the fourth data; restore the fourth data using the inverse of the scaling factor to obtain the current value of the low-rank part. That is, V_t and U_t. Subtract the current value of the low-rank part from the first data to obtain the current value of the unstructured sparse part, that is, S_t. Repeat the above steps until the compression loss is less than the second preset value. The second preset value can be 0 or other values, which is not limited in this application.

[0134] The large-scale pre-trained model compression method provided in the embodiment of the present application constructs weight constraints, sparsity constraints and rank constraints according to the weights, preset sparsity and preset rank of the trained large-scale pre-trained model, and compresses the weights into low-rank parts and unstructured sparse parts using the constructed weight constraints, sparsity constraints and rank constraints to obtain a compressed large-scale pre-trained model. By combining unstructured sparsity and low-rank decomposition methods to compress the weights of large-scale pre-trained models, the data volume of compressed LPMs can be about 50% of the original LPMs, which improves the model compression rate, and the compression algorithm in the embodiment of the present application is simple to calculate, and its calculation amount is less than that of the pruning algorithm, which can reduce the calculation amount of LPMs compression.

[0135] The present application also provides an image data processing method, see Figure 5 , the method specifically comprises the following steps:

[0136] S501: Obtain image data to be processed and a compressed large-scale pre-trained model, where the compressed large-scale pre-trained model is obtained by any large-scale pre-trained model compression method described in any of the embodiments of the present application;

[0137] S502: Input the image data to be processed into the compressed large-scale pre-trained model to obtain the image processing result.

[0138] In the related art, when large-scale pre-trained models are applied to abnormal behavior detection scenarios, see Figure 6 ,exist Figure 6 In the process, real-time scene pictures and prompt words are first input, the real-time scene pictures are sorted and spliced, and the processed pictures are input into the large-scale pre-trained model without compression for inference calculation to obtain the output results.

[0139] For example, abnormal behavior detection is to detect the behavior of a chef not wearing a chef's hat. The real-time scene image is a picture of a chef working in the kitchen, and the prompt words are keywords, including: not wearing a chef's hat. The output result is a picture marked with a chef not wearing a chef's hat.

[0140] In the embodiment of the present application, when the large-scale pre-training model is applied to the abnormal behavior detection application scenario, see Figure 7 ,exist Figure 7 In the process, real-time scene pictures and prompt words are first input, the real-time scene pictures are sorted and spliced, and the processed pictures are input into the compressed large-scale pre-trained model for inference calculation to obtain the output results.

[0141] In the related technology, when large-scale pre-trained models are applied to customer service dialogue application scenarios, see Figure 8 ,exist Figure 8 In the process, user input is obtained and tokenized, and the tokenized user input is input to a large-scale pre-trained model that has not been compressed for inference calculation to obtain the output result.

[0142] In the embodiment of this application, when the large-scale pre-trained model is applied to the customer service dialogue application scenario, see Fig. 9 ,exist Fig. 9 In the process, user input is obtained and tokenized, and the tokenized user input is fed into a compressed large-scale pre-trained model for inference calculation to obtain the output result.

[0143] In the embodiment of the present application, the processing speed can be increased by using a compressed large-scale pre-trained model to process image data. This is because the weights of the compressed large-scale pre-trained model are represented by low-rank parts and unstructured sparse parts, and the number is about 50% of the original, so the time required to load the weights during the model operation can also be saved by 50%. Therefore, in theory, the speed of image data processing can be increased to 2 times the original speed. After actual testing, the speed of image data processing can be increased to 1.6-1.7 times the original speed, which greatly increases the image processing speed.

[0144] The present application also provides a large-scale pre-training model compression device, see Fig.10 , the device comprises:

[0145] An acquisition module 1001 is used to acquire the weight, preset sparsity and preset rank of the trained large-scale pre-trained model;

[0146] A first construction module 1002 is used to construct a weight constraint condition according to the weight, wherein the weight constraint condition is used to constrain the unstructured sparse part and the low-rank part;

[0147] A second construction module 1003 is used to construct a sparsity constraint condition according to a preset sparsity, wherein the sparsity constraint condition is used to constrain the sparsity of the unstructured sparse part;

[0148] The third constructing module 1004 is used to construct a rank constraint condition according to a preset rank, wherein the rank constraint condition is used to constrain the rank of the low-rank part;

[0149] The compression module 1005 is used to compress the weights into a low-rank part and an unstructured sparse part using weight constraints, sparsity constraints and rank constraints as constraints to obtain a compressed large-scale pre-trained model.

[0150] In a possible embodiment, the weight constraint is expressed as:

[0151] ;

[0152] Among them, X represents the input data of the large-scale pre-trained model, W represents the weight, L represents the low-rank part, and S represents the unstructured sparse part.

[0153] In a possible embodiment, the sparsity constraint is expressed as:

[0154] stCard(S)≤k

[0155] Wherein, k represents the preset sparsity, and S represents the unstructured sparse part.

[0156] In one possible embodiment, the rank constraint is expressed as:

[0157] strank(L)≤r

[0158] Among them, r represents the preset rank and L represents the low rank part.

[0159] In a possible embodiment, the compression module is used to:

[0160] Obtaining a current value of the unstructured sparse part, wherein an initial value of the unstructured sparse part is a first preset value;

[0161] According to the current value and weight of the unstructured sparse part, the current value of the low-rank part is determined by using the weight constraint and the rank constraint;

[0162] According to the current value and weight of the low-rank part, the current value of the unstructured sparse part is updated using the weight constraint and the sparsity constraint;

[0163] Return to the execution step: According to the current value and weight of the unstructured sparse part, the current value of the low-rank part is determined by using the weight constraint and the rank constraint, until the preset number of returns is reached or the compression loss is less than the second preset value, and the compressed large-scale pre-trained model is obtained, wherein the compression loss represents the difference between the weight and the sum of the two parts, and the sum of the two parts is the sum of the low-rank part and the unstructured sparse part, and the weight of the compressed large-scale pre-trained model is represented by the low-rank part and the unstructured sparse part.

[0164] In a possible embodiment, the compression module is used to:

[0165] Obtaining preset input data, calculating first data after the weight acts on the input data, and calculating second data after the current value of the unstructured sparse part acts on the input data;

[0166] The third data is calculated based on the first data and the second data using a weight constraint condition;

[0167] Performing singular value decomposition on the third data to obtain fourth data;

[0168] The fourth data is restored using the input data to obtain the current value of the low-rank part.

[0169] In a possible embodiment, the compression module is used to:

[0170] According to the current value and weight of the low-rank part, the current representation function of the weight constraint is obtained;

[0171] The current representation function is simplified using the Taylor expansion formula to obtain the current simplified function;

[0172] Taking the sparsity constraint as a constraint, the second-order Hessian matrix and the preset index are used to trim the current simplification function to obtain the current value of the unstructured sparse part.

[0173] The present application also provides an image data processing device, see Fig.11 , the device comprises:

[0174] An acquisition module 1101 is used to acquire data to be processed and a compressed large-scale pre-trained model, where the compressed large-scale pre-trained model is obtained based on the large-scale pre-trained model compression device in the embodiment of the present application;

[0175] The input module 1102 is used to input the data to be processed into the compressed large-scale pre-training model to obtain an output result.

[0176] An embodiment of the present application also provides an electronic device, comprising: a memory for storing computer programs; and a processor for executing the programs stored in the memory to implement any of the large-scale pre-trained model compression or image data processing methods described in the embodiments of the present application.

[0177] In an example, Fig.12 As shown, the memory may include: a main processor 1201, an external memory 1202 and a neural network processor 1203; the neural network processor 1203 includes: an input memory 12031, a weight memory 12032, an operation circuit 12033, a controller 12034, a storage unit access controller 12035, a unified memory 12036, a vector calculation unit 12037, an accumulator 12038, an instruction fetch memory 12039 and a bus interface unit 120310.

[0178] The main processor 1201 is used to execute machine codes, process data and improve computing efficiency.

[0179] The external memory 1202 is used to store programs and data that need to be kept online but are not used temporarily.

[0180] Input memory 12031 is used to store input data for processing by the neural network model.

[0181] The weight memory 12032 is used to store the weight parameters of the neural network model so that these weight parameters can be quickly and accurately applied for calculation when processing images or data.

[0182] The operation circuit 12033 is mainly used to implement mathematical operations of the neural network model, including multiplication and addition, so as to improve the accuracy of the neural network model, the accuracy of the output results, and reduce resource consumption.

[0183] Controller 12034, which plays the role of "brain", has intelligent features such as self-learning and self-adaptation, and is very suitable for controller design.

[0184] The storage unit access controller 12035 is mainly used for interface conversion, converting the read and write commands issued by the master device into signals that the memory can recognize, and completing address decoding and data format conversion between the master device and the memory.

[0185] Unified memory 12036 is used to manage and organize data for efficient neural network computing.

[0186] The vector computing unit 12037 is used to complete vector-related operations and can realize calculations between vectors and scalars, or between two vectors. Its functions cover various basic and customized calculation types.

[0187] The accumulator 12038 is used to store intermediate results generated by the calculation.

[0188] Instruction fetch memory 12039 is used to increase the operating speed of the processor and optimize program execution efficiency.

[0189] The bus interface unit 120310 is used to transfer data and instructions between the processor core and external components.

[0190] In another embodiment provided in the present application, a computer-readable storage medium is also provided, which stores a computer program. When the computer program is executed by a processor, the steps of any of the above-mentioned large-scale pre-trained model compression and image data processing methods are implemented.

[0191] In another embodiment provided in the present application, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute any of the large-scale pre-trained model compression and image data processing methods in the above-mentioned embodiments.

[0192] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.

[0193] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0194] The term "at least one" in this application means one or more, and the term "plurality" in this application means two or more, for example, a plurality of data packets means two or more data packets.

[0195] It should be understood that the terms used in the description of various examples herein are only for describing specific examples and are not intended to be limiting. As used in the description of various examples and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0196] It should also be understood that determining B based on A does not mean determining B only based on A. B can also be determined based on A and / or other information.

[0197] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the large-scale pre-trained model compression and image data processing device, electronic device, and storage medium embodiments, since they are basically similar to the large-scale pre-trained model compression and image data processing method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0198] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A large-scale pre-trained model compression method, characterized in that: The model is applied to abnormal behavior detection, and the method comprises: Obtain the weights, preset sparsity, and preset rank of the trained large-scale pre-trained model; Constructing a weight constraint condition according to the weight, wherein the weight constraint condition is used to constrain an unstructured sparse part and a low-rank part; Constructing a sparsity constraint condition according to the preset sparsity, wherein the sparsity constraint condition is used to constrain the sparsity of the unstructured sparse part; Constructing a rank constraint condition according to the preset rank, wherein the rank constraint condition is used to constrain the rank of the low-rank part; Acquire a current value of the unstructured sparse part, wherein an initial value of the unstructured sparse part is a first preset value; Determine the current value of the low-rank part according to the current value of the unstructured sparse part and the weight, using the weight constraint and the rank constraint; According to the current value of the low-rank part and the weight, using the weight constraint and the sparsity constraint, updating the current value of the unstructured sparse part; Return to the execution step: according to the current value of the unstructured sparse part and the weight, using the weight constraint and the rank constraint, determine the current value of the low-rank part until a preset number of returns is reached or the compression loss is less than a second preset value, and obtain a compressed large-scale pre-trained model; Wherein, determining the current value of the low-rank part according to the current value of the unstructured sparse part and the weight, using the weight constraint and the rank constraint, includes: Obtaining preset input data, calculating first data after the weight acts on the input data, and calculating second data after the current value of the unstructured sparse part acts on the input data; wherein the input data includes: a real-time scene picture and a prompt word, the real-time scene picture is a picture to be detected for abnormal behavior, and the prompt word is a text or instruction used to guide the model to generate a specific type of response; According to the first data and the second data, third data is calculated using the weight constraint condition; Performing singular value decomposition on the third data to obtain fourth data; Restoring the fourth data using the input data to obtain a current value of the low-rank part; The updating of the current value of the unstructured sparse part according to the current value of the low-rank part and the weight, using the weight constraint and the sparsity constraint, comprises: According to the current value of the low-rank part and the weight, a current representation function of the weight constraint is obtained; The current representation function is simplified using the Taylor expansion formula to obtain the current simplified function; The sparsity constraint condition is used as a constraint, and the current simplification function is trimmed using a second-order Hessian matrix and a preset index to obtain a current value of the unstructured sparse part.

2. The method according to claim 1, characterized in that: The weight constraint is expressed as: ； Among them, X represents the input data of the large-scale pre-training model, W represents the weight, L represents the low-rank part, and S represents the unstructured sparse part.

3. The method according to claim 1, characterized in that The sparsity constraint is expressed as: stCard(S)≤k Wherein, k represents the preset sparsity, and S represents the unstructured sparse part.

4. The method according to claim 1, characterized in that The rank constraint is expressed as: strank(L)≤r Among them, r represents the preset rank, and L represents the low rank part.

5. The method according to claim 1, characterized in that The compression loss represents the difference between the weight and the sum of two parts, where the sum of the two parts is the sum of the low-rank part and the unstructured sparse part, and the weight of the compressed large-scale pre-trained model is represented by the low-rank part and the unstructured sparse part.

6. A method for processing image data, characterized in that: The method is applied to abnormal behavior detection, and the method comprises: Obtaining image data to be processed and a compressed large-scale pre-trained model, wherein the compressed large-scale pre-trained model is obtained by compression processing using any of the methods described in claims 1-5, and the image data to be processed is a real-time scene picture to be subjected to abnormal behavior detection; The image data to be processed is input into the compressed large-scale pre-trained model to obtain an image processing result.

7. A large-scale pre-trained model compression device, characterized in that: The model is applied to abnormal behavior detection, and the device comprises: An acquisition module, used to obtain the weight, preset sparsity and preset rank of the trained large-scale pre-trained model; A first construction module is used to construct a weight constraint condition according to the weight, wherein the weight constraint condition is used to constrain an unstructured sparse part and a low-rank part; A second construction module is used to construct a sparsity constraint condition according to the preset sparsity, wherein the sparsity constraint condition is used to constrain the sparsity of the unstructured sparse part; A third construction module is used to construct a rank constraint condition according to the preset rank, wherein the rank constraint condition is used to constrain the rank of the low-rank part; A compression module is used to obtain the current value of the unstructured sparse part, wherein the initial value of the unstructured sparse part is a first preset value; according to the current value of the unstructured sparse part and the weight, the current value of the low-rank part is determined by using the weight constraint and the rank constraint; according to the current value of the low-rank part and the weight, the current value of the unstructured sparse part is updated by using the weight constraint and the sparsity constraint; return to the execution step: according to the current value of the unstructured sparse part and the weight, the current value of the low-rank part is determined by using the weight constraint and the rank constraint, until the preset number of returns is reached or the compression loss is less than the second preset value, and the compressed large-scale pre-trained model is obtained; Wherein, the compression module, when determining the current value of the low-rank part according to the current value of the unstructured sparse part and the weight, using the weight constraint and the rank constraint, is used to obtain preset input data, calculate the first data after the weight acts on the input data, and calculate the second data after the current value of the unstructured sparse part acts on the input data; wherein, the input data includes: a real-time scene picture and a prompt word, the real-time scene picture is a picture to be detected for abnormal behavior, and the prompt word is a text or instruction for guiding the model to generate a specific type of response; according to the first data and the second data, the weight constraint is used to calculate the third data; the third data is subjected to singular value decomposition to obtain the fourth data; the fourth data is restored using the input data to obtain the current value of the low-rank part; The compression module, when updating the current value of the unstructured sparse part according to the current value of the low-rank part and the weight, using the weight constraint and the sparsity constraint, is used to obtain the current representation function of the weight constraint according to the current value of the low-rank part and the weight; simplify the current representation function using the Taylor expansion formula to obtain the current simplified function; and use the sparsity constraint as a constraint to trim the current simplified function using the second-order Hessian matrix and preset indicators to obtain the current value of the unstructured sparse part.

8. An image data processing device, characterized in that: The device is applied to abnormal behavior detection, and comprises: An acquisition module, used for acquiring image data to be processed and a compressed large-scale pre-trained model, wherein the compressed large-scale pre-trained model is obtained based on the device according to claim 7, and the image data to be processed is a real-time scene picture to be subjected to abnormal behavior detection; An input module is used to input the image data to be processed into the compressed large-scale pre-trained model to obtain an image processing result.

Images