A method, device, and medium for constructing a large model
By combining block processing and 2D Kronecker decomposition with LoRA technology, the number of parameters in the large pathological vision-language model is reduced, solving the deployment problem of the model on edge devices and achieving efficient and low-cost pathological diagnosis support.
Patent Information
- Application Number
- CN202511330816.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-17
- Publication Date
- 2026-05-01
- Estimated Expiration
- 2045-09-17
AI Technical Summary
The existing pathological vision-language large model has a huge number of parameters, making it difficult to deploy it widely in general hospitals and on edge devices. In addition, the existing compression methods have limited compression rates and significant performance loss.
We employ a block-based processing approach combined with two-dimensional Kronecker decomposition and LoRA techniques. By dividing the Transformer weight matrix into blocks and performing low-rank approximation, we freeze the parameters of the smaller matrices and use LoRA to obtain additional parameters. We then train the compressed model using the dataset.
It effectively reduces the size of model parameters and resource requirements, maintains high diagnostic performance, is suitable for ordinary GPU or CPU devices, achieves a balance between high compression ratio and high accuracy, and supports widespread deployment on edge devices.
Smart Images

Figure CN121168556B_ABST
Abstract
Description
A method, apparatus, equipment, and medium for constructing a large model Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, and medium for constructing large models. Background Technology
[0002] In recent years, the rapid development of digital pathology has driven the widespread application of large-scale pathology vision-language models. Such models, such as the Prov-GigaPath model jointly released by Microsoft and Providence, have excellent diagnostic performance and can accurately classify tumors, identify subtypes, and predict molecular pathology. However, the number of parameters in Prov-GigaPath and similar models is enormous (in the hundreds of millions), and they usually require expensive and high-performance server-grade GPUs (such as NVIDIA A100) for inference, making it difficult to deploy them widely in ordinary hospitals and on edge devices.
[0003] Therefore, there is an urgent need in this field for an innovative compression method that can significantly reduce the parameter size and resource requirements of large pathological models while maintaining high diagnostic performance, especially supporting efficient inference deployment on ordinary GPUs or CPUs. Summary of the Invention
[0004] The purpose of this invention is to address the problem that large models in related technologies have excessively high processor performance requirements and are difficult to widely apply, by providing a method, apparatus, device, and medium for constructing large models. This method can reduce the parameter scale and resource requirements of existing large models, form applicable large models, effectively reduce performance loss, and facilitate widespread deployment on edge devices.
[0005] To achieve the above objectives, in a first aspect of the present invention, a method for constructing a large model is provided, comprising: S1, dividing the Transformer weight matrix of the target model into blocks to form multiple sub-matrices; S2, performing low-rank approximation on each sub-matrix through two-dimensional Kronecker decomposition to obtain the Kronecker product of the two smaller matrices corresponding to each sub-matrix; S3, freezing the parameters of all smaller matrices and using LoRA to obtain the additional parameters corresponding to all smaller matrices to obtain a two-dimensional Kronecker-LoRA compressed model; S4, using the target model as a teacher model and training the two-dimensional Kronecker-LoRA compressed model using a dataset to obtain a large model.
[0006] In a second aspect of the invention, a large model construction apparatus is provided, comprising: a weight partitioning module for partitioning the Transformer weight matrix of the target model into multiple sub-matrices; a two-dimensional Kronecker approximation module for performing low-rank approximation on each sub-matrix through two-dimensional Kronecker decomposition to obtain the Kronecker product of the two smaller matrices corresponding to each sub-matrix; a low-rank adaptation module for freezing the parameters of all smaller matrices and using LoRA to obtain the additional parameters corresponding to all smaller matrices to obtain a two-dimensional Kronecker-LoRA compressed model; and a distillation training module for using the target model as a teacher model and training the two-dimensional Kronecker-LoRA compressed model using a dataset to obtain a large model.
[0007] In a third aspect of the invention, an electronic device is provided, including a memory and a processor coupled to the memory, the processor being configured to perform the method of the first aspect based on instructions stored in the memory.
[0008] In a fourth aspect of the invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method of the first aspect.
[0009] The technical solution of this invention, based on the combination of block processing and two-dimensional Kronecker decomposition, reduces the matrix size of a single processing step, thereby effectively reducing the computational complexity of subsequent decomposition and fine-tuning. It is also more in line with the attention mechanism characteristics of Transformer, and can retain more global information of cross-token interaction. Furthermore, LORA is applied to the small matrix after Kronecker decomposition, and the approximate structure after decomposition is further adjusted through the low-rank matrix. That is, based on the synergistic effect of two-dimensional Kronecker decomposition and LORA, a compression-compensation closed loop is formed, avoiding the high computational cost of directly applying LoRA to the original large matrix, effectively reducing the number of parameters of the low-rank matrix, while maintaining parameter efficiency. Finally, student training is used to enhance the compensation effect of model performance, achieving a balance between high compression ratio and high accuracy, and enabling deep adaptation of dynamic inference and image processing workflow. Attached Figure Description
[0010] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0011] Figure 1 is a flowchart of the method for constructing the large model described in the embodiment;
[0012] Figure 2 is a flowchart of the two-dimensional Kronecker decomposition described in the embodiment;
[0013] Figure 3 is a schematic diagram of the structure of the large model construction device described in the embodiment;
[0014] Figure 4 is a schematic diagram of the structure of the electronic device described in the embodiment. Detailed Implementation
[0015] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.
[0016] Existing compression methods (such as pruning, quantization, knowledge distillation, etc.) usually do not take into account the special structure of the weight matrix of pathological models, resulting in limited compression ratios and significant performance loss. Meanwhile, although the existing LoRA (low-rank adaptation) technique can reduce fine-tuning parameters, it does not fully utilize the weight structure for further optimization. In addition, Kronecker low-rank approximation has shown initial success in language model compression, but its potential has not yet been explored in the field of large-scale pathological visual models.
[0017] This application is described below with reference to the accompanying drawings and specific embodiments.
[0018] As shown in Figure 1, a method for constructing a large model in this embodiment includes: S1, dividing the Transformer weight matrix of the target model into blocks to form multiple sub-matrices; S2, performing low-rank approximation on each sub-matrix using two-dimensional Kronecker decomposition to obtain the Kronecker product of the two smaller matrices corresponding to each sub-matrix; S3, freezing the parameters of all smaller matrices and using LoRA to obtain the additional parameters corresponding to all smaller matrices to obtain a two-dimensional Kronecker-LoRA compressed model; S4, using the target model as the teacher model and training the two-dimensional Kronecker-LoRA compressed model using the dataset to obtain a large model.
[0019] The target model can be an existing pathological model to be compressed and optimized, such as Microsoft's Prov-GigaPath model. Existing large pathological models are Transformer models, and the core parameters are usually high-dimensional matrices. Therefore, the computational complexity and storage cost of directly performing low-rank decomposition on high-dimensional matrices are still high. Conventional compression methods (such as pruning, quantization, knowledge distillation, etc.) do not consider the special structure of the weight matrix of the pathological model, resulting in limited compression ratio and significant performance loss.
[0020] In step S2, the two-dimensional Kronecker decomposition is a special case of K-FAC (Kronecker-Factored Approximate Curvature) decomposition, which is suitable for image data processing. The creative application of two-dimensional Kronecker decomposition to the block processing of the Transformer weight matrix can effectively utilize the correlation of pathological data (such as the local texture of tissue slices) to achieve efficient compression, which is especially suitable for the processing of medical image data. Compared with traditional low-rank decomposition (such as SVD) which only captures the correlation in a single row and column direction, the Kronecker product can simultaneously model the row-column intersection pattern and capture the global dependency pattern of the weight matrix, which is more suitable for the low-rank structure of the high-dimensional Transformer matrix, while retaining more semantic information.
[0021] In step S3, LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning technique. By introducing a low-rank matrix for parameter adjustment and freezing the parameters of all small matrices after the two-dimensional Kronecker decomposition, it effectively reduces the number of parameters during fine-tuning, thereby lowering computational and storage costs. Simultaneously, it avoids or reduces structural biases that may be introduced by Kronecker decomposition, and strives to restore the fine-grained feature representation capabilities lost due to compression.
[0022] In step S4, the two-dimensional Kronecker-LoRA compressed model is used as a student model for distillation training. By combining it with a dataset, specifically a professional pathology dataset, domain knowledge can be injected to optimize the model's specialized capabilities and improve its pathology recognition performance.
[0023] This embodiment combines block processing and two-dimensional Kronecker decomposition to reduce the matrix size of a single processing step, thereby effectively reducing the computational complexity of subsequent decomposition and fine-tuning. It also better aligns with the attention mechanism characteristics of Transformer, preserving more global information from cross-token interactions. Furthermore, LORA is applied to the smaller matrix after Kronecker decomposition, and the approximate structure after decomposition is further adjusted through a low-rank matrix. This forms a compression-compensation closed loop based on the synergistic effect of two-dimensional Kronecker decomposition and LORA, avoiding the high computational cost of directly applying LoRA to the original large matrix. This effectively reduces the number of parameters in the low-rank matrix while maintaining parameter efficiency. Finally, student training is used to enhance the compensation effect of model performance, achieving a balance between high compression ratio and high accuracy, and enabling deep adaptation of dynamic inference and pathological image processing workflows.
[0024] In some optional implementations, in step S1, the Transformer weight matrix of the target model is divided into blocks to form multiple sub-matrices, including:
[0025] Obtain the attention entropy value of each row in the Transformer weight matrix, determine the high-entropy and low-entropy rows of the Transformer weight matrix based on the first threshold, and determine the size of the submatrix based on the proportion of high-entropy and low-entropy rows in the submatrix.
[0026] Compared to processing blocks of fixed size, this method quantifies the importance of tokens by using attention entropy values, thereby achieving adaptive blocking. This enables on-demand blocking with overall balance, significantly reducing the number of blocks while preserving key interaction information. For example, if a submatrix contains a high proportion of high-entropy rows, the submatrix can use smaller blocks, such as 64*64 submatrixes, to retain fine-grained interaction information. If a submatrix contains a high proportion of low-entropy rows, the submatrix can use larger blocks, such as 258*258 submatrixes, to reduce the amount of block data and computation.
[0027] The dividing line between large and small blocks is determined based on the working conditions and there are different judgment criteria. It can be determined according to the actual situation. The specific proportion judgment criteria can be determined according to the actual situation. For example, if the proportion of high-entropy rows is set to be greater than 50% of the second threshold, then it is considered that the proportion of high-entropy rows is relatively high. The first threshold will also be determined according to the actual situation. When the attention entropy value of a row is greater than or equal to the first threshold, it can be identified as a high-entropy row, and otherwise it is a low-entropy row.
[0028] In some optional implementations, in step S3, LoRA is used to obtain the additional parameters corresponding to all small matrices to obtain a two-dimensional Kronecker-LoRA compression model, including:
[0029] Based on the Kronecker product of the two smaller matrices corresponding to each submatrix, LoRA is used to add the corresponding low-rank matrix, and lightweight fine-tuning is performed using the training dataset. The fine-tuned low-rank matrix is then used as the corresponding additional parameter.
[0030] The training dataset can use general datasets as needed, such as publicly available large-scale tissue slice images, or specific pathological datasets, such as datasets for specific scenarios like tumor grading. The fine-tuned low-rank matrix is used as the corresponding additional parameter. That is, different sub-matrices correspond to their own low-rank matrices. Therefore, after lightweight fine-tuning, multiple low-rank matrices are used as additional parameters for different sub-matrices, which can effectively compensate for the performance loss caused by compression.
[0031] In some alternative implementations, LoRA is used to add the corresponding low-rank matrix for lightweight fine-tuning based on the Kronecker product of the two smaller matrices corresponding to each submatrix, including:
[0032] The rank of a low-rank matrix is determined by the proportion of high-entropy rows and low-entropy rows in the submatrix, and the rank of the low-rank matrix is directly proportional to the proportion of high-entropy rows.
[0033] Compared to LORA, which uses a fixed rank to obtain additional parameters, the dynamic LORA mechanism achieves on-demand fine-tuning by sensing the attention region, which can further reduce the number of parameters to be fine-tuned while improving model performance.
[0034] In some optional implementations, in step S2, each submatrix is approximated by a low-rank decomposition using two-dimensional Kronecker decomposition to obtain the Kronecker product of the two smaller matrices corresponding to each submatrix, including:
[0035] The submatrices are rearranged according to the preset Kronecker factor dimensions to generate a rearranged matrix; the rearranged matrix is subjected to singular value decomposition to obtain singular vectors; the singular vectors are rearranged inversely to obtain a Kronecker factor matrix; the Kronecker factor matrices are combined to obtain the Kronecker product of the two smaller matrices.
[0036] Compared to using fixed rank for two-dimensional Kronecker decomposition, the combination of matrix rearrangement and singular value decomposition enables each submatrix to minimize decomposition error while retaining the main information, effectively reducing accuracy loss and storage requirements. In addition, least squares optimization can be performed on the weighted submatrix to find the optimal small matrix.
[0037] In some optional implementations, in step S4, the target model is used as the teacher model, and the two-dimensional Kronecker-LoRA compressed model is used as the student model. Training of the two-dimensional Kronecker-LoRA compressed model is then performed using the dataset, including:
[0038] The dataset is input into the target model, and the output is used as the soft label supervision signal for distillation. At least one of KL divergence loss, cosine similarity loss, and L2 loss is used as the distillation loss function. The low-rank matrix is trained by optimizing the distillation loss function to complete the parameter optimization of the two-dimensional Kronecker-LoRA compression model.
[0039] The training objects for the student model can include either the low-rank matrix added by LoRA or the parameters frozen in step S3. Furthermore, by selecting only the low-rank matrix as the training object for the student model, that is, by continuing to keep all small matrix parameters frozen, the amount of parameter training can be effectively reduced during the distillation process, and the computational power requirement can be significantly reduced.
[0040] Among them, KL divergence loss (KL divergence loss), also known as relative entropy, is used to measure the difference between two probability distributions. By minimizing the KL divergence, it makes the distribution of the student model closer to that of the teacher model. Cosine similarity loss is used to measure the directional similarity between two vectors, which can make the output vector of the student model closer to that of the teacher model. L2 loss (mean squared error) is used to calculate the average of the squares of the differences between corresponding elements of two vectors. It focuses on the absolute difference in numerical values, penalizes large errors, and is suitable for scenarios that require smooth optimization.
[0041] Depending on the operating environment of the large model, the GPU or CPU is dynamically selected to perform inference. In some optional implementations, this may include step S5, which dynamically selects the GPU or CPU to perform inference based on the resolution of the input image and in combination with two-dimensional Kronecker decomposition.
[0042] Through dynamic inference deployment technology, it can automatically adapt to GPU or CPU execution in different resource environments. Specifically, it can automatically select whether to execute inference on GPU (when under high load) or CPU (when under low load or when GPU is unavailable) by monitoring indicators such as GPU memory usage and computing latency in real time.
[0043] Further applied to medical scenarios, this mechanism can automatically select between block-based parallel computation on the GPU (for high-resolution images) or serial processing on the CPU (for low-resolution images or when GPU memory is insufficient) by real-time monitoring of the resolution and GPU memory usage of the input pathological images. Compared to general dynamic inference methods, this scenario-aware inference mechanism can effectively reduce inference latency in pathological diagnosis tasks. Combined with the block-based computation characteristics of Kronecker decomposition, the CPU can further optimize inference efficiency by performing multi-threaded parallel processing of submatrix operations. This enables on-demand resource allocation, i.e., real-time switching or allocation of computation tasks based on device resource load, to ensure the efficiency and real-time nature of the inference process, thus adapting to various hardware deployment scenarios.
[0044] Based on the specific requirements of the working conditions, a certain selection and combination of various optional implementation methods were made to form the following implementation method, which is to take the Prov-GigaPath model as the target model, divide the Transformer weight matrix of the Prov-GigaPath model into rows and columns according to a predetermined size such as 256×256, so that the original large-scale weight matrix is split into multiple sub-matrices of moderate size.
[0045] Thus, a unified and standardized block-sharing strategy is proposed for the weights of pathological Transformer models. The block-sharing method is simple to implement, highly versatile, and applicable to various Transformer models with different architectures. At the same time, it provides a structured foundation for subsequent Kronecker approximation, significantly improving the approximation accuracy and efficiency.
[0046] For each submatrix obtained by splitting, a low-rank approximation is performed using two-dimensional Kronecker decomposition. Each 256×256 submatrix can be approximately decomposed into two 16×16 matrix Kronecker products for representation, thereby significantly reducing storage parameters, storage requirements and computational costs.
[0047] Furthermore, the fixed-rank two-dimensional Kronecker decomposition may lead to over- or under-decomposition. To efficiently find the optimal small matrix and achieve significant parameter compression, an adaptive rank selection based on the singular value distribution of the submatrices can be used. Each submatrix is dynamically assigned the optimal decomposition rank, as shown in Figure 2. Specifically, based on the target Kronecker factor dimensions A1, B1, C1, and D1, the original 256×256 matrix is rearranged into a rearranged matrix. Here, the 256×256 submatrices are initially set as the original matrix, and the goal is to find two low-rank matrices F∈R. A1×C1 G∈R B1×D1 This allows for a 256×256 submatrix ≈ F×G, satisfying dimension matching: A1×B1=256, C1×D1=256. The purpose of matrix rearrangement is to rearrange the elements of the submatrix according to the inherent structure of the Kronecker product, constructing a rearranged matrix E such that it can be approximately represented as the outer product of vectors F and G, facilitating subsequent SVD decomposition. The constructed rearranged matrix E∈R (A1*C1)×(B1*D1)That is, the rearranged matrix E is equivalent to the outer product of the column vectors of two smaller matrices. Since the rearranged matrix E is a rank-1 matrix (if F and G are full rank), but in practice the submatrices may not be able to be precisely decomposed into Kronecker products, the rearranged matrix E needs to be approximated by a low-rank matrix. SVD is the best tool for low-rank approximation. Then, singular value decomposition is performed on the rearranged matrix to extract its principal components, thereby obtaining the maximum singular value and the corresponding left and right singular vectors. The left and right singular vectors are then rearranged in reverse to form Kronecker factor matrices. The maximum singular value is absorbed into one of the factors and combined to form the final Kronecker product approximation.
[0048] Thus, applying the two-dimensional Kronecker low-rank approximation method in the field of pathological vision Transformer effectively utilizes the inherent tensor structure of the weight matrix of the pathological model, significantly compressing the parameter scale. It is significantly superior to existing simple low-rank or LoRA adaptation methods, combining the advantages of compression and performance preservation.
[0049] Furthermore, based on freezing the two-dimensional Kronecker approximation weights, i.e., keeping the parameters of the Kronecker product of all 16×16 matrices unchanged, a low-rank matrix is introduced into each submatrix using the LoRA technique. For example, a low-rank matrix with a rank of 8 is added. Specifically, this can be expressed by the following formula: W = B×C + W1 = B×C + A*D, where W is a 256×256 submatrix, B×C is the Kronecker product of matrix B and matrix C, and B∈R 16×16 , C∈R 16×16 Let W1 = A * D be a low-rank matrix, where A is a left-hand low-rank factor matrix and D is a right-hand low-rank factor matrix. If the rank of W1 is 8, then A ∈ R. 16×8 , D∈R 8×16 During optimization, the original matrix B×C is kept frozen, and only the parameters of A and D are adjusted. Specifically, a pathological image dataset is used to train the low-rank matrix, thereby achieving efficient fine-tuning training for each sub-matrix. The low-rank matrix after training is the additional parameter corresponding to each sub-matrix. Through LoRA fine-tuning, the performance of the compressed model can be effectively restored while controlling the computational amount of parameters.
[0050] Then, a knowledge distillation framework is adopted, using the original Prov-GigaPath model as the teacher model and the two-dimensional Kronecker-LoRA compressed model as the student model. The parameters of the student model are optimized using a pathology dataset. Specifically, the pathology dataset is input into the target model, and the output is used as the soft label supervision signal for distillation. KL divergence loss, cosine similarity loss, and L2 loss are all used as distillation loss functions. For example, the final distillation loss can be determined through weighted calculation. The low-rank matrix is trained by optimizing the distillation loss function. Specifically, the parameters of the low-rank matrix are updated through backpropagation to complete the parameter optimization of the two-dimensional Kronecker-LoRA compressed model. That is, during distillation, the student model not only learns the soft labels of the teacher model but also the attention entropy distribution of the teacher model.
[0051] Finally, during application inference, the GPU or CPU will be automatically and dynamically selected to perform inference based on the actual operating environment, adapting to various hardware deployment scenarios. Specifically, it can be based on the resolution of the input pathological image and combined with two-dimensional Kronecker decomposition to dynamically select the GPU or CPU to perform inference.
[0052] Furthermore, when dividing the Transformer weight matrix of the Prov-GigaPath model into rows and columns, we can first obtain the attention entropy value of each row in the Transformer weight matrix. Specifically, each row of the attention matrix is normalized using Softmax to obtain the probability distribution, so the entropy formula can be: , where H(a) i ) is the attention entropy value of the i-th row, a ij It is the attention weight in the i-th row and j-th column. By calculating the entropy value of each row, the degree of attention dispersion of each token can be quantified. If the entropy value of a row in the matrix is greater than or equal to the preset first threshold, it is determined to be a high-entropy row. If the entropy value of a row is less than the first threshold, it is determined to be a low-entropy row. The first threshold can be the average of the attention entropy values obtained from all rows.
[0053] Then, when dividing the submatrix, the proportion of high-entropy rows and low-entropy rows is determined. For example, if the proportion of high-entropy rows in the submatrix exceeds 50%, a 64×64 matrix is used to preserve fine-grained interaction information as much as possible. If the proportion of high-entropy rows in the submatrix is less than 50%, a 512×512 matrix can be used to reduce the overall number of blocks and the amount of computation.
[0054] Furthermore, when introducing a low-rank matrix using the LORA technique, the rank of the low-rank matrix can be determined based on the proportion of high-entropy rows and low-entropy rows within the submatrix. For example, if the proportion of high-entropy rows in the submatrix exceeds 50%, a low-rank matrix with a higher rank, such as a low-rank matrix with a rank of 16, is introduced. Conversely, if the proportion of high-entropy rows in the submatrix is less than 50%, a low-rank matrix with a lower rank, such as a low-rank matrix with a rank of 8, is introduced. Alternatively, the original frozen weights can be used directly without introducing a low-rank matrix. This allows for on-demand fine-tuning through the focus of the attention region, reducing the number of fine-tuning parameters while maximizing model performance.
[0055] Furthermore, the LORA technology introduces a low-rank matrix, which can determine the rank based on the difficulty of the sample. Difficult samples have higher rank, while easy samples have lower rank. The difficulty of a sample can be determined by the teacher model to predict the confidence level of the sample. When the confidence level is lower than a preset threshold, the sample is marked as a difficult sample, thus improving the classification accuracy of difficult samples.
[0056] The large model construction method in this embodiment, through block-based approach and Kronecker low-rank reconstruction, can compress Prov-GigaPath-like vision-language large models from hundreds of millions of parameters to less than 1 byte of parameters, achieving a compression rate of over 85%, while maintaining an accuracy loss of less than 0.5%. The low-rank approximation combined with LoRA technology ensures that the performance of the distilled large model is close to that of the target model, with the loss controlled within the required range. Memory consumption is reduced to below 6GB, allowing ordinary medical institutions to complete inference tasks using conventional GPUs or CPUs, significantly reducing costs. The Kronecker product structure enables efficient matrix operations, suitable for CPU / GPU deployment on edge devices. Supporting GPU / CPU hybrid inference, it is suitable for efficient deployment in hospitals and medical edge devices, and can be widely applied in hospital digital pathology recognition scenarios, effectively lowering the deployment threshold for pathology models, promoting the industrialization of digital pathology and smart healthcare, and yielding significant economic and social benefits. Furthermore, it is not only applicable to the pathology field but can also be extended to other medical vision Transformers or language models with similar structures.
[0057] The following describes a large model construction apparatus provided by an embodiment of this application. The apparatus described below can be referred to in correspondence with the method described above. Based on the above embodiment, Figure 3 is a structural schematic diagram of a large model construction apparatus provided by this embodiment.
[0058] As shown in Figure 3, a large model construction device includes: a weight partitioning module 10, used to partition the Transformer weight matrix of the target model into multiple sub-matrices; a two-dimensional Kronecker approximation module 20, used to perform low-rank approximation on each sub-matrix through two-dimensional Kronecker decomposition to obtain the Kronecker product of the two smaller matrices corresponding to each sub-matrix; a low-rank adaptation module 30, used to freeze the parameters of all smaller matrices and use LoRA to obtain the additional parameters corresponding to all smaller matrices to obtain a two-dimensional Kronecker-LoRA compressed model; and a distillation training module 40, used to use the target model as a teacher model and use the dataset to train the two-dimensional Kronecker-LoRA compressed model to obtain a large model.
[0059] In addition, it may include a dynamic GPU / CPU hybrid inference module 50, which dynamically selects GPU or CPU to perform inference based on the resolution of the input pathological image and in combination with two-dimensional Kronecker decomposition. That is, it will dynamically select GPU or CPU to perform inference according to the running environment of the large model.
[0060] Based on the above method embodiments, please refer to Figure 4, which is a schematic diagram of the structure of an electronic device provided in the embodiments of this application. An electronic device 300 provided in the embodiments of this application includes: a processor 301 and a memory 302. The memory 302 stores machine-readable instructions that can be executed by the processor 301. When the machine-readable instructions are executed by the processor 301, the above method is performed. The electronic device can be a physical device.
[0061] Based on the above method embodiments, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the method as in any of the above embodiments.
[0062] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0063] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more flowchart illustrations and / or one or more block diagrams.
[0064] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.
[0065] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.
[0066] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0067] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.
[0068] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.
[0069] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.
[0070] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0071] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.
Claims
1. A method for constructing large models for image processing, characterized in that, include: S1. The Transformer weight matrix of the target model is divided into blocks to form multiple sub-matrices. The attention entropy value of each row in the Transformer weight matrix is obtained. A first threshold is used to determine the high-entropy and low-entropy rows of the Transformer weight matrix. The size of the sub-matrix is determined based on the proportion of high-entropy and low-entropy rows in each sub-matrix. S2. Each sub-matrix is approximated using a low-rank method through two-dimensional Kronecker decomposition to obtain the Kronecker product of the two smaller matrices corresponding to each sub-matrix. The sub-matrices are rearranged according to a preset Kronecker factor dimension to generate a rearranged matrix. Singular value decomposition is performed on the rearranged matrix to obtain singular vectors. The singular vectors are then rearranged inversely to form a Kronecker factor matrix, which is then combined with the Kronecker factor matrix. The necker factor matrix yields the Kronecker product of two smaller matrices; S3, all parameters of the smaller matrices are frozen, and LoRA is used to obtain the additional parameters corresponding to all smaller matrices to obtain a two-dimensional Kronecker-LoRA compressed model. Based on the Kronecker product of the two smaller matrices corresponding to each submatrix, LoRA is used to add the corresponding low-rank matrix, and lightweight fine-tuning is performed using the training dataset. The rank of the low-rank matrix is determined based on the proportion of high-entropy and low-entropy rows in the submatrix, and the rank of the low-rank matrix is proportional to the proportion of high-entropy rows. The fine-tuned low-rank matrix is used as the corresponding additional parameter; S4, the target model is used as the teacher model, and the two-dimensional Kronecker-LoRA compressed model is trained using the dataset to obtain a large model.
2. The method for constructing a large image processing model according to claim 1, characterized in that, In step S4, the target model is used as the teacher model, and the two-dimensional Kronecker-LoRA compressed model is used as the student model. The training of the two-dimensional Kronecker-LoRA compressed model is completed using the dataset, including: inputting the dataset into the target model, using the obtained output as the soft label supervision signal for distillation, using at least one of KL divergence loss, cosine similarity loss, and L2 loss as the distillation loss function, and training the low-rank matrix by optimizing the distillation loss function to complete the parameter optimization of the two-dimensional Kronecker-LoRA compressed model.
3. The method for constructing a large image processing model according to claim 1, characterized in that, It also includes step S5, which dynamically selects the GPU or CPU to perform inference based on the resolution of the input image and in combination with two-dimensional Kronecker decomposition.
4. An apparatus for constructing large models for image processing, characterized in that, include: The weight partitioning module divides the Transformer weight matrix of the target model into multiple sub-matrices. It obtains the attention entropy value of each row in the Transformer weight matrix, determines the high-entropy and low-entropy rows based on a first threshold, and determines the size of each sub-matrix based on the proportion of high-entropy and low-entropy rows. The two-dimensional Kronecker approximation module performs low-rank approximation on each sub-matrix using two-dimensional Kronecker decomposition, obtaining the Kronecker product of the two smaller matrices corresponding to each sub-matrix. The sub-matrices are then rearranged according to a preset Kronecker factor dimension to generate a rearranged matrix. Singular value decomposition is performed on the rearranged matrix to obtain singular vectors, and these singular vectors are then rearranged inversely to form a Kronecker factor matrix. The Kronecker factor matrix is combined to obtain the Kronecker product of two smaller matrices. A low-rank adaptation module freezes the parameters of all smaller matrices and uses LoRA to obtain the additional parameters corresponding to all smaller matrices, resulting in a two-dimensional Kronecker-LoRA compressed model. Specifically, based on the Kronecker product of the two smaller matrices corresponding to each submatrix, LoRA is used to add the corresponding low-rank matrix, and lightweight fine-tuning is performed using the training dataset. The rank of the low-rank matrix is determined based on the proportion of high-entropy and low-entropy rows in the submatrix, and the rank of the low-rank matrix is proportional to the proportion of high-entropy rows. The fine-tuned low-rank matrix is used as the corresponding additional parameter. A distillation training module uses the target model as the teacher model and trains the two-dimensional Kronecker-LoRA compressed model using the dataset to obtain a larger model.
5. An electronic device, characterized in that, It includes a memory and a processor coupled to the memory, the processor being configured to perform the method as described in any one of claims 1 to 3 based on instructions stored in the memory.
6. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 3.
Citation Information
Patent Citations
Rapid low-memory image compression perception method
CN107170019A
Large model LoRA fine tuning improvement method and device based on tensor parallelism and medium
CN118133927A
