Intelligent health large model fine-tuning method and device based on gradient iterative pruning

Abstract

This invention discloses a method and apparatus for fine-tuning a large-scale smart health model based on gradient iterative pruning, relating to the field of model fine-tuning. The method includes: obtaining the updated weight matrix of the large-scale smart health model to be fine-tuned, performing singular value decomposition to obtain a first low-rank approximation matrix and a second low-rank approximation matrix, and iteratively updating the cumulative importance of each element. After multiple iterations, a first mask matrix and a second mask matrix are constructed, and iterative pruning is performed on the first and second low-rank approximation matrices respectively. An adaptive weight allocation algorithm with contribution calibration is used to calculate the first and second weight proportion coefficients, obtaining a calibration weight matrix. A conflict suppression regularization term, a key position constraint regularization term, and a sparse merging evaluation regularization term are introduced to construct a joint loss function. Fine-tuning is performed based on the joint loss function to obtain the fine-tuned large-scale smart health model. This invention solves the problems of element redundancy and weight distribution imbalance in low-rank matrices.

Description

Technical Field

[0001] This invention relates to the field of model fine-tuning, specifically to a method and apparatus for fine-tuning a large-scale smart health model based on gradient iterative pruning. Background Technology

[0002] The efficient deployment of large-scale smart health models is a crucial research direction within the smart home health service system. With the development of smart home terminals, mobile health devices, and wearable devices, large-scale smart health models are gradually extending their deployment from the cloud to the edge and terminal sides. A large-scale smart health model is a large-scale language model built for home health service scenarios. It can be used for tasks such as health Q&A and multi-turn interactive companionship. Its input can include health consultation text, text or voice interaction processed by the front end, and its output includes health advice, risk warnings, and decision support results. Collaborative deployment on the terminal side refers to deploying model capabilities on demand in the cloud, edge, or terminal devices, achieving collaborative reasoning and service response while balancing privacy and real-time performance.

[0003] Existing efficient parameter fine-tuning methods for large-scale models often employ low-rank update mechanisms to achieve task adaptation with a relatively small number of trainable parameters. However, in the collaborative deployment of smart health large-scale models on both the device and edge, low-rank updates still suffer from issues such as redundancy of low-rank matrix elements and imbalance in the weight distribution of update components. Specifically, element redundancy in the low-rank matrix increases the model training workload; weight bias across matrices can easily cause fusion interference, reducing stability and reliability under different conditions for different populations and devices. Summary of the Invention

[0004] The purpose of this application is to propose a method and apparatus for fine-tuning a smart health model based on gradient iterative pruning, which addresses the aforementioned technical problems.

[0005] In a first aspect, the present invention provides a method for fine-tuning a large-scale intelligent health model based on gradient iterative pruning, comprising the following steps:

[0006] The updated weight matrix of the smart health model to be fine-tuned is obtained and subjected to singular value decomposition to construct a first low-rank approximation matrix and a second low-rank approximation matrix. Based on the task loss function of the current round of the smart health model during fine-tuning, the importance of each element in the first and second low-rank approximation matrices for the current round is calculated using the gradients of these matrices. The cumulative importance of each element in the first and second low-rank approximation matrices for the current round is iteratively updated. After multiple iterations, a first mask matrix and a second mask matrix are constructed. Based on the first and second mask matrices, the gradients of the first and second low-rank approximation matrices are then applied to the first and second low-rank approximation matrices. The matrix is ​​iteratively pruned to obtain the first pruning weight matrix and the second pruning weight matrix. Based on the first pruning weight matrix and the second pruning weight matrix, the first weight ratio coefficient and the second weight ratio coefficient are calculated using an adaptive weight allocation algorithm with contribution calibration. The calibration weight matrix is ​​calculated based on the first weight ratio coefficient, the second weight ratio coefficient, the first pruning weight matrix and the second pruning weight matrix. The conflict suppression regularization term, the key position constraint regularization term and the sparse merging evaluation regularization term are introduced based on the calibration weight matrix. A joint loss function is constructed by combining the task loss function. The first low-rank approximation matrix and the second low-rank approximation matrix are fine-tuned based on the joint loss function to obtain the fine-tuned smart health big model.

[0007] The input text or voice is obtained and input into the fine-tuned smart health model. The calibration weight matrix is ​​calculated using the first low-rank approximation matrix and the second low-rank approximation matrix in the fine-tuned smart health model. The fusion weight matrix is ​​calculated based on the calibration weight matrix. The forward propagation calculation is performed using the fusion weight matrix to obtain the response statement.

[0008] Preferably, the updated weight matrix of the smart health model to be fine-tuned is obtained, singular value decomposition is performed, and a first low-rank approximation matrix and a second low-rank approximation matrix are constructed, specifically including:

[0009] The updated weight matrix is ​​subjected to singular value decomposition to obtain the left singular vector matrix, the singular value diagonal matrix, and the right singular vector matrix, as shown in the following equation:

[0010] ;

[0011] in, This indicates updating the weight matrix. Describes a left singular vector matrix. Represents a singular value diagonal matrix. This represents the first to the rth singular values. Represents a diagonal matrix. Describes a right singular vector matrix. Represents the transpose of a matrix;

[0012] By coupling the singular value diagonal matrix with the left and right singular vector matrices respectively in square root form, we obtain the first low-rank approximation matrix and the second low-rank approximation matrix, as shown in the following equation:

[0013] ;

[0014] ;

[0015] in, Denotes the first low-rank approximation matrix. This represents the second low-rank approximation matrix.

[0016] Preferably, based on the task loss function of the smart health model during fine-tuning, the importance of each element in the first and second low-rank approximation matrices for the current round is calculated using the gradient of the first and second low-rank approximation matrices. The cumulative importance of each element in the first and second low-rank approximation matrices for the current round is then iteratively updated. After multiple iterations, a first mask matrix and a second mask matrix are constructed. Based on the first and second mask matrices, iterative pruning is performed on the first and second low-rank approximation matrices to obtain a first pruning weight matrix and a second pruning weight matrix, specifically including:

[0017] Let the task loss function of the aforementioned smart health model during the fine-tuning process be... ;

[0018] The importance of each element in the first and second low-rank approximation matrices for the current round is obtained by calculating the first Taylor expansion of the task loss function with respect to the first or second low-rank approximation matrix. The calculation process is as follows:

[0019] ;

[0020] ;

[0021] in, Describes the first low-rank approximation matrix. OK The importance of column elements In the second low-rank approximation matrix, the first... OK The importance of column elements Describes the first low-rank approximation matrix The OK Column elements, Describes the second low-rank approximation matrix The OK Column elements, Indicates the first The task loss function of the round of iterations approximates the first low-rank matrix. The OK The partial derivatives of the elements of the column, Indicates the first The task loss function for each round approximation matrix is ​​applied to the second low-rank matrix. The OK Partial derivatives of the elements of the column;

[0022] The cumulative importance of each element in the first and second low-rank approximation matrices of the previous round is updated by using the importance of each element in the first and second low-rank approximation matrices of the current round, as shown in the following formula:

[0023] ;

[0024] ;

[0025] in, Indicates the first The first low-rank approximation matrix of each round The Middle OK The cumulative importance of the elements in a column. Indicates the first The first low-rank approximation matrix of each round The Middle OK The cumulative importance of the elements in a column. Indicates the first The second low-rank approximation matrix for each round The Middle OK The cumulative importance of the elements in a column. Indicates the first The second low-rank approximation matrix for each round The Middle OK The cumulative importance of the elements in the column;

[0026] go through After the nth iteration, the nth and nth results are obtained respectively. The first low-rank approximation matrix of each round The Middle OK Cumulative importance of column elements and the The second low-rank approximation matrix for each round The Middle OK Cumulative importance of column elements ;

[0027] Set pruning threshold ,in accordance with and Construct the first mask matrix Second mask matrix The calculation process is as follows:

[0028] ;

[0029] ;

[0030] in, Represents the first mask matrix The Middle OK Column elements, Represents the second mask matrix The Middle OK Column elements;

[0031] Using the first mask matrix For the first low-rank approximation matrix Pruning is performed to obtain the first pruning weight matrix; a second mask matrix is ​​then used. For the second low-rank approximation matrix Pruning is performed to obtain the second pruning weight matrix, as shown in the following formula:

[0032] ;

[0033] ;

[0034] in, This represents the Hadamard product. This represents the first pruning weight matrix. This represents the second pruning weight matrix.

[0035] Preferably, the first weight ratio coefficient and the second weight ratio coefficient are calculated using an adaptive weight allocation algorithm with contribution calibration based on the first pruning weight matrix and the second pruning weight matrix, specifically including:

[0036] Taking the absolute values ​​of the first pruning weight matrix and the second pruning weight matrix respectively, we obtain the first magnitude matrix. Second magnitude matrix ;

[0037] The first magnitude matrix was selected multiple times using a non-replacement method. Second magnitude matrix The indices corresponding to the maximum values ​​of the elements in the set constitute the first index set. Second index set As shown in the following formula:

[0038] ;

[0039] ;

[0040] ;

[0041] ;

[0042] in, Indicates the first The first magnitude matrix is ​​selected next. The first index set is obtained by taking the index corresponding to the maximum value of the elements in the set. Indicates the first The second magnitude matrix is ​​selected next. The second set of indices is obtained by taking the index corresponding to the maximum value of the elements in the set. Indicates the first The first magnitude matrix is ​​selected next. The index corresponding to the maximum value of the elements in the array. Indicates the first The second magnitude matrix is ​​selected next. The index corresponding to the maximum value of the elements in the array. , This indicates the total number of selections. This indicates the index corresponding to the maximum value;

[0043] Contribution aggregation is performed on the first index set and the second index set respectively to obtain the first contribution metric and the second contribution metric, as shown in the following formula:

[0044] ;

[0045] ;

[0046] in, and These represent the first contribution metric and the second contribution metric, respectively.

[0047] The balance factor is calculated based on the first pruning weight matrix and the second pruning weight matrix, as shown in the following formula:

[0048] ;

[0049] in, This represents the summation of all elements of a matrix. As the first stable term, Indicates the balance factor;

[0050] The first and second contribution measures are calibrated using a balance factor to obtain the first and second calibrated contribution measures, as shown in the following formula:

[0051] ;

[0052] ;

[0053] in, and These represent the first calibration contribution metric and the second calibration contribution metric, respectively.

[0054] The first weighting coefficient and the second weighting coefficient are calculated based on the first calibration contribution metric and the second calibration contribution metric, as shown in the following formula:

[0055] ;

[0056] ;

[0057] in, and These represent the first weighting coefficient and the second weighting coefficient, respectively. , It is the second stable term.

[0058] Preferably, the formula for calculating the calibration weight matrix is ​​as follows:

[0059] ;

[0060] in, Represents the calibration weight matrix. Indicates the global scaling factor;

[0061] The formula for calculating the fusion weight matrix is ​​as follows:

[0062] ;

[0063] in, Represents the fusion weight matrix. This represents the original weight matrix that is frozen in the smart health model to be fine-tuned.

[0064] Preferably, a conflict suppression regularization term, a key location constraint regularization term, and a sparse merging evaluation regularization term are introduced based on the calibration weight matrix, and a joint loss function is constructed by combining it with the task loss function, specifically including:

[0065] For calibration weight matrix Set the preset sparsity Only the first one is retained. Important elements generate sparse mask matrix As shown in the following formula:

[0066] ;

[0067] in, This indicates that the calibration weight matrix is ​​adjusted according to a preset sparsity rate. Operators for generating sparse mask matrices,

[0068] The sparse fusion matrix is ​​calculated based on the sparse mask matrix and the calibration weight matrix. As shown in the following formula:

[0069] ;

[0070] The sparsity ratio of the matrix is ​​calculated based on the sparse fusion matrix, as shown in the following formula:

[0071] ;

[0072] in, Represents the sparsity of a matrix. These represent the total number of rows and columns of the sparse fusion matrix, respectively. express Pseudonorm;

[0073] The importance score for each element is calculated based on the calibration weight matrix, as shown in the following formula:

[0074] ;

[0075] in, Represents the first element in the calibration weight matrix. OK The importance score of the elements in the column. Represents the first element in the calibration weight matrix. OK Column elements, Represents the L2 norm. Represents a constant;

[0076] The retention mask matrix is ​​calculated based on the importance score, as shown in the following formula:

[0077] ;

[0078] in, Indicates hyperparameters, Represents the retention mask matrix The Middle OK Column elements;

[0079] The conflict suppression regularization term is constructed based on the retention mask matrix and the sparse fusion matrix, as shown in the following equation:

[0080] ;

[0081] in, This indicates a conflict suppression regularization term. Describing the Frobenius norm, Represents the Hadamard product;

[0082] The key position constraint regularization term is constructed based on the sparse mask matrix and the calibration weight matrix, as shown in the following equation:

[0083] ;

[0084] in, express Norm, Indicates the key position constraint regularization term;

[0085] The sparse merging evaluation regularization term is constructed based on the matrix sparsity rate and the preset sparsity rate, as shown in the following formula:

[0086] ;

[0087] in, This represents the regularization term for sparse merging evaluation;

[0088] Construct the joint loss function as shown in the following equation: calibrate the weight matrix.

[0089] ;

[0090] in, This represents the task loss function. Denotes the joint loss function. These represent the weight hyperparameters corresponding to the conflict suppression regularization term, the critical position constraint regularization term, and the sparse merging evaluation regularization term, respectively.

[0091] Secondly, the present invention provides a fine-tuning device for a large-scale intelligent health model based on gradient iterative pruning, comprising:

[0092] The pruning module is configured to perform singular value decomposition on the updated weight matrix of the smart health model to be fine-tuned, and construct a first low-rank approximation matrix and a second low-rank approximation matrix. Based on the task loss function of the current round of the smart health model during fine-tuning, it calculates the importance of each element in the first and second low-rank approximation matrices for the current round, and iteratively updates the cumulative importance of each element in the first and second low-rank approximation matrices for the current round. After multiple iterations, it constructs a first mask matrix and a second mask matrix. Based on the first and second mask matrices, it then performs singular value decomposition on the first and second low-rank approximation matrices respectively. The rank approximation matrix is ​​iteratively pruned to obtain the first pruning weight matrix and the second pruning weight matrix. Based on the first pruning weight matrix and the second pruning weight matrix, the first weight ratio coefficient and the second weight ratio coefficient are calculated using an adaptive weight allocation algorithm with contribution calibration. The calibration weight matrix is ​​calculated based on the first weight ratio coefficient, the second weight ratio coefficient, the first pruning weight matrix and the second pruning weight matrix. The conflict suppression regularization term, the key position constraint regularization term and the sparse merging evaluation regularization term are introduced based on the calibration weight matrix. A joint loss function is constructed by combining the task loss function. The first low-rank approximation matrix and the second low-rank approximation matrix are fine-tuned based on the joint loss function to obtain the fine-tuned smart health big model.

[0093] The generation module is configured to acquire input text or speech and input it into the fine-tuned smart health big model. It calculates the calibration weight matrix using the first low-rank approximation matrix and the second low-rank approximation matrix in the fine-tuned smart health big model. It then calculates the fusion weight matrix based on the calibration weight matrix and performs forward propagation calculation using the fusion weight matrix to obtain the response statement.

[0094] Thirdly, the present invention provides an electronic device including one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any implementation of the first aspect.

[0095] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any of the implementations of the first aspect.

[0096] Fifthly, the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the method as described in any of the implementations in the first aspect.

[0097] Compared with the prior art, the present invention has the following beneficial effects:

[0098] (1) The smart health large model fine-tuning method based on gradient iterative pruning mentioned in this invention performs low-rank approximate decomposition on the updated weight matrix and combines gradient information to perform iterative pruning on the first low-rank approximate matrix and the second low-rank approximate matrix, so as to reduce the redundancy of elements in the matrix while maintaining the low-rank recombinable update structure.

[0099] (2) The smart health large model fine-tuning method based on gradient iterative pruning mentioned in this invention extracts and calibrates key contributions to the first pruning weight matrix and the second pruning weight matrix, and calculates the first weight ratio coefficient and the second weight ratio coefficient by using the adaptive weight allocation algorithm of contribution calibration, thereby realizing cross-matrix scale calibration and ratio allocation, thereby improving the stability of fusion inference in multi-task calling scenarios.

[0100] (3) The gradient-based iterative pruning method for fine-tuning the smart health model mentioned in this invention introduces conflict suppression regularization, key position constraint regularization and sparse merging evaluation regularization, and constructs a joint loss function in combination with the task loss function to suppress update conflicts at key positions and enhance the robustness of end-side deployment, so as to improve fusion stability and maintain model generalization performance, thereby meeting the lightweight, high reliability and scalable deployment requirements of family health scenarios. Attached Figure Description

[0101] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0102] Figure 1 This is a flowchart illustrating the method for fine-tuning a large-scale smart health model based on gradient iterative pruning, as an embodiment of this application.

[0103] Figure 2 This is a schematic diagram of the iterative pruning process in the gradient-based iterative pruning method for fine-tuning a large smart health model, as described in an embodiment of this application.

[0104] Figure 3 This is a schematic diagram of the process of calculating the calibration weight matrix through an adaptive weight allocation process in the gradient iterative pruning-based smart health large model fine-tuning method of this application.

[0105] Figure 4 This is a schematic diagram of a gradient-based iterative pruning-based smart health model fine-tuning device according to an embodiment of this application.

[0106] Figure 5 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0107] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0108] Figure 1 This application illustrates an embodiment of a method for fine-tuning a large-scale smart health model based on gradient iterative pruning, comprising the following steps:

[0109] S1. Obtain the updated weight matrix of the smart health model to be fine-tuned, perform singular value decomposition, and construct a first low-rank approximation matrix and a second low-rank approximation matrix. Based on the task loss function of the current round of the smart health model during fine-tuning, calculate the importance of each element in the first and second low-rank approximation matrices for the current round, and iteratively update the cumulative importance of each element in the first and second low-rank approximation matrices for the current round. After multiple iterations, construct a first mask matrix and a second mask matrix. Based on the first and second mask matrices, perform singular value decomposition on the first and second low-rank approximation matrices respectively. The matrix is ​​iteratively pruned to obtain a first pruning weight matrix and a second pruning weight matrix. Based on the first and second pruning weight matrices, an adaptive weight allocation algorithm with contribution calibration is used to calculate the first and second weight ratio coefficients. According to the first and second weight ratio coefficients, the first and second pruning weight matrices, a calibration weight matrix is ​​calculated. Based on the calibration weight matrix, conflict suppression regularization, key position constraint regularization, and sparse merging evaluation regularization are introduced, and a joint loss function is constructed in conjunction with the task loss function. Based on the joint loss function, the first and second low-rank approximation matrices are fine-tuned to obtain the fine-tuned smart health big model.

[0110] Specifically, the smart health big model in the embodiments of this application uses a pre-trained large language model as the core semantic reasoning and text generation entity, which can be a Transformer-based sequence modeling architecture. For text-based tasks, the input of the smart health big model is a text sequence or an encoded instruction sequence, and the output is the corresponding text result. For speech recognition tasks, the input end can first convert the speech signal into an acoustic feature representation, and further map it into a sequence representation that the smart health big model can process, and then the smart health big model generates the corresponding text result. Thus, the model architecture applicable to this fine-tuning method can be summarized as a unified sequence processing architecture based on Transformer and oriented towards multi-task understanding and generation.

[0111] In a smart home scenario, user voice data, text data, and multi-turn health interaction data are collected to construct a training set. Each sample is accompanied by a corresponding transcription label, emotion label, intent label, or response label. In one example, AISHELL-1 can be used as the instruction fine-tuning dataset, and Qwen2.5-3B-Instruct can be selected as the base model to provide a model foundation for subsequent low-rank approximation, gradient iterative pruning, and contribution calibration. Finally, a large-scale smart health model with speech recognition, emotion understanding, health inquiry, and companion dialogue generation capabilities is obtained through fine-tuning.

[0112] In a specific embodiment, the updated weight matrix of the smart health model to be fine-tuned is obtained, singular value decomposition is performed, and a first low-rank approximation matrix and a second low-rank approximation matrix are constructed, specifically including:

[0113] The updated weight matrix is ​​subjected to singular value decomposition to obtain the left singular vector matrix, the singular value diagonal matrix, and the right singular vector matrix, as shown in the following equation:

[0114] ;

[0115] in, This indicates updating the weight matrix. Describes a left singular vector matrix. Represents a singular value diagonal matrix. This represents the first to the rth singular values. Represents a diagonal matrix. Describes a right singular vector matrix. Represents the transpose of a matrix;

[0116] By coupling the singular value diagonal matrix with the left and right singular vector matrices respectively in square root form, we obtain the first low-rank approximation matrix and the second low-rank approximation matrix, as shown in the following equation:

[0117] ;

[0118] ;

[0119] in, Denotes the first low-rank approximation matrix. This represents the second low-rank approximation matrix.

[0120] For details, please refer to Figure 2 In the embodiments of this application, the weight matrix of the smart health big data model to be fine-tuned includes the original weight matrix. and update the weight matrix Original weight matrix Freeze the data and update the weight matrix. The pruning process involves first updating the weight matrix of the smart health model to be fine-tuned. Performing SVD decomposition yields a left singular vector matrix, a singular value diagonal matrix, and a right singular vector matrix. Then, the singular value diagonal matrix is ​​coupled to the left and right singular vector matrices in square root form to obtain the first low-rank approximation matrix. Second low-rank approximation matrix Therefore, the first low-rank approximation matrix Second low-rank approximation matrix They can share the same singular value diagonal matrix and satisfy a reconfigurable low-rank structure: .

[0121] In a specific embodiment, the importance of each element in the first and second low-rank approximation matrices in the current round is calculated based on the gradient of the first and second low-rank approximation matrices according to the task loss function of the smart health big model during fine-tuning. The cumulative importance of each element in the first and second low-rank approximation matrices in the current round is then iteratively updated. After multiple iterations, a first mask matrix and a second mask matrix are constructed. Based on the first and second mask matrices, iterative pruning is performed on the first and second low-rank approximation matrices to obtain a first pruning weight matrix and a second pruning weight matrix, specifically including:

[0122] Let the task loss function of the aforementioned smart health model during the fine-tuning process be... ;

[0123] The importance of each element in the first and second low-rank approximation matrices for the current round is obtained by calculating the first Taylor expansion of the task loss function with respect to the first or second low-rank approximation matrix. The calculation process is as follows:

[0124] ;

[0125] ;

[0126] in, Describes the first low-rank approximation matrix. OK The importance of column elements In the second low-rank approximation matrix, the first... OK The importance of column elements Describes the first low-rank approximation matrix The OK Column elements, Describes the second low-rank approximation matrix The OK Column elements, Indicates the first The task loss function of the round of iterations approximates the first low-rank matrix. The OK The partial derivatives of the elements of the column, Indicates the first The task loss function for each round approximation matrix is ​​applied to the second low-rank matrix. The OK Partial derivatives of the elements of the column;

[0127] The cumulative importance of each element in the first and second low-rank approximation matrices of the previous round is updated by using the importance of each element in the first and second low-rank approximation matrices of the current round, as shown in the following formula:

[0128] ;

[0129] ;

[0130] in, Indicates the first The first low-rank approximation matrix of each round The Middle OK The cumulative importance of the elements in a column. Indicates the first The first low-rank approximation matrix of each round The Middle OK The cumulative importance of the elements in a column. Indicates the first The second low-rank approximation matrix for each round The Middle OK The cumulative importance of the elements in a column. Indicates the first The second low-rank approximation matrix for each round The Middle OK The cumulative importance of the elements in the column;

[0131] go through After the nth iteration, the nth and nth results are obtained respectively. The first low-rank approximation matrix of each round The Middle OK Cumulative importance of column elements and the The second low-rank approximation matrix for each round The Middle OK Cumulative importance of column elements ;

[0132] Set pruning threshold ,in accordance with and Construct the first mask matrix Second mask matrix The calculation process is as follows:

[0133] ;

[0134] ;

[0135] in, Represents the first mask matrix The Middle OK Column elements, Represents the second mask matrix The Middle OK Column elements;

[0136] Using the first mask matrix For the first low-rank approximation matrix Pruning is performed to obtain the first pruning weight matrix; a second mask matrix is ​​then used. For the second low-rank approximation matrix Pruning is performed to obtain the second pruning weight matrix, as shown in the following formula:

[0137] ;

[0138] ;

[0139] in, This represents the Hadamard product. This represents the first pruning weight matrix. This represents the second pruning weight matrix.

[0140] Specifically, on this low-rank structure, respectively... and Evaluation of the importance of execution weights. Let the task loss function used by the smart health model during fine-tuning training be... The embodiments of this application do not modify the loss function; it remains a conventional loss function for large language models, such as the cross-entropy loss function. In the current round, training data is input into the intelligent health large model to obtain the output text sequence, and the task loss function for the current round is calculated. and When a zero-perturbation is applied to any element in the algorithm, the change in loss caused by the perturbation can be expressed by the task loss function of the current round with respect to that element. and The first-order Taylor expansion of the elements in the formula is approximated, and therefore the task loss function of the current round can be obtained with respect to this formula. and The first-order Taylor expansion of the elements in the equation is respectively related to the... and The first low-rank approximation matrix for the current round is obtained by calculating the elements in the matrix. With the second low-rank approximation matrix The importance of each element Loss function right The Middle OK The partial derivatives of column elements represent gradient information. Element-level importance metrics are constructed using the gradient information available during training on this low-rank structure.

[0141] Then, a gradient-guided iterative pruning strategy is used to prune the branches. and The importance of each element is evaluated round by round, and the cumulative importance of each round is maintained. In the first During each iteration, respectively... and The cumulative importance of each element in the set is updated, that is, the importance of the element in the set is updated at the 1st rank. The first low-rank approximation matrix of each round With the second low-rank approximation matrix Based on the cumulative importance of each element, the first element is added... The first low-rank approximation matrix of each round With the second low-rank approximation matrix The importance of each element can be determined to obtain the first... The first low-rank approximation matrix of each round With the second low-rank approximation matrix The cumulative importance of each element in the equation. Represents the total number of iterations, when At that time, the first The first low-rank approximation matrix of each round With the second low-rank approximation matrix The cumulative importance of each element is initialized and can be set to 0.

[0142] After After the first iteration, set the pruning threshold. According to Article The first low-rank approximation matrix of each round With the second low-rank approximation matrix The first mask matrix is ​​constructed iteratively by calculating the cumulative importance of each element. Second mask matrix For the first low-rank approximation matrix With the second low-rank approximation matrix The Middle OK The elements of the column are iteratively used to generate the first and second mask matrices. During the iteration, only the cumulative importance is updated and not immediately set to zero. After the iteration, based on the... The first low-rank approximation matrix of each round With the second low-rank approximation matrix The cumulative importance of each element in the matrix is ​​used to generate a first mask matrix and a second mask matrix, and pruning is performed to obtain the first pruning weight matrix. Second pruning weight matrix The fine-tuned smart health model includes the original weight matrix and the first pruned weight matrix. Second pruning weight matrix By using the first and second mask matrices to perform layer-by-layer element filtering on the first and second low-rank approximation matrices respectively, the approximation can be achieved. and Iterative pruning is employed. While maintaining low-rank representation and update recombinability, it reduces element redundancy within the matrix and improves the fine-tuning stability of the smart health model.

[0143] In a specific embodiment, the first weight ratio coefficient and the second weight ratio coefficient are calculated based on the first pruning weight matrix and the second pruning weight matrix using an adaptive weight allocation algorithm with contribution calibration, specifically including:

[0144] Taking the absolute values ​​of the first pruning weight matrix and the second pruning weight matrix respectively, we obtain the first magnitude matrix. Second magnitude matrix ;

[0145] The first magnitude matrix was selected multiple times using a non-replacement method. Second magnitude matrix The indices corresponding to the maximum values ​​of the elements in the set constitute the first index set. Second index set As shown in the following formula:

[0146] ;

[0147] ;

[0148] ;

[0149] ;

[0150] in, Indicates the first The first magnitude matrix is ​​selected next. The first index set is obtained by taking the index corresponding to the maximum value of the elements in the set. Indicates the first The second magnitude matrix is ​​selected next. The second set of indices is obtained by taking the index corresponding to the maximum value of the elements in the set. Indicates the first The first magnitude matrix is ​​selected next. The index corresponding to the maximum value of the elements in the array. Indicates the first The second magnitude matrix is ​​selected next. The index corresponding to the maximum value of the elements in the array. , This indicates the total number of selections. This indicates the index corresponding to the maximum value;

[0151] Contribution aggregation is performed on the first index set and the second index set respectively to obtain the first contribution metric and the second contribution metric, as shown in the following formula:

[0152] ;

[0153] ;

[0154] in, and These represent the first contribution metric and the second contribution metric, respectively.

[0155] The balance factor is calculated based on the first pruning weight matrix and the second pruning weight matrix, as shown in the following formula:

[0156] ;

[0157] in, This represents the summation of all elements of a matrix. As the first stable term, Indicates the balance factor;

[0158] The first and second contribution measures are calibrated using a balance factor to obtain the first and second calibrated contribution measures, as shown in the following formula:

[0159] ;

[0160] ;

[0161] in, and These represent the first calibration contribution metric and the second calibration contribution metric, respectively.

[0162] The first weighting coefficient and the second weighting coefficient are calculated based on the first calibration contribution metric and the second calibration contribution metric, as shown in the following formula:

[0163] ;

[0164] ;

[0165] in, and These represent the first weighting coefficient and the second weighting coefficient, respectively. , It is the second stable term.

[0166] For details, please refer to Figure 3 The fine-tuned smart health big data model in the embodiments of this application first adjusts the first pruning weight matrix. Second pruning weight matrix Taking the absolute values ​​of each, we obtain the first magnitude matrix and the second magnitude matrix, defined as follows: Select the largest value from the first and second magnitude matrices. We obtain a key subset by identifying key elements. In other words, we select the elements from the first and second magnitude matrices in descending order without replacement. Given the indices of the elements, we obtain the first index set. Second index set In the first magnitude matrix Second magnitude matrix In and the first index set Second index set The corresponding elements are used as the key subset. First magnitude matrix. Second magnitude matrix This is used to retain significant contributions and suppress redundancy. Next, contribution aggregation is performed on the key subset to obtain the first contribution metric. Second contribution measure .

[0167] Subsequently, a balance factor was introduced. Used for the first pruning weight matrix Second pruning weight matrix The cross-matrix contribution magnitude is uniformly calibrated, and the balance factor is used for this. This is used to suppress fusion imbalance caused by the difference in magnitude between the two sides. This is achieved through the introduction of a balancing factor. Measurement of first contribution Second contribution measure Perform calibration and calculate the first calibration contribution metric. Second calibration contribution metric , and A symmetrical calibration is formed.

[0168] Finally, based on the first calibration contribution metric Second calibration contribution metric Construct the first weighting ratio Second weighting coefficient , This is used to implement weight allocation for cross-matrix updates. Acting on respectively Then, the calibration weight matrix can be generated. The global scaling factor It is used to compensate for the amplitude decay caused by the weight product and to adaptively adjust the update step size.

[0169] In a specific embodiment, a conflict suppression regularization term, a key location constraint regularization term, and a sparse merging evaluation regularization term are introduced based on the calibration weight matrix, and a joint loss function is constructed by combining it with the task loss function, specifically including:

[0170] For calibration weight matrix Set the preset sparsity Only the first one is retained. Important elements generate sparse mask matrix As shown in the following formula:

[0171] ;

[0172] in, This indicates that the calibration weight matrix is ​​adjusted according to a preset sparsity rate. Operators for generating sparse mask matrices,

[0173] The sparse fusion matrix is ​​calculated based on the sparse mask matrix and the calibration weight matrix. As shown in the following formula:

[0174] ;

[0175] The sparsity ratio of the matrix is ​​calculated based on the sparse fusion matrix, as shown in the following formula:

[0176] ;

[0177] in, Represents the sparsity of a matrix. These represent the total number of rows and columns of the sparse fusion matrix, respectively. express Pseudonorm;

[0178] The importance score for each element is calculated based on the calibration weight matrix, as shown in the following formula:

[0179] ;

[0180] in, Represents the first element in the calibration weight matrix. OK The importance score of the elements in the column. Represents the first element in the calibration weight matrix. OK Column elements, Represents the L2 norm. Represents a constant;

[0181] The retention mask matrix is ​​calculated based on the importance score, as shown in the following formula:

[0182] ;

[0183] in, Indicates hyperparameters, Represents the retention mask matrix The Middle OK Column elements;

[0184] The conflict suppression regularization term is constructed based on the retention mask matrix and the sparse fusion matrix, as shown in the following equation:

[0185] ;

[0186] in, This indicates a conflict suppression regularization term. Describing the Frobenius norm, Represents the Hadamard product;

[0187] The key position constraint regularization term is constructed based on the sparse mask matrix and the calibration weight matrix, as shown in the following equation:

[0188] ;

[0189] in, express Norm, Indicates the key position constraint regularization term;

[0190] The sparse merging evaluation regularization term is constructed based on the matrix sparsity rate and the preset sparsity rate, as shown in the following formula:

[0191] ;

[0192] in, This represents the regularization term for sparse merging evaluation;

[0193] The joint loss function is constructed as shown in the following equation:

[0194] ;

[0195] in, This represents the task loss function. Denotes the joint loss function. These represent the weight hyperparameters corresponding to the conflict suppression regularization term, the critical position constraint regularization term, and the sparse merging evaluation regularization term, respectively.

[0196] Specifically, regarding the calibration weight matrix Set the preset sparsity Only the first one is retained. Important elements generate sparse mask matrix And calculate the sparse fusion matrix. Simultaneously calculate the matrix sparsity. The sparsity of this matrix The closer to 1, the sparser the data. Sparse mask matrix. Used to constrain sparse merging structures.

[0197] Next, the calibration weight matrix is... Construct element-level importance score Based on this, a retention mask matrix is ​​constructed. This is used to represent the criticality of elements in the calibration weight matrix, when the retention mask matrix... The larger the element in the calibration weight matrix, the more important the corresponding element is in the pre-trained large language model. This plays a role in strengthening the protection or suppression of key position updates, thereby reducing the destruction of general knowledge.

[0198] Next, design a conflict mitigation regularizer. This approach aims to suppress and preserve sparse updates that conflict at important positions. By minimizing this conflict suppression regularization term, a freezing tendency is applied to key parameters carrying general knowledge, maximizing the generalization ability of large language models and preventing catastrophic forgetting.

[0199] Furthermore, two types of sparse correlation constraints are introduced: key location constraint regularization terms. This is used to suppress redundant updates at non-critical locations, and to implement sparse merging evaluation regularization. Used to constrain the actual matrix sparsity. This ensures that the sparse merging result meets the target sparsity level.

[0200] Finally, the task loss function is combined with the three regularization terms mentioned above to construct the joint loss function. In the joint loss function The first and second low-rank approximation matrices are updated by backpropagation to achieve synergistic optimization of sparse merging constraints, key position constraints and conflict suppression, thereby improving the fusion stability of the smart health big model under lightweight deployment conditions and maintaining its generalization performance.

[0201] S2: Obtain the input text or speech and input it into the fine-tuned smart health big model. Calculate the calibration weight matrix using the first low-rank approximation matrix and the second low-rank approximation matrix in the fine-tuned smart health big model. Calculate the fusion weight matrix based on the calibration weight matrix. Perform forward propagation calculation using the fusion weight matrix to obtain the response statement.

[0202] In a specific embodiment, the formula for calculating the calibration weight matrix is ​​as follows:

[0203] ;

[0204] in, Represents the calibration weight matrix. Indicates the global scaling factor;

[0205] The formula for calculating the fusion weight matrix is ​​as follows:

[0206] ;

[0207] in, Represents the fusion weight matrix. This represents the original weight matrix that is frozen in the smart health model to be fine-tuned.

[0208] Specifically, the fine-tuned smart health model is deployed, and the calibration weight matrix is ​​obtained from the first and second low-rank approximation matrices after the fine-tuning through the above calculations. The original weight matrix is ​​then injected to obtain the final fused weight matrix. In the forward propagation process of the inference phase of the smart health big data model, the linear mapping, attention projection, or feedforward network transformation of the corresponding layer are all based on the fusion weight matrix. It performs calculations, thus directly participating in the reasoning process for tasks such as decision-making. Therefore, stable contribution calibration and adaptive weight reallocation for edge deployment of large-scale smart health models can be achieved without introducing additional training parameters.

[0209] Further reference Figure 4 As an implementation of the methods shown in the above figures, this application provides an embodiment of a smart health large-scale model fine-tuning device based on gradient iterative pruning. This device embodiment is similar to... Figure 1 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0210] This application provides a device for fine-tuning a large-scale smart health model based on gradient iterative pruning, including:

[0211] Pruning module 1 is configured to perform singular value decomposition on the updated weight matrix of the smart health model to be fine-tuned, and construct a first low-rank approximation matrix and a second low-rank approximation matrix. Based on the task loss function of the current round of the smart health model during fine-tuning, it calculates the importance of each element in the first and second low-rank approximation matrices for the current round, and iteratively updates the cumulative importance of each element in the first and second low-rank approximation matrices for the current round. After multiple iterations, it constructs a first mask matrix and a second mask matrix. Based on the first and second mask matrices, it then performs singular value decomposition on the first and second low-rank approximation matrices for the current round. The low-rank approximation matrix is ​​iteratively pruned to obtain the first pruning weight matrix and the second pruning weight matrix. Based on the first pruning weight matrix and the second pruning weight matrix, the first weight ratio coefficient and the second weight ratio coefficient are calculated using an adaptive weight allocation algorithm with contribution calibration. The calibration weight matrix is ​​calculated based on the first weight ratio coefficient, the second weight ratio coefficient, the first pruning weight matrix and the second pruning weight matrix. The conflict suppression regularization term, the key position constraint regularization term and the sparse merging evaluation regularization term are introduced based on the calibration weight matrix. A joint loss function is constructed by combining the task loss function. The first low-rank approximation matrix and the second low-rank approximation matrix are fine-tuned based on the joint loss function to obtain the fine-tuned smart health big model.

[0212] The generation module 2 is configured to acquire the input text or voice and input it into the fine-tuned smart health big model. The calibration weight matrix is ​​calculated using the first low-rank approximation matrix and the second low-rank approximation matrix in the fine-tuned smart health big model. The fusion weight matrix is ​​calculated based on the calibration weight matrix. The forward propagation calculation is performed using the fusion weight matrix to obtain the response statement.

[0213] Figure 5 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. For example... Figure 5 As shown, the electronic device of this embodiment includes a processor 501 and a memory 502; wherein the memory 502 is used to store computer execution instructions; and the processor 501 is used to execute the computer execution instructions stored in the memory to implement the various steps performed by the electronic device in the above embodiment. For details, please refer to the relevant descriptions in the foregoing method embodiments.

[0214] Alternatively, the memory 502 can be either standalone or integrated with the processor 501.

[0215] When the memory 502 is set up independently, the electronic device also includes a bus 503 for connecting the memory 502 and the processor 501.

[0216] This invention also provides a computer storage medium storing computer execution instructions, which, when executed by processor 501, implement the above method.

[0217] This invention also provides a computer program product, including a computer program that, when executed by a processor 501, implements the above-described method.

[0218] In the embodiments provided by this invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0219] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to implement the solution of this embodiment according to actual needs.

[0220] Furthermore, the functional modules in the various embodiments of this invention can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit formed by the above modules can be implemented in hardware or in the form of hardware plus software functional units.

[0221] The integrated modules implemented as software functional modules described above can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor 501 to execute some steps of the methods of the various embodiments of this application.

[0222] It should be understood that the processor 501 described above can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor, or the processor 501 can be any conventional processor 501. The steps of the method disclosed in this invention can be directly manifested as the hardware processor 501 executing the steps, or as a combination of hardware and software modules within the processor 501 executing the steps.

[0223] The memory 502 may include high-speed RAM memory, and may also include non-volatile memory NVM, such as at least one disk storage device, and may also be a USB flash drive, portable hard drive, read-only memory, disk or optical disc, etc.

[0224] Bus 503 can be an Industry Standard Architecture (ISA), a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Bus 503 can be divided into address bus, data bus, control bus, etc. For ease of illustration, the bus 503 in the accompanying drawings of this application is not limited to only one bus 503 or one type of bus 503.

[0225] The aforementioned storage medium can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium accessible to general-purpose or special-purpose computers.

[0226] An exemplary storage medium is coupled to processor 501, enabling processor 501 to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of processor 501. Processor 501 and storage medium can reside in application-specific integrated circuits (ASICs). Alternatively, processor 501 and storage medium can exist as discrete components in an electronic device or host device.

[0227] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0228] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for fine-tuning a large-scale smart health model based on gradient iterative pruning, characterized in that, Includes the following steps: The updated weight matrix of the smart health model to be fine-tuned is obtained and subjected to singular value decomposition to construct a first low-rank approximation matrix and a second low-rank approximation matrix. Based on the task loss function of the current round of the smart health model during fine-tuning, the importance of each element in the first and second low-rank approximation matrices for the current round is calculated using the gradient of the first and second low-rank approximation matrices. The cumulative importance of each element in the first and second low-rank approximation matrices for the current round is iteratively updated. After multiple iterations, a first mask matrix and a second mask matrix are constructed. The first and second low-rank approximation matrices are then processed based on the first and second mask matrices. Iterative pruning is performed to obtain a first pruning weight matrix and a second pruning weight matrix. Based on the first pruning weight matrix and the second pruning weight matrix, an adaptive weight allocation algorithm with contribution calibration is used to calculate a first weight ratio coefficient and a second weight ratio coefficient. According to the first weight ratio coefficient, the second weight ratio coefficient, the first pruning weight matrix and the second pruning weight matrix, a calibration weight matrix is ​​calculated. According to the calibration weight matrix, a conflict suppression regularization term, a key position constraint regularization term and a sparse merging evaluation regularization term are introduced, and a joint loss function is constructed in combination with the task loss function. Based on the joint loss function, the first low-rank approximation matrix and the second low-rank approximation matrix are fine-tuned to obtain the fine-tuned smart health big model. The input text or voice is obtained and input into the fine-tuned smart health big model. The calibration weight matrix is ​​calculated using the first low-rank approximation matrix and the second low-rank approximation matrix in the fine-tuned smart health big model. The fusion weight matrix is ​​calculated based on the calibration weight matrix. The forward propagation calculation is performed using the fusion weight matrix to obtain the response statement.

2. The method for fine-tuning a large-scale smart health model based on gradient iterative pruning according to claim 1, characterized in that, The updated weight matrix of the smart health model to be fine-tuned is obtained, and singular value decomposition is performed to construct the first low-rank approximation matrix and the second low-rank approximation matrix, specifically including: The updated weight matrix is ​​subjected to singular value decomposition to obtain a left singular vector matrix, a singular value diagonal matrix, and a right singular vector matrix, as shown in the following equation: ； in, This indicates updating the weight matrix. Describes a left singular vector matrix. Represents a singular value diagonal matrix. This represents the first to the rth singular values. Represents a diagonal matrix. Describes a right singular vector matrix. Represents the transpose of a matrix; The singular value diagonal matrix is ​​coupled to the left singular vector matrix and the right singular vector matrix respectively in square root form to obtain the first low-rank approximation matrix and the second low-rank approximation matrix, as shown in the following equation: ； ； in, Denotes the first low-rank approximation matrix. This represents the second low-rank approximation matrix.

3. The method for fine-tuning a large-scale smart health model based on gradient iterative pruning according to claim 1, characterized in that, Based on the task loss function of the smart health model during fine-tuning, the importance of each element in the first and second low-rank approximation matrices in the current round is calculated using the gradients of the first and second low-rank approximation matrices. The cumulative importance of each element in the first and second low-rank approximation matrices in the current round is then iteratively updated. After multiple iterations, a first mask matrix and a second mask matrix are constructed. Based on the first and second mask matrices, iterative pruning is performed on the first and second low-rank approximation matrices to obtain a first pruning weight matrix and a second pruning weight matrix, specifically including: Let the task loss function of the aforementioned smart health model during the fine-tuning process be... ; The importance of each element in the first and second low-rank approximation matrices for the current round is obtained by calculating the first Taylor expansion of the task loss function with respect to the first or second low-rank approximation matrix. The calculation process is as follows: ； ； in, Represents the first low-rank approximation matrix. OK The importance of column elements Describes the second low-rank approximation matrix as the first... OK The importance of column elements Describes the first low-rank approximation matrix The OK Column elements, Describes the second low-rank approximation matrix The OK Column elements, Indicates the first The task loss function of the round of iterations approximates the first low-rank matrix. The OK The partial derivatives of the elements of the column, Indicates the first The task loss function for each round approximation matrix is ​​applied to the second low-rank matrix. The OK Partial derivatives of the elements of the column; The cumulative importance of each element in the first and second low-rank approximation matrices of the previous round is updated by using the importance of each element in the first and second low-rank approximation matrices of the current round, as shown in the following formula: ； ； in, Indicates the first The first low-rank approximation matrix of each round The Middle OK The cumulative importance of the elements in a column. Indicates the first The first low-rank approximation matrix of each round The Middle OK The cumulative importance of the elements in a column. Indicates the first The second low-rank approximation matrix for each round The Middle OK The cumulative importance of the elements in a column. Indicates the first The second low-rank approximation matrix for each round The Middle OK The cumulative importance of the elements in the column; go through After the nth iteration, the nth and nth results are obtained respectively. The first low-rank approximation matrix of each round The Middle OK Cumulative importance of column elements and the The second low-rank approximation matrix for each round The Middle OK Cumulative importance of column elements ; Set pruning threshold ,in accordance with and Construct the first mask matrix Second mask matrix The calculation process is as follows: ； ； in, Represents the first mask matrix The Middle OK Column elements, Represents the second mask matrix The Middle OK Column elements; Using the first mask matrix For the first low-rank approximation matrix Pruning is performed to obtain the first pruning weight matrix; the second mask matrix is ​​then used. For the second low-rank approximation matrix Pruning is performed to obtain the second pruning weight matrix, as shown in the following formula: ； ； in, This represents the Hadamard product. This represents the first pruning weight matrix. This represents the second pruning weight matrix.

4. The method for fine-tuning a large-scale smart health model based on gradient iterative pruning according to claim 1, characterized in that, Based on the first pruning weight matrix and the second pruning weight matrix, an adaptive weight allocation algorithm with contribution calibration is used to calculate the first weight ratio coefficient and the second weight ratio coefficient, specifically including: Taking the absolute values ​​of the first pruning weight matrix and the second pruning weight matrix respectively, we obtain the first magnitude matrix. Second magnitude matrix ; The first magnitude matrix is ​​selected multiple times using a non-replacement method. Second magnitude matrix The indices corresponding to the maximum values ​​of the elements in the set constitute the first index set. Second index set As shown in the following formula: ； ； ； ； in, Indicates the first The first magnitude matrix is ​​selected next. The first index set is obtained by taking the index corresponding to the maximum value of the elements in the set. Indicates the first The second magnitude matrix is ​​selected next. The second set of indices is obtained by taking the index corresponding to the maximum value of the elements in the set. Indicates the first The first magnitude matrix is ​​selected next. The index corresponding to the maximum value of the elements in the array. Indicates the first The second magnitude matrix is ​​selected next. The index corresponding to the maximum value of the elements in the array. , This indicates the total number of selections. This indicates the index corresponding to the maximum value; Contribution aggregation is performed on the first index set and the second index set respectively to obtain the first contribution metric and the second contribution metric, as shown in the following formula: ； ； in, and These represent the first contribution metric and the second contribution metric, respectively. The balance factor is calculated based on the first pruning weight matrix and the second pruning weight matrix, as shown in the following formula: ； in, This represents the summation of all elements of a matrix. As the first stable term, Indicates the balance factor; The first contribution metric and the second contribution metric are calibrated using the balance factor to obtain the first calibrated contribution metric and the second calibrated contribution metric, as shown in the following formula: ； ； in, and These represent the first calibration contribution metric and the second calibration contribution metric, respectively. The first weighting coefficient and the second weighting coefficient are calculated based on the first calibration contribution metric and the second calibration contribution metric, as shown in the following formula: ； ； in, and These represent the first weighting coefficient and the second weighting coefficient, respectively. , It is the second stable term.

5. The method for fine-tuning a large-scale smart health model based on gradient iterative pruning according to claim 4, characterized in that, The formula for calculating the calibration weight matrix is ​​as follows: ； in, Represents the calibration weight matrix. Indicates the global scaling factor; The formula for calculating the fusion weight matrix is ​​as follows: ； in, Represents the fusion weight matrix. This represents the original weight matrix that is frozen in the smart health model to be fine-tuned.

6. The method for fine-tuning a large-scale smart health model based on gradient iterative pruning according to claim 1, characterized in that, Based on the calibration weight matrix, conflict suppression regularization, key location constraint regularization, and sparse merging evaluation regularization are introduced, and a joint loss function is constructed by combining it with the task loss function, specifically including: For the calibration weight matrix Set the preset sparsity Only the first one is retained. Important elements generate sparse mask matrix As shown in the following formula: ； in, This indicates that the calibration weight matrix is ​​adjusted according to a preset sparsity rate. Operators for generating sparse mask matrices, The sparse fusion matrix is ​​calculated based on the sparse mask matrix and the calibration weight matrix. As shown in the following formula: ； The sparsity ratio of the matrix is ​​calculated based on the sparse fusion matrix, as shown in the following formula: ； in, Represents the sparsity of a matrix. These represent the total number of rows and the total number of columns of the sparse fusion matrix, respectively. express Pseudonorm; The importance score for each element is calculated based on the calibration weight matrix, as shown in the following formula: ； in, In the calibration weight matrix, the first... OK The importance score of the elements in the column. In the calibration weight matrix, the first... OK Column elements, Describing the L2 norm, Represents a constant; The retention mask matrix is ​​calculated based on the importance score, as shown in the following formula: ； in, Indicates hyperparameters, Represents the retention mask matrix The Middle OK Column elements; Based on the retention mask matrix and the sparse fusion matrix, a conflict suppression regularization term is constructed as shown in the following equation: ； in, This indicates a conflict suppression regularization term. Denotes the Frobenius norm. Represents the Hadamard product; The key position constraint regularization term is constructed based on the sparse mask matrix and the calibration weight matrix, as shown in the following equation: ； in, express Norm, Indicates the key position constraint regularization term; Based on the matrix sparsity rate and the preset sparsity rate, a sparse merging evaluation regularization term is constructed, as shown in the following formula: ； in, This indicates a regularization term for sparse merging evaluation; The joint loss function is constructed as shown in the following equation: ； in, This represents the task loss function. Denotes the joint loss function. These represent the weight hyperparameters corresponding to the conflict suppression regularization term, the critical position constraint regularization term, and the sparse merging evaluation regularization term, respectively.

7. A fine-tuning device for a large-scale intelligent health model based on gradient iterative pruning, characterized in that, include: The pruning module is configured to perform singular value decomposition on the updated weight matrix of the smart health model to be fine-tuned, and construct a first low-rank approximation matrix and a second low-rank approximation matrix. Based on the task loss function of the smart health model in the current round of fine-tuning, the module calculates the importance of each element in the first and second low-rank approximation matrices for the current round, and iteratively updates the cumulative importance of each element in the first and second low-rank approximation matrices for the current round. After multiple iterations, a first mask matrix and a second mask matrix are constructed. The module then performs singular value decomposition on the first and second low-rank approximation matrices based on the first and second mask matrices. The approximate matrix is ​​iteratively pruned to obtain a first pruning weight matrix and a second pruning weight matrix. Based on the first pruning weight matrix and the second pruning weight matrix, an adaptive weight allocation algorithm with contribution calibration is used to calculate a first weight ratio coefficient and a second weight ratio coefficient. According to the first weight ratio coefficient, the second weight ratio coefficient, the first pruning weight matrix and the second pruning weight matrix, a calibration weight matrix is ​​calculated. According to the calibration weight matrix, a conflict suppression regularization term, a key position constraint regularization term and a sparse merging evaluation regularization term are introduced, and a joint loss function is constructed in combination with the task loss function. Based on the joint loss function, the first low-rank approximate matrix and the second low-rank approximate matrix are fine-tuned to obtain the fine-tuned smart health big model. The generation module is configured to acquire input text or speech and input it into the fine-tuned smart health big model. It calculates a calibration weight matrix using the first low-rank approximation matrix and the second low-rank approximation matrix in the fine-tuned smart health big model. It calculates a fusion weight matrix based on the calibration weight matrix and performs forward propagation calculation using the fusion weight matrix to obtain the response statement.

8. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-6.

Images