Human resource large model parameter fine tuning method, system and equipment based on low-rank decomposition

Abstract

The invention provides a human resource large model parameter fine tuning method, system and equipment based on low-rank decomposition, and the method comprises the steps: selecting a layer which needs to be subjected to parameter fine tuning according to a parameter structure in a human resource large model, and then carrying out the low-rank decomposition of a parameter matrix of the selected layer; and selecting a plurality of parameters which have the greatest influence on a model output result from the parameter matrix after low-rank decomposition as key parameters, performing learning rate distribution on each key parameter according to the importance degree, and performing optimization adjustment on the model parameters based on the adjusted learning rate. According to the invention, efficient fine tuning of the large model is realized.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning, and specifically relates to a method, system and device for fine-tuning parameters of a large human resources model based on low-rank decomposition. Background Art

[0002] With the rapid development of natural language processing (NLP) and deep learning technologies, methods based on large models (such as GPT and BERT) have made significant progress in text understanding and knowledge integration. Large models, through pre-training, acquire rich language expression and knowledge reasoning capabilities, providing new intelligent solutions for human resource management in various industries. In human resource management, large models, through their powerful semantic understanding and knowledge reasoning capabilities, can help companies achieve intelligent allocation and management, including key aspects such as resume screening, job matching, employee performance evaluation, and training needs analysis. This is especially true in multi-tiered organizations like the power industry, where the diversity of personnel and the complexity of positions place higher demands on the accuracy of human resource allocation.

[0003] With the widespread application of large models, model fine-tuning has become a key approach to adapting to specific application scenarios. Currently, mainstream fine-tuning techniques include full-parameter fine-tuning and frozen-partial parameter fine-tuning. Existing full-parameter fine-tuning methods require powerful hardware and enormous computing resources. Especially for complex tasks like human resource allocation, the required hardware (such as GPUs and TPUs) and computational costs are extremely high. This makes large-scale full-parameter fine-tuning unaffordable for many organizations, limiting the adoption of models in industry applications. Furthermore, full-parameter fine-tuning and frozen-partial parameter fine-tuning methods have limited adaptability to specific industry tasks and are unable to flexibly meet the dynamic human resource allocation needs of multi-level organizations. In organizations with volatile business needs, such as the power industry, the lack of an adaptive and flexible fine-tuning method makes it difficult for models to effectively support frequently changing task scenarios. This lack of adaptability limits the model's performance in dynamic human resource allocation tasks, reducing the practical value of large models in industry applications.

[0004] Therefore, developing an efficient fine-tuning method that can reduce computing resources, meet the needs of model accuracy, and adapt to diverse business scenarios has become the focus of technological development. Summary of the Invention

[0005] The purpose of the present invention is to address the above-mentioned problems existing in the prior art and to provide a method, system and equipment for fine-tuning the parameters of a large human resource model based on low-rank decomposition, which meets the human resource allocation needs of complex multi-level organizations such as the power industry.

[0006] To achieve the above objectives, the technical solutions of the present invention are as follows:

[0007] In a first aspect, the present invention proposes a method for fine-tuning parameters of a large human resource model based on low-rank decomposition, comprising:

[0008] S1. Select the layer that needs parameter fine-tuning based on the parameter structure in the human resources model;

[0009] S2. Perform low-rank decomposition on the parameter matrix of the selected layer;

[0010] S3. Select several parameters that have the greatest impact on the model output results from the parameter matrix after low-rank decomposition as key parameters;

[0011] S4. Assign learning rates to key parameters and optimize and adjust model parameters based on the assigned learning rates.

[0012] The S4 uses the following radial function to distribute the learning rates of key parameters:

[0013] η w =η base exp(γ|IScore w -IScore min | 2 )

[0014] In the above formula, η w IScore is the learning rate assigned to the key parameter w, min IScore is the importance of the key parameters that have the least impact on the model output results. w is the importance of parameter w, γ is an adjustable parameter, η base is the baseline learning rate.

[0015] The S2 includes: for the parameter matrix with dimension d*d Decompose it into a low-rank matrix through low-rank decomposition and low-rank matrix Where k is the dimension of the low-rank matrix, and

[0016] Said S1 comprises:

[0017] The gradient norm of each layer in the human resource model is calculated according to the following formula, and the layer with the largest gradient norm is selected as the layer that needs parameter fine-tuning:

[0018] L2=∑w ij 2

[0019] In the above formula, L2 is the gradient norm of a certain layer, w ij is the parameter in this layer;

[0020] The S3 includes: for the parameter matrix after low-rank decomposition, first calculating the gradient of the loss function for each parameter therein, then sorting the parameters according to the gradient size, and selecting the parameters with the gradient values in the top N as key parameters.

[0021] In the second aspect, the present invention proposes a parameter fine-tuning system for a large human resource model based on low-rank decomposition, comprising a parameter fine-tuning layer selection module, a low-rank decomposition module, a key parameter screening module, and a fine-tuning module;

[0022] The parameter fine-tuning layer selection module is used to select the layer that needs parameter fine-tuning according to the parameter structure in the human resources large model;

[0023] The low-rank decomposition module is used to perform low-rank decomposition on the parameter matrix of the selected layer;

[0024] The key parameter screening module is used to select several parameters that have the greatest impact on the model output results from the parameter matrix after low-rank decomposition as key parameters;

[0025] The fine-tuning module is used to allocate the learning rate of each key parameter and optimize and adjust the model parameters based on the allocated learning rate.

[0026] The fine-tuning module uses the following radial function to distribute the learning rates of key parameters:

[0027] η w =η base exp(γ|IScore w -IScore min | 2 )

[0028] In the above formula, η w IScore is the learning rate assigned to the key parameter w, min IScore is the importance of the key parameters that have the least impact on the model output results. w is the importance of parameter w, γ is an adjustable parameter, η base is the baseline learning rate.

[0029] The low-rank decomposition module adopts the following strategy:

[0030] For a parameter matrix of dimension d*d Decompose it into a low-rank matrix through low-rank decomposition and low-rank matrix Where k is the dimension of the low-rank matrix, and

[0031] The parameter fine-tuning layer selection module is used to calculate the gradient norm of each layer in the human resource model according to the following formula, and select the layer with the largest gradient norm as the layer that needs parameter fine-tuning:

[0032] L2=Σw ij 2

[0033] In the above formula, L2 is the gradient norm of a certain layer, w ij is the parameter in this layer;

[0034] The key parameter screening module is used to calculate the gradient of the loss function for each parameter in the parameter matrix after low-rank decomposition, sort the parameters according to the gradient size, and select the parameters with the top N gradient values as the key parameters that have the greatest impact on the model output results.

[0035] In a third aspect, the present invention proposes a parameter fine-tuning device for a large human resource model based on low-rank decomposition, comprising a processor and a memory;

[0036] The memory is used to store computer program code and transmit the computer program code to the processor;

[0037] The processor is used to execute the aforementioned method for fine-tuning parameters of a large human resources model based on low-rank decomposition according to the instructions in the computer program code.

[0038] In a fourth aspect, the present invention proposes a computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the aforementioned method for fine-tuning parameters of a large human resources model based on low-rank decomposition.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] 1. The present invention provides a method for fine-tuning parameters of a large human resource model based on low-rank decomposition. The method first selects the layer that needs to be fine-tuned according to the parameter structure in the large human resource model, then performs low-rank decomposition on the parameter matrix of the selected layer, and then selects several parameters that have the greatest impact on the model output results from the parameter matrix after low-rank decomposition as key parameters. Finally, a learning rate is allocated to each key parameter according to its importance, and the model parameters are optimized and adjusted based on the allocated learning rate. On the one hand, this method preliminarily selects the layer for parameter fine-tuning based on the parameter structure of the large model, effectively reducing the number of parameters involved in fine-tuning. By performing low-rank decomposition on the parameter matrix of the selected layer, the computational complexity of the model is greatly reduced. On the other hand, by screening key parameters and allocating learning rates, the optimization process is mainly focused on the part that most significantly improves the model performance. Under the premise of ensuring the accuracy of the model, the resource utilization efficiency of the fine-tuning process can be improved, so that the large model can adapt to the complex and changeable business scenarios in multi-level organizations such as the power industry.

[0041] 2. The present invention provides a parameter fine-tuning method for a large human resource model based on low-rank decomposition, which uses a radial function to adjust the learning rate of key parameters. This adjustment strategy enables the learning rate to decrease significantly as the importance of the parameter (i.e., the gradient) decreases, and to increase significantly as the importance of the parameter increases, thereby ensuring that important parameters achieve faster learning efficiency and improving the computational efficiency of the model under resource constraints. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a structural diagram of the system described in Example 2.

[0043] Figure 2 This is a structural diagram of the equipment described in Example 3. DETAILED DESCRIPTION

[0044] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0045] The present invention proposes a parameter fine-tuning method for a large human resource model based on low-rank decomposition. This method uses low-rank decomposition technology to maintain the main capabilities of the model while reducing unnecessary redundant calculations, thereby reducing the computational cost of the model by 40-60%, enabling large-scale model fine-tuning to be achieved even in resource-limited environments. The business adaptability of the model is achieved through an adaptive optimization method, and the key parameters can be automatically selected and the fine-tuning process optimized according to the specific needs of human resource management in the power industry. Compared with traditional full-parameter fine-tuning, it exhibits higher accuracy and flexibility in dynamic tasks, can adapt to different job requirements and configuration scenarios, and greatly improves the efficiency of human resource allocation.

[0046] Example 1:

[0047] This example uses the large human resource model (GPT) of the power industry as the research object and implements the parameter fine-tuning method of the large human resource model based on low-rank decomposition described in the present invention. The specific steps are as follows:

[0048] 1. For the attention layer and embedding layer of the large human resource model, calculate the gradient of each layer through forward propagation and backpropagation, compare the gradient norm of each layer, and select the attention layer and embedding layer with the larger gradient norm (that is, the layer with the most significant impact on the model output) as the layer that needs parameter fine-tuning. The gradient norm of each layer is calculated according to the following formula:

[0049] L2=∑w ij 2

[0050] In the above formula, L2 is the gradient norm of a parameter layer, w ij is the parameter in this parameter layer.

[0051] The selection of parameter layers can also be combined with actual business needs (such as the complexity and adaptability requirements of the job) to ensure that the selected parameter layers can optimize the actual tasks.

[0052] 2. Perform low-rank decomposition on the parameter matrices of the selected attention and embedding layers.

[0053] Low-rank decomposition is a matrix approximation method that can significantly reduce the computational complexity and resource requirements of a model by performing low-rank decomposition on large parameter matrices. This method is particularly useful in resource-constrained environments, such as human resource management scenarios in the power industry. The specific steps are as follows:

[0054] Taking the attention layer in the model as an example, the original parameter matrix W has a dimension of 1024×1024. Through low-rank decomposition, it is decomposed into a low-rank matrix U with a dimension of 1024×256 and a low-rank matrix V with a dimension of 256×1024. The decomposition form is as follows:

[0055] W≈U·V.

[0056] The parameter reduction calculation is:

[0057]

[0058] Through this decomposition, the number of parameters after decomposition = 1024 × 256 + 256 × 1024 = 524,288, achieving 50% parameter compression. Moreover, when the decomposition rank k = 256, the model performance is almost unaffected, thus maintaining the model accuracy.

[0059] The original weight matrix is approximately reconstructed by calculating U·V, thereby ensuring minimal loss of model performance.

[0060] 3. Adaptive optimization of parameter importance.

[0061] Different parameters have different effects on model output results. This embodiment can effectively improve resource utilization efficiency and optimization accuracy of model fine-tuning by identifying and prioritizing key parameters. This includes:

[0062] 3.1. For the parameter matrix after low-rank decomposition, calculate the gradient of the loss function with respect to each parameter, and use this as a measure of parameter importance, denoted as IScore.

[0063] 3.2. Sort the parameters according to their gradient size and select the parameters with the top 20% gradient ranking as the key parameters that have the greatest impact on the output results.

[0064] 3.3. For the selected key parameters, the following radial function is used to distribute the learning rate:

[0065] η w =η base exp(γ|IScore w -IScore min | 2 )

[0066] In the above formula, η w IScore is the learning rate assigned to the key parameter w, min IScore is the importance of the key parameter that has the least impact on the model output, that is, the minimum gradient value among the key parameters. w is the importance of parameter w (i.e. gradient value), γ is an adjustable parameter, η base is the baseline learning rate.

[0067] 4. Fine-tune the model parameters based on the assigned learning rate. This process is divided into the following stages:

[0068] (1) Pre-training stage: Pre-training is performed on the target dataset (e.g., human resource allocation data in the power industry) to ensure that the model has a certain understanding of the basic characteristics of the field.

[0069] (2) Fine-tuning phase: After the key parameters are screened, the model parameters are optimized in a targeted manner using the assigned learning rate. Layer-by-layer optimization is used to ensure that the fine-tuning process can maximize the model's adaptability to the task.

[0070] (3) Dynamic adjustment of learning rate: During fine-tuning, the learning rate is dynamically adjusted based on the model’s convergence. This includes lowering the baseline learning rate when the model loss decreases gradually. This process is implemented through the learning rate scheduler in PyTorch, allowing the model to maintain good stability and accuracy during fine-tuning.

[0071] 5. After fine-tuning is completed, verify the performance of the model through actual tasks to ensure that the model fine-tuning effect meets business requirements.

[0072] Taking the human resource allocation task in the power industry as an example, the main evaluation indicators include job matching accuracy and response time. The specific verification steps are as follows:

[0073] (1) Matching accuracy: The matching accuracy of the fine-tuned model was verified on the job matching test set. The test results showed that the matching accuracy increased by about 15%.

[0074] (2) Resource consumption: By comparing the traditional full-parameter fine-tuning method, the resource consumption during the fine-tuning process was tested. The test showed that the low-rank decomposition and adaptive optimization strategy reduced the computational cost by about 50%, significantly reducing the demand for hardware resources.

[0075] (3) Response speed: In a real-world deployment test environment, the model response speed is measured by simulating actual job configuration requirements.

[0076] The above experiments prove that the fine-tuned model shortens the business request response time by about 40%, meeting the power industry's demand for real-time human resource allocation.

[0077] Example 2:

[0078] A parameter fine-tuning system for large human resource models based on low-rank decomposition, such as Figure 1 As shown, it includes parameter fine-tuning layer selection module, low-rank decomposition module, key parameter screening module, and fine-tuning module.

[0079] The parameter fine-tuning layer selection module is used to select the layer that needs parameter fine-tuning based on the parameter structure in the human resources model. The specific strategy is as follows:

[0080] The gradient norm of each layer in the human resource model is calculated according to the following formula, and the layer with the largest gradient norm is selected as the layer that needs parameter fine-tuning:

[0081] L2=∑w ij 2

[0082] In the above formula, L2 is the gradient norm of a certain layer, w ij are the parameters in this layer.

[0083] The low-rank decomposition module is used to perform low-rank decomposition on the parameter matrix of the selected layer. The specific strategy is:

[0084] For a parameter matrix of dimension d*d Decompose it into a low-rank matrix through low-rank decomposition and low-rank matrix Where k is the dimension of the low-rank matrix, and

[0085] The key parameter screening module is used to first calculate the gradient of the loss function for each parameter in the parameter matrix after low-rank decomposition, then sort the parameters according to the gradient size, and select the parameters with the top N gradient values as the key parameters that have the greatest impact on the output results.

[0086] The fine-tuning module is used to distribute the learning rates of key parameters using the following radial function, and optimize the model parameters based on the adjusted learning rates:

[0087] η w =η base exp(γ|IScore w -IScore min | 2 )

[0088] In the above formula, η w IScore is the learning rate assigned to the key parameter w, min IScore is the importance of the key parameter that has the least impact on the model output, that is, the minimum gradient value among the key parameters. w is the importance of parameter w (i.e. gradient value), γ is an adjustable parameter, η base is the baseline learning rate.

[0089] Example 3:

[0090] A parameter fine-tuning device for large human resource models based on low-rank decomposition, such as Figure 2 As shown, it includes a processor and a memory;

[0091] The memory is used to store computer program code and transmit the computer program code to the processor;

[0092] The processor is configured to execute the method for fine-tuning parameters of a large human resources model based on low-rank decomposition as described in Example 1 according to the instructions in the computer program code.

[0093] Example 4:

[0094] A computer storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the method for fine-tuning parameters of a large human resources model based on low-rank decomposition described in Example 1 are implemented.

Claims

1. A method for fine-tuning parameters of a large human resource model based on low-rank decomposition, characterized in that: The method comprises: S1. Select the layer that needs parameter fine-tuning based on the parameter structure in the human resources model; S2. Perform low-rank decomposition on the parameter matrix of the selected layer; S3. Select several parameters that have the greatest impact on the model output results from the parameter matrix after low-rank decomposition as key parameters; S4. Assign learning rates to each key parameter according to its importance, and optimize and adjust the model parameters based on the assigned learning rates.

2. The method for fine-tuning parameters of a large human resource model based on low-rank decomposition according to claim 1 is characterized in that: The S4 uses the following radial function to distribute the learning rates of key parameters: η w =η base exp(γ|IScore w -IScore min | 2 ) In the above formula, η w IScore is the learning rate assigned to the key parameter w, min IScore is the importance of the key parameters that have the least impact on the model output results. w is the importance of parameter w, γ is an adjustable parameter, η base is the baseline learning rate.

3. The method for fine-tuning parameters of a large human resource model based on low-rank decomposition according to claim 1 is characterized in that: The S2 includes: for the parameter matrix with dimension d*d Decompose it into a low-rank matrix through low-rank decomposition and low-rank matrix Where k is the dimension of the low-rank matrix, and 4. The method for fine-tuning parameters of a large human resource model based on low-rank decomposition according to claim 1, characterized in that: Said S1 comprises: The gradient norm of each layer in the human resource model is calculated according to the following formula, and the layer with the largest gradient norm is selected as the layer that needs parameter fine-tuning: In the above formula, L2 is the gradient norm of a certain layer, w ij is the parameter in this layer; The S3 includes: for the parameter matrix after low-rank decomposition, first calculating the gradient of the loss function for each parameter therein, then sorting the parameters according to the gradient size, and selecting the parameters with the gradient values in the top N as key parameters.

5. A parameter fine-tuning system for a large human resource model based on low-rank decomposition, characterized in that: The system includes a parameter fine-tuning layer selection module, a low-rank decomposition module, a key parameter screening module, and a fine-tuning module; The parameter fine-tuning layer selection module is used to select the layer that needs parameter fine-tuning according to the parameter structure in the human resources large model; The low-rank decomposition module is used to perform low-rank decomposition on the parameter matrix of the selected layer; The key parameter screening module is used to select several parameters that have the greatest impact on the model output results from the parameter matrix after low-rank decomposition as key parameters; The fine-tuning module is used to allocate learning rates to key parameters according to their importance, and optimize and adjust model parameters based on the allocated learning rates.

6. The human resource model parameter fine-tuning system based on low-rank decomposition according to claim 5 is characterized in that: The fine-tuning module uses the following radial function to distribute the learning rates of key parameters: η w =η base exp(γ|IScore w -IScore min | 2 ) In the above formula, η w IScore is the learning rate assigned to the key parameter w, min IScore is the importance of the key parameters that have the least impact on the model output results. w is the importance of parameter w, γ is an adjustable parameter, η base is the baseline learning rate.

7. The human resource model parameter fine-tuning system based on low-rank decomposition according to claim 5 is characterized in that: The low-rank decomposition module adopts the following strategy: For a parameter matrix of dimension d*d Decompose it into a low-rank matrix through low-rank decomposition and low-rank matrix Where k is the dimension of the low-rank matrix, and 8. The human resource model parameter fine-tuning system based on low-rank decomposition according to claim 5 is characterized in that: The parameter fine-tuning layer selection module is used to calculate the gradient norm of each layer in the human resource model according to the following formula, and select the layer with the largest gradient norm as the layer that needs parameter fine-tuning: <h2 style=";text-align:left;direction:ltr">L2 = ∑w<h2 style=";text-align:left;direction:ltr"> ij <h2 style=";text-align:left;direction:ltr"> 2 In the above formula, L2 is the gradient norm of a certain layer, w ij is the parameter in this layer; The key parameter screening module is used to calculate the gradient of the loss function for each parameter in the parameter matrix after low-rank decomposition, sort the parameters according to the gradient size, and select the parameters with the top N gradient values as the key parameters that have the greatest impact on the model output results.

9. A parameter fine-tuning device for a large human resource model based on low-rank decomposition, characterized in that: including a processor and a memory; The memory is used to store computer program code and transmit the computer program code to the processor; The processor is used to execute the method for fine-tuning parameters of a large human resources model based on low-rank decomposition according to any one of claims 1 to 4 according to the instructions in the computer program code.

10. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of a method for fine-tuning parameters of a large human resources model based on low-rank decomposition according to any one of claims 1 to 4 are implemented.

Images