Task awareness-based expert large model occupational injury identification fine tuning method and system

Abstract

The invention relates to the technical field of large models, and provides an expert large model occupational injury identification fine tuning method and system based on task awareness. The task awareness-based expert large model industrial injury identification fine tuning method comprises the following steps: constructing a structural industrial injury identification data set based on a teacher model; designing a student model-oriented task perception fine tuning strategy; performing knowledge distillation driven supervised fine tuning; and evaluating professional performance and general performance of the fine-tuned large model. According to the method, on the basis of an LLamaFactory fine tuning framework and a structural chemical injury identification data set, injury condition diagnosis and similar terms are taken as model input, a thinking process and an identification result are taken as expected output, a student model is supervised and trained by utilizing a task perception fine tuning strategy, and the fine tuning performance of the model is evaluated. According to the method, efficient and accurate fine adjustment can be carried out according to the framework characteristics of the MoE model under the finite computing power, so that the professional ability of model occupational injury identification is improved, and meanwhile, the universal performance of the model occupational injury identification is guaranteed.

Description

Technical Field

[0001] This invention relates to the field of large model technology, and in particular to a method and system for fine-tuning work-related injury identification based on task awareness expert large models. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] With the continuous increase in the number of work-related injury assessments, the complexity of injury diagnosis is also increasing. Currently, work-related injury assessment relies heavily on expert review, and its process suffers from problems such as low efficiency, inconsistent standards, and strong subjectivity, making it difficult to meet the urgent needs of work-related injury assessment for efficiency and accuracy. Large language models, with their powerful natural language understanding and generation capabilities, provide a new technical path for the intelligent and standardized process of work-related injury assessment. Currently, the mainstream large language model architectures are mainly divided into dense models and hybrid expert (MoE) models. Dense models adopt the standard Transformer architecture, where all parameters are activated when processing each input. Although powerful, their computational cost is high, limiting their large-scale deployment and fine-tuning. Hybrid expert models, as a sparse architecture, introduce a gating mechanism to activate only a portion of the expert network for different inputs. This significantly reduces computational overhead and inference latency while maintaining the performance and capacity of dense models, demonstrating great potential for handling complex professional tasks.

[0004] However, adapting hybrid expert models to the highly specialized field of work injury assessment still faces significant challenges. First, there is a lack of domain-specific knowledge: general-purpose models lack a deep understanding of clauses, injury diagnoses, and assessment rules, leading to issues such as missing key information or improper clause citations when generating assessment conclusions. Second, full-parameter fine-tuning is extremely resource-intensive: for MoE models with hundreds of billions of parameters, conventional full-parameter fine-tuning requires extremely high GPU memory and computing power, making it difficult to implement in resource-constrained scenarios. Although parameter-efficient fine-tuning methods such as LoRA can effectively reduce computational overhead, their fixed rank dimension is difficult to adapt to the diverse sub-tasks in work injury assessment (such as clause matching, logical reasoning, and result determination), resulting in insufficient model adaptability. Furthermore, the unique architecture of the MoE model introduces new technical difficulties; a knowledge gap exists between its expert and non-expert layers, with the expert layer's professional characteristics not being effectively utilized by the non-expert layer, leading to logical inconsistencies in the model's reasoning process. Simultaneously, direct fine-tuning is highly susceptible to catastrophic forgetting; while the model acquires professional knowledge, its original general knowledge and reasoning abilities significantly degrade, affecting the reliability of the output. Summary of the Invention

[0005] To address the aforementioned technical issues, this invention provides a task-aware expert large model work injury assessment fine-tuning method and system, which can efficiently and accurately fine-tune the MoE model architecture under limited computing power, thereby improving the model's professional work injury assessment capabilities while ensuring its general performance.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: The first aspect of the present invention provides a method for fine-tuning a task-aware expert large model for work-related injury identification.

[0007] In one or more embodiments, a method for fine-tuning a task-aware expert large model for work-related injury identification is provided, including: A structured work injury assessment dataset was constructed based on a teacher model, which includes work injury assessment data such as injury diagnosis, similar clauses, thought process, and assessment results. The design of a task-aware fine-tuning strategy for student models includes: employing a dynamic rank adaptive method to automatically adjust the LoRA rank based on task type to optimize parameter allocation; introducing a cross-layer attention compensation module to transfer expert layer features to non-expert layers through attention weights to eliminate knowledge gaps between layers; designing a hybrid loss function that integrates expert layer feature matching loss to enhance the consistency within the MoE model under specific tasks; and applying a hierarchical unfreezing strategy to unlock parameters layer by layer from top to bottom and stack LoRA to effectively suppress overfitting. The process of performing knowledge distillation-driven supervised fine-tuning is based on the LLamaFactory framework and a structured chemical injury identification dataset. The model takes injury diagnosis and similar clauses as inputs and the thought process and identification results as expected outputs. The student model is trained in a supervised manner using a task-aware fine-tuning strategy.

[0008] A second aspect of the present invention provides a task-aware expert large model-based fine-tuning system for work-related injury identification.

[0009] In one or more embodiments, a task-aware expert large model-based fine-tuning system for work-related injury assessment includes: The dataset construction module is used to build a structured work injury identification dataset based on the teacher model. It includes work injury identification data such as injury diagnosis, similar clauses, thought process and identification results. The fine-tuning strategy design module is used to design task-aware fine-tuning strategies for student models. These strategies include: adopting a dynamic rank adaptive method to automatically adjust the LoRA rank based on the task type to optimize parameter allocation; introducing a cross-layer attention compensation module to transfer expert layer features to non-expert layers through attention weights to eliminate knowledge gaps between layers; designing a hybrid loss function that integrates expert layer feature matching loss to enhance the consistency of the MoE model under specific tasks; and applying a hierarchical unfreezing strategy to unlock parameters layer by layer from top to bottom and stack LoRA to effectively suppress overfitting. The supervised fine-tuning module is used to perform knowledge distillation-driven supervised fine-tuning. The process is based on the LLamaFactory framework and the structured chemical injury identification dataset. It takes injury diagnosis and similar clauses as model inputs and thought process and identification results as expected outputs. It uses a task-aware fine-tuning strategy to supervise the training of student models.

[0010] A third aspect of the present invention provides a computer-readable storage medium.

[0011] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the task-aware expert large model work injury identification fine-tuning method described above.

[0012] A fourth aspect of the present invention provides an electronic device.

[0013] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the task-aware expert large model work injury identification fine-tuning method described above.

[0014] Compared with the prior art, the beneficial effects of the present invention are: This invention provides a task-aware fine-tuning method for expert large-scale models in work injury assessment. By constructing a structured dataset based on a teacher model, it provides high-quality, standardized supervision signals for model fine-tuning, ensuring the model learns accurate professional knowledge and reasoning logic for work injury assessment. A task-aware fine-tuning strategy is designed, employing the dynamic rank adaptive LoRA method to intelligently allocate computational resources based on task complexity, significantly improving parameter efficiency while maintaining performance. A cross-layer attention compensation module is introduced to effectively promote knowledge flow between expert and non-expert layers, enhancing the logical consistency and coherence of the model in the professional reasoning process. A method integrating expert... A hybrid loss function based on layer feature matching enhances the consistency between layers within the model and improves task adaptability. A layered unfreezing and LoRA stacking strategy ensures stable and controllable training, effectively suppressing overfitting risks. Supervised fine-tuning driven by knowledge distillation efficiently transfers the professional capabilities of the teacher model to the student model, significantly improving the model's accuracy in answering professional questions related to work injury assessment. With limited computing power, this approach integrates task-aware fine-tuning strategies and knowledge distillation to achieve efficient and accurate fine-tuning of a large-scale hybrid expert model, improving its question-answering accuracy in the field of work injury assessment while effectively mitigating the forgetting of general knowledge. Attached Figure Description

[0015] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0016] Figure 1 This is a flowchart of the task-aware expert large model work injury identification fine-tuning method according to an embodiment of the present invention; Figure 2 This is an overall structural diagram of the design fine-tuning strategy according to an embodiment of the present invention; Figure 3 This is a graph showing the loss variation during fine-tuning training in an embodiment of the present invention. Figure 4 This is a performance comparison chart of the model before and after fine-tuning in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of the task-aware expert large model work injury identification fine-tuning system according to an embodiment of the present invention. Detailed Implementation

[0017] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0018] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0019] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0020] Figure 1 A flowchart of the task-aware expert large model-based fine-tuning method for work-related injury identification, according to an embodiment of the present invention, is provided. Figure 1 The task-aware expert large model work injury identification fine-tuning method of this embodiment may include the following steps S1 to S5.

[0021] The specific implementation process of steps S1 to S5 is as follows: Step S1: Construct a structured work injury assessment dataset based on the teacher model, which includes work injury assessment data such as injury diagnosis, similar clauses, thought process, and assessment results.

[0022] In the specific implementation process, based on the national standard provisions, and combined with the search enhancement generation (RAGflow) technology, the Qwen3-235B-A22B teacher model interface is called to generate a structured fine-tuning dataset. Among them, model input Including injury diagnosis and similar terms Expected output Includes the thought process and identification results .

[0023] The process of constructing a structured chemical injury identification dataset based on a teacher model includes: S11. Construct a clause pool by extracting and organizing clauses from relevant documents (such as "Disability Assessment of Employees Due to Work-Related Injuries and Occupational Diseases"). Individual work injury clauses, constructing a pool of national standard clauses. Each clause describes the level of assessment and the details of the injuries in the case. S12. Generate work injury assessment data, for each national standard clause. Similar terms are retrieved by searching the RAGflow knowledge base. ,in This represents a knowledge base in the field of work-related injury assessment. It includes instructions and prompts for constructing integrated clauses. Input into the teacher model Generates information including injury diagnosis, thought process, and assessment results;

[0024] S13. Format the work injury assessment data, including the injury diagnosis. Similar terms Concatenate to build model input The thought process With the identification results Concatenate to build the model and expect the output Reconstruct training samples ; S14. Set filtering rules This process cleans up vague expressions and digressions in injury diagnoses, removes self-created clauses from assessment results, further improves the quality of work injury assessment data, and retains the filtered high-quality data. This data will serve as a monitoring signal source for subsequent fine-tuning.

[0025] Step S2: Design a task-aware fine-tuning strategy for the student model, which includes adopting a dynamic rank adaptive method to automatically adjust the LoRA rank according to the task type to optimize parameter allocation; introducing a cross-layer attention compensation module to transfer expert layer features to non-expert layers through attention weights to eliminate knowledge gaps between layers; designing a hybrid loss function that integrates expert layer feature matching loss to enhance the consistency of the MoE model under specific tasks; and applying a hierarchical unfreezing strategy to unlock parameters layer by layer from top to bottom and stack LoRA to effectively suppress overfitting.

[0026] This embodiment uses Qwen3-30B-A3B as the student model requiring fine-tuning. Considering its sparse activation architecture and the professional requirements of the work injury assessment task, a fine-tuning scheme is designed, such as... Figure 2 As shown.

[0027] S21. This embodiment employs the dynamic rank adaptive LoRA method to address the challenge of fixed-rank LoRA adapting to the diverse nature of work injury assessment tasks (such as clause matching, reasoning logic, and result determination). Its core principle is based on the current input sample... The task complexity is automatically calculated and a dedicated LoRA rank is assigned. The implementation process of the dynamic rank adaptive LoRA method is as follows: S211. Constructing a lightweight task-aware router network The network consists of an embedding layer and a multilayer perceptron, used to analyze input samples. The semantic features are extracted, and a fixed-dimensional task embedding vector is output. Its mathematical expression is:

[0028] in, Represents the first in the dataset The input text for each sample, To obtain learnable parameters for the router network. This represents the type of input task generated by the router. Dimensional task embedding vector.

[0029] S212, Based on task embedding vectors The LoRA rank applicable to the current sample is dynamically calculated using a rank predictor. This rank predictor uses multiple preset rank candidate values. This is achieved through soft selection, by calculating the weight vector for each candidate rank. :

[0030] in, and Let these represent the weight matrix and bias vector of the rank predictor, respectively. This is a temperature coefficient used to control the smoothness of the weight distribution. Given a probability distribution vector, its elements Represents candidate rank The weight, It is a normalized exponential function.

[0031] S213, The final dynamic rank All candidate ranks are based on weights The weighted expected value is calculated using the following formula:

[0032] in, The calculated value applicable to the current input The dynamic LoRA rank of is a continuous real value. For the preset first One candidate rank value.

[0033] During the forward propagation of the model, for the linear layers injected with LoRA, the output of the model is supplemented with a LoRA adaptation term that has been dynamically rank-scaled, based on the original activation values. The specific calculation is as follows: ; in, It is a dynamic rank; This represents the original input activation value of the linear layer. Let be the lower projection matrix of LoRA. Let be the upper projection matrix of LoRA. For the dimension of the hidden layer, The maximum basic rank is the preset value, representing and The actual number of columns and rows; This means that when calculating the gradient, the variable inside the parentheses is treated as a constant, preventing the gradient from backpropagating to this path; This is the final output activation value of the linear layer. For relatively simple subtasks (such as direct clause lookup), the model automatically assigns a smaller effective rank to save computational resources; for complex subtasks (such as those involving judgment of injury logic), the model assigns a larger effective rank to enhance the model's representational ability and task performance.

[0034] S22. This embodiment introduces a cross-layer attention compensation module. By constructing a feature information path from the expert layer to the non-expert layer, it enhances the logical consistency of the model during the reasoning process and alleviates the knowledge disconnect problem between the expert layer and the non-expert layer of the hybrid expert model. The implementation process of the cross-layer attention compensation module is as follows: S221. Define the input feature representation for the non-expert layer. Let the first layer be... The input to the layer Transformer block is ,in This indicates the batch size during model training. This represents the number of tokens in the input sequence. This represents the dimension of the model's hidden layers.

[0035] S222. Construct an expert layer feature library. Collect the first... The nearest layer The output features of each expert layer form the expert layer feature set:

[0036] in, Represents the feature set of the expert layer. Indicates the first The output characteristics of each expert layer The number of expert layers selected.

[0037] S223. Concatenate all features in the expert layer feature set along the sequence dimension to form a unified key-value pair feature library. :

[0038] S224. Calculate cross-layer attention compensation. This compensation mechanism uses non-expert layer features as queries and the expert layer feature library as keys and values, and is implemented using standard attention calculation methods:

[0039] in, , and These represent the query, key, and value projection matrices, respectively. Let be the dimension of the key vector in the attention mechanism. Let be the dimension of the value vector in the attention mechanism. This represents the calculated cross-layer attention compensation feature.

[0040] S225. Finally, the cross-layer attention compensation features are weighted and fused with the original attention output to obtain the final output of the non-expert layer:

[0041] in, This represents the original self-attention mechanism output from the non-expert layer. These are learnable scaling factors used to dynamically adjust the contribution of the compensation features. This represents the enhanced output features of the non-expert layer. The cross-layer attention compensation module effectively establishes a feature information flow channel from the expert layer to the non-expert layer, enabling the professional knowledge contained in the expert layer to be effectively transferred to the non-expert layer, significantly improving the model's reasoning consistency and knowledge utilization efficiency in work injury identification tasks.

[0042] S23. This embodiment designs a hybrid loss function to achieve a balanced improvement in model performance through multi-objective optimization, ensuring language modeling ability, learning teacher knowledge, and enhancing the consistency between layers within the MoE model. The hybrid loss function consists of three parts, and its mathematical definition and implementation process are as follows: S231. Define a standard causal language modeling loss, which is a continuation of the pre-training task, ensuring that the model can accurately predict the next token and maintain basic language generation capabilities:

[0043] in, This represents the loss in standard causal language modeling. The length of the output sequence. Indicates the first position in the output sequence The real token for each location These are the learnable parameters for the student model. This indicates that the student model performs well given input. and historical output Under the condition of predicting the first Each token is The conditional probability.

[0044] S232. Define the expert layer feature matching loss to constrain the non-expert layer output distribution to align with the expert layer output distribution, thereby enhancing the consistency between layers within the MoE model:

[0045] in, This represents the feature matching loss at the expert level. This represents the selected set of non-expert layers to which constraints need to be imposed. Indicates the first The expert network output features selected by the gated network in each MoE layer. Indicates the first The output features of the next standard feedforward neural network layer (non-expert layer) adjacent to the MoE layer, For a normalized exponential function, This means that when calculating the gradient, the input variables are treated as constants, preventing the gradient from backpropagating. Represents relative entropy, used to quantify the difference between two probability distributions.

[0046] S233. Define knowledge distillation loss, which transfers knowledge from the teacher model to the student model by aligning the output distributions of the student model and the teacher model:

[0047] in, This represents the knowledge distillation loss. This indicates that the student model responds to the input. The output logits, This indicates that the teacher model responds to the input. The output logits, This represents the temperature coefficient of knowledge distillation, used to control the smoothness of the output distribution.

[0048] S234. In summary, the formula for calculating the mixture loss function is as follows:

[0049] in, For the final hybrid loss function, The weighting coefficients representing the knowledge distillation loss. This represents the weight coefficients of the feature matching loss at the expert level. By designing a hybrid loss function, a multi-objective optimization strategy is implemented, effectively improving the model's professional performance in work injury identification tasks while mitigating the catastrophic forgetting problem.

[0050] S24. This embodiment employs a layered unfreezing strategy, which gradually unlocks model parameters from top to bottom and then adds a LoRA adapter to achieve progressive release of model capacity and fine control over the training process. The implementation process of the layered unfreezing strategy is as follows: S241. Define the parameter unfreezing strategy during the training process. Let the total number of training stages be... The total number of layers in the model is Before training begins, freeze all parameters of the student model:

[0051] in, This represents the set of model parameters that are frozen in the initial state. Indicates the index layer. Indicates the lowest level. Indicates the top level. Indicates the first The set of basic parameters for a layer.

[0052] S242. Unfreeze the model parameters in stages, following a top-down approach. In the... During each training phase, the set of unfrozen layers is defined as follows:

[0053] in, Indicates the first The set of layers that were unfrozen during each training phase. Indicates the first The starting layer index for unfreezing during each training phase. The top-down unfreezing strategy satisfies... When unfreezing each layer, a dynamically rank-adaptive LoRA adapter is simultaneously superimposed on the linear projection module of that layer. For the The layer, whose LoRA parameter set is defined as:

[0054] in, Indicates based on input sample The LoRA rank obtained through dynamic calculation Indicates the first The number of linear projection modules contained in the layer. Indicates the first Layer LoRA projection matrix of each linear module Indicates the first Layer The LoRA projection matrix of each linear module.

[0055] S243. For the MoE model, the parameter update method is as follows:

[0056] in, Indicates the first The set of trainable parameters for each MoE layer Indicates the first Parameters of the gated network for each MoE layer Indicates the first Number of experts per MoE layer Indicates the first Layer A set of LoRA parameters from several experts was used. A tiered unfreezing strategy was employed to achieve fine-tuning of large-scale MoE models. This method allows the model to be adaptively adjusted progressively from high-level semantic concepts to low-level language representations, effectively controlling model capacity and training dynamics while ensuring training stability, significantly reducing the risk of overfitting, and improving fine-tuning efficiency.

[0057] Step S3: Perform knowledge distillation-driven supervised fine-tuning. The process is based on the LLamaFactory framework and the structured chemical injury identification dataset. The model inputs are injury diagnosis and similar clauses, and the expected outputs are the thought process and identification results. The student model is trained in a supervised manner using a task-aware fine-tuning strategy.

[0058] In the process of performing knowledge distillation-driven supervised fine-tuning, let the structural chemical damage identification dataset be... , the dataset The dataset was divided into training, validation, and test sets in a 7:2:1 ratio. During fine-tuning, injury diagnoses and similar clauses generated by the teacher's model were used as input to the model. The expected output is the thought process and assessment results generated by the teacher model. By minimizing the mixture loss function This drives the large model to learn professional knowledge and reasoning logic in the field of work injury assessment. The curve showing the change of the loss function during model fine-tuning is shown below. Figure 3 As shown.

[0059] The specific process of performing knowledge distillation-driven supervised fine-tuning is as follows: S31. Fine-tune the environment and framework configuration. Configure the DeepSpeed ​​distributed training environment to support large-scale model training. Use ZeRO-3 to partition optimizer parameters, gradients, and model parameters to avoid memory overflow. Initialize the LLamaFactory high-efficiency fine-tuning framework, integrate fine-tuning strategies, and uniformly set batch size, initial learning rate, and maximum number of training epochs. The learning rate scheduler uses a warm-up cosine annealing strategy, and the optimizer uses AdamW.

[0060] S32. Data loading and preprocessing: The constructed structured dataset The dataset was divided into training, validation, and test sets in a 7:2:1 ratio. In the data loader, each sample was standardized: the input (injury diagnosis and similar clauses) was concatenated into prompts according to a predefined template; the expected output (thought process and assessment result) was concatenated into a complete answer sequence.

[0061] S33. Training Loop and Knowledge Distillation: In each training iteration, the following core steps are performed: S331, Forward Propagation: Input data is fed into the student model. During processing, the model dynamically calculates the LoRA rank, performs cross-layer attention compensation, and finally outputs the probability distribution of the predicted sequence. S332. Calculate the gradient: Feed the model output and the true labels into the defined mixture loss function. In this process, the gradient of the total loss with respect to the trainable parameters of the model is calculated. S333, Backpropagation: Following a layered unfreezing strategy, gradient updates are performed only on the model layers that have been unfrozen during the current training phase (including their basic parameters and the stacked LoRA adapter parameters). Gradient pruning is used to prevent gradient explosion. S334. Training Monitoring: Real-time recording of training loss, loss values ​​for each component, and evaluation metrics. Periodic evaluation on the validation set, monitoring the model status based on validation set loss or task-specific metrics (accuracy of identification results), and deciding whether to stop training early or adjust the learning rate accordingly to prevent overfitting.

[0062] S34. Model Saving and Export: After the model training converges, save the final fine-tuned student model weights, LoRA adapter parameters, and complete model configuration file. The fine-tuned Qwen3-SFT expert model encapsulates the professional knowledge and reasoning logic of work injury assessment distilled from the teacher model, while retaining the general capabilities of the base model, preparing for subsequent professional and general performance tests.

[0063] Step S4: Evaluate the professional performance of the fine-tuned model. Using injury diagnoses and similar clauses from the test set as input, supplemented by pre-defined identification rule prompts, the fine-tuned model is invoked to generate responses. Includes model thinking patterns and identification results By comparing the identification results output by the model with the true labels in the test set. The accuracy of the model is calculated to quantitatively evaluate its performance improvement in work injury assessment. The specific steps for performance evaluation are as follows: S41. The input part (injury diagnosis and similarity clause) of each sample in the test set is concatenated with a predefined structured identification rule prompt to form a reasoning instruction with complete contextual information.

[0064] S42. Input this series of instructions into the fine-tuned student model in batches to drive the model to perform forward calculations and automatically generate a process including the thought process. and identification results The complete sequence of answers.

[0065] S43. Using a programmed post-processing script, parse all the answers generated by the model and extract the identification result fields. Compared with the real labels pre-labeled in the test set Perform matching and comparison.

[0066] S44, Statistics and The accuracy of the model in the work injury assessment task is obtained by calculating the number of samples with the same assessment level and their proportion of the total test samples. These steps need to be performed separately in the base model and the fine-tuned model to compare their professional performance in the work injury assessment task.

[0067] Step S5: Evaluate the general performance of the fine-tuned model. Using the EvalScope general evaluation framework, benchmark tests are conducted in four domains: test questions, knowledge, mathematics, and code. This verifies that the method of this invention effectively alleviates the catastrophic forgetting problem while injecting professional knowledge. The specific steps for general performance evaluation are as follows: S51. Select the standard evaluation sets of the four domains of test questions, knowledge, mathematics and code in the EvalScope framework, and load the base model before fine-tuning and the model after fine-tuning into the unified testing environment respectively.

[0068] S52. Input the standard test questions into the two models in sequence, and obtain the model output in batches, covering subject answers, knowledge questions, mathematical reasoning and code generation tasks.

[0069] S53. The evaluation framework automatically calculates and records the performance index scores of the two models in various fields, and generates a structured evaluation report.

[0070] S54. Compare the scores of the fine-tuned model in each domain with the base model before fine-tuning, analyze the overall performance of the model, and verify the effectiveness of the present invention in alleviating catastrophic forgetting.

[0071] Table 1. Evaluation results of the model's professional and general performance before and after fine-tuning.

[0072] Based on the evaluation results in Table 1, the fine-tuning method proposed in this invention demonstrates significant effectiveness in optimizing model performance. In terms of professional performance, the model's accuracy in the work injury identification task increased dramatically from 82.8% before fine-tuning to 95.2%, a relative improvement of 15.0%, proving that this method can effectively inject domain-specific knowledge and significantly enhance the model's ability to judge professional issues. In terms of general performance, the model's overall performance remained stable: the score in the knowledge domain increased from 73.4% to 75.5%, showing a certain positive transfer effect; the score in the question domain remained basically stable, indicating that the model's basic cognitive ability was largely preserved; although there were slight decreases in the mathematics and coding domains (2.7% and 3.6% respectively), the fluctuation range was controlled within a reasonable range, and no serious catastrophic forgetting phenomenon occurred. The performance comparison before and after model fine-tuning is shown below. Figure 4 As shown. In summary, this invention significantly improves professional performance while maintaining a good balance in the model's general capabilities, verifying the effectiveness and superiority of this method in terms of vertical domain professional knowledge transfer and general capability retention.

[0073] This invention, verified through a comprehensive performance evaluation system, effectively mitigates the catastrophic forgetting problem while incorporating professional knowledge, enabling the model to maintain stable performance in general capability assessments. Ultimately, under conditions of limited GPU memory resources, it achieves efficient fine-tuning of a 30-parameter expert model. The resulting model combines high accuracy in its professional field with strong adaptability to general scenarios, providing a reliable technical path for large-scale model applications in vertical industries, and possessing significant practical value and promising prospects for widespread adoption.

[0074] like Figure 5 As shown, the task-aware expert large model work injury assessment fine-tuning system provided in this embodiment of the invention is implemented in software. The task-aware expert large model work injury assessment fine-tuning system includes: The dataset construction module 501 is used to construct a structured work injury identification dataset based on the teacher model. It contains work injury identification data including injury diagnosis, similar clauses, thought process and identification results. The fine-tuning strategy design module 502 is used to design a task-aware fine-tuning strategy for the student model. It includes adopting a dynamic rank adaptive method to automatically adjust the LoRA rank according to the task type to optimize parameter allocation; introducing a cross-layer attention compensation module to transfer expert layer features to non-expert layers through attention weights to eliminate knowledge gaps between layers; designing a hybrid loss function that integrates expert layer feature matching loss to enhance the consistency of the MoE model under specific tasks; and applying a hierarchical unfreezing strategy to unlock parameters layer by layer from top to bottom and stack LoRA to effectively suppress overfitting. The supervised fine-tuning module 503 is used to perform knowledge distillation-driven supervised fine-tuning. The process is based on the LLamaFactory framework, using a structured chemical injury identification dataset, with injury diagnosis and similar clauses as model inputs, and the thought process and identification results as expected outputs. The student model is trained in a supervised manner using a task-aware fine-tuning strategy.

[0075] It should be noted that each module in the task-aware expert large model work injury assessment fine-tuning system of this invention corresponds one-to-one with each step in the task-aware expert large model work injury assessment fine-tuning method in the above embodiments, and their specific implementation processes are the same, so they will not be repeated here.

[0076] In some embodiments, the task-aware expert large model work injury assessment fine-tuning system provided in this invention can be implemented using a combination of hardware and software. As an example, the task-aware expert large model work injury assessment fine-tuning system provided in this invention can be a processor in the form of a hardware decoding processor, programmed to execute the task-aware expert large model work injury assessment fine-tuning method provided in this invention. For example, the hardware decoding processor can employ one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0077] As an example, a processor can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where a general-purpose processor can be a microprocessor or any conventional processor, etc.

[0078] As an example of the hardware implementation of the task-aware expert large model work injury assessment fine-tuning system provided in this embodiment of the invention, the device provided in this embodiment of the invention can be directly executed by a processor in the form of a hardware decoding processor. For example, it can be executed by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components to implement the task-aware expert large model work injury assessment fine-tuning method provided in this embodiment of the invention.

[0079] The memory in this embodiment of the invention is used to store various types of data to support the operation of the task-aware expert large model work injury identification fine-tuning system, or to store data for execution. Figure 1 The program code for the method shown. Examples of this data include: any executable instructions for operation on a task-aware expert large model-based work injury assessment fine-tuning system, such as executable instructions that can be included in the executable instructions to implement the task-aware expert large model-based work injury assessment fine-tuning method of this embodiment of the invention.

[0080] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including functions for executing... Figure 1 The program code for the method shown. In such an embodiment, the computer program can be downloaded and installed from a network via a communication component, and / or installed from a removable medium. When the computer program is executed by the central processing unit, it performs the various functions defined in the apparatus of this application.

[0081] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0082] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for fine-tuning a task-aware expert large model for work-related injury identification, characterized in that, include: A structured work injury assessment dataset was constructed based on a teacher model, which includes work injury assessment data such as injury diagnosis, similar clauses, thought process, and assessment results. The design of a task-aware fine-tuning strategy for student models includes adopting a dynamic rank adaptive method to automatically adjust the LoRA rank according to the task type to optimize parameter allocation; introducing a cross-layer attention compensation module to transfer expert layer features to non-expert layers through attention weights to eliminate knowledge gaps between layers; and designing a hybrid loss function that integrates expert layer feature matching loss to enhance the consistency of the MoE model under specific tasks. A layered unfreezing strategy is applied, which unlocks parameters layer by layer from top to bottom and superimposes LoRA to effectively suppress overfitting; The process of performing knowledge distillation-driven supervised fine-tuning is based on the LLamaFactory framework and a structured chemical injury identification dataset. The model takes injury diagnosis and similar clauses as inputs and the thought process and identification results as expected outputs. The student model is trained in a supervised manner using a task-aware fine-tuning strategy.

2. The method for fine-tuning work-related injury identification based on a task-aware expert large model as described in claim 1, characterized in that, The process of automatically adjusting the LoRA rank based on the task type to optimize parameter allocation using the dynamic rank adaptive method is as follows: Building a lightweight task-aware router network Used to analyze input samples The semantic features are extracted, and a fixed-dimensional task embedding vector is output. Its mathematical expression is: ;in, Represents the first in the dataset The input text for each sample, To obtain learnable parameters for the router network. This represents the type of input task generated by the router. 3D task embedding vector; Based on task embedding vectors The LoRA rank applicable to the current sample is dynamically calculated using a rank predictor. The rank predictor uses multiple preset rank candidate values. This is achieved through soft selection, by calculating the weight vector for each candidate rank. : ;in, and Let these represent the weight matrix and bias vector of the rank predictor, respectively. This is a temperature coefficient used to control the smoothness of the weight distribution. Given a probability distribution vector, its elements Represents candidate rank The weight, A normalized exponential function; Final dynamic rank All candidate ranks are based on weights The weighted expected value is calculated using the following formula: ;in, The calculated value applicable to the current input The dynamic LoRA rank of is a continuous real value. For the preset first One candidate rank value.

3. The method for fine-tuning work-related injury identification based on a task-aware expert large model as described in claim 1, characterized in that, During the forward propagation of the student model, for the linear layers injected with LoRA in the student model, the output is calculated by adding a LoRA adaptation term, dynamically rank-scaled, to the original activation values, as follows: ； in, It is a dynamic rank; This represents the original input activation value of the linear layer. Let be the lower projection matrix of LoRA. Let be the upper projection matrix of LoRA. For the dimension of the hidden layer, The maximum basic rank is the preset value, representing and The actual number of columns and rows; This means that when calculating the gradient, the variable inside the parentheses is treated as a constant, preventing the gradient from backpropagating to this path; This is the final output activation value of the linear layer.

4. The method for fine-tuning work-related injury identification based on a task-aware expert large model as described in claim 1, characterized in that, Final output of the non-expert layer : in, This represents the original self-attention mechanism output from the non-expert layer. These are learnable scaling factors used to dynamically adjust the contribution of the compensation features. This represents the enhanced output features of the non-expert layer; This represents the calculated cross-layer attention compensation features; This represents the feature set of the expert layer.

5. The method for fine-tuning work-related injury identification based on a task-aware expert large model as described in claim 1, characterized in that, The hybrid loss function that incorporates the feature matching loss from the expert layer is: in, This represents the loss in standard causal language modeling. The length of the output sequence. Indicates the first position in the output sequence The real token for each location These are the learnable parameters for the student model. This indicates that the student model performs well given input. and historical output Under the condition of predicting the first Each token is The conditional probability; This represents the feature matching loss at the expert level. This represents the selected set of non-expert layers to which constraints need to be imposed. Indicates the first The expert network output features selected by the gated network in each MoE layer. Indicates the first The output features of the next standard feedforward neural network layer (non-expert layer) adjacent to the MoE layer, For a normalized exponential function, This means that when calculating the gradient, the input variables are treated as constants, preventing the gradient from backpropagating. Represents relative entropy, used to quantify the difference between two probability distributions; This represents the knowledge distillation loss. This indicates that the student model responds to the input. The output logits, This indicates that the teacher model responds to the input. The output logits, The temperature coefficient representing knowledge distillation is used to control the smoothness of the output distribution; For the final hybrid loss function, The weighting coefficients representing the knowledge distillation loss. This represents the weight coefficients of the feature matching loss at the expert layer.

6. The method for fine-tuning work-related injury identification based on a task-aware expert large model as described in claim 1, characterized in that, The process of applying a layered unfreezing strategy, unlocking parameters layer by layer from top to bottom and then stacking LoRA is as follows: Define the parameter unfreezing strategy during the training process; Let the total number of training stages be... The total number of layers in the model is Before training begins, freeze all parameters of the student model: ; This represents the set of model parameters that are frozen in the initial state. Indicates the index layer. Indicates the lowest level. Indicates the top level. Indicates the first The basic parameter set of a layer; Following a top-down approach, the model parameters are unfrozen in stages; in the first stage... During each training phase, the set of unfrozen layers is defined as follows: ; Indicates the first The set of layers that were unfrozen during each training phase. Indicates the first The starting layer index at which unfreeze begins during each training phase; the top-down unfreezing strategy satisfies... When unfreezing each layer, a dynamically rank-adaptive LoRA adapter is simultaneously superimposed on the linear projection module of that layer.

7. The method for fine-tuning work-related injury identification based on a task-aware expert large model as described in claim 6, characterized in that, For the Layer, LoRA parameter set is defined as ;in, Indicates based on input sample The LoRA rank obtained through dynamic calculation Indicates the first The number of linear projection modules contained in the layer. Indicates the first Layer LoRA projection matrix of each linear module Indicates the first Layer The LoRA projection matrix of each linear module; For the MoE model, the parameter update method is as follows: ;in, Indicates the first The set of trainable parameters for each MoE layer Indicates the first Parameters of the gated network for each MoE layer Indicates the first Number of experts per MoE layer Indicates the first Layer A set of LoRA parameters for each expert.

8. A task-aware expert large model-based fine-tuning system for work-related injury assessment, characterized in that, The method for fine-tuning work-related injury identification based on a task-aware expert large model as described in any one of claims 1-7 includes: The dataset construction module is used to build a structured work injury identification dataset based on the teacher model. It includes work injury identification data such as injury diagnosis, similar clauses, thought process and identification results. The fine-tuning strategy design module is used to design task-aware fine-tuning strategies for student models. These strategies include: adopting a dynamic rank adaptive method to automatically adjust the LoRA rank based on the task type to optimize parameter allocation; introducing a cross-layer attention compensation module to transfer expert layer features to non-expert layers through attention weights to eliminate knowledge gaps between layers; designing a hybrid loss function that integrates expert layer feature matching loss to enhance the consistency of the MoE model under specific tasks; and applying a hierarchical unfreezing strategy to unlock parameters layer by layer from top to bottom and stack LoRA to effectively suppress overfitting. The supervised fine-tuning module is used to perform knowledge distillation-driven supervised fine-tuning. The process is based on the LLamaFactory framework and the structured chemical injury identification dataset. It takes injury diagnosis and similar clauses as model inputs and thought process and identification results as expected outputs. It uses a task-aware fine-tuning strategy to supervise the training of student models.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the task-aware expert large model work injury identification fine-tuning method as described in any one of claims 1-7.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the task-aware expert large model work injury identification fine-tuning method as described in any one of claims 1-7.

Images