Parameter efficient fine tuning method and device based on hybrid expert system, and image segmentation method and device

Abstract

The invention provides a parameter efficient fine tuning method based on a hybrid expert system, and an image segmentation method and device. The method comprises the following steps: based on an improved sensitivity constraint function and a low-rank matrix constraint function in a self-adaptive low-rank adaptation module; obtaining a target low-rank matrix corresponding to a to-be-trained model parameter of a to-be-trained network layer in the pre-trained segmentation model; obtaining an output result of the to-be-trained network layer according to the target low-rank matrix, a routing decision value of the hybrid expert system and the sample image; according to a segmentation loss value between an image segmentation result corresponding to an output result of the to-be-trained network layer and an image segmentation label of the sample image, and an equilibrium loss value corresponding to each expert module in the hybrid expert system, obtaining a target loss value of the pre-trained segmentation model; and according to the target loss value, performing parameter efficient fine tuning on the to-be-trained model parameter iteration to obtain an image segmentation model. According to the method, the robustness is improved, the inter-task interference is reduced, and the task processing performance is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method for efficient parameter fine-tuning based on a hybrid expert system, and an image segmentation method and device. Background Art

[0002] With the development of deep learning, pre-trained models have achieved remarkable results in natural language processing and computer vision tasks. Although pre-trained models have strong generalization capabilities, in order to best adapt to the needs of specific tasks, they usually need to be fine-tuned based on the pre-trained models.

[0003] At present, the parameters of pre-trained models are usually fine-tuned by adding multi-type data in specific fields to achieve the task processing ability of the fine-tuned model in specific fields. However, on the one hand, these data in specific fields are inevitably subject to external interference during the collection process, making it difficult to ensure their quality, resulting in the difficulty of fine-tuning models based on existing parameter-efficient fine-tuning methods to cope with such complex scenarios and poor robustness; on the other hand, the increase in professional field data types inevitably brings about the problem of interference between tasks during the training process, resulting in poor task processing performance of models fine-tuned based on existing parameter-efficient fine-tuning methods.

[0004] Therefore, there is an urgent need for an efficient parameter fine-tuning method, image segmentation method and device based on a hybrid expert system to solve the above technical problems. Summary of the invention

[0005] The present invention provides a method for efficient parameter fine-tuning, an image segmentation method and a device based on a hybrid expert system, so as to solve the defects of poor robustness and poor task processing performance in the prior art of performing efficient parameter fine-tuning training on a pre-trained model by adding multiple types of data in a specific field, thereby improving the robustness and reducing the interference between tasks to improve the task processing performance.

[0006] The present invention provides a method for efficient parameter fine-tuning based on a hybrid expert system, comprising: Based on the improved sensitivity constraint function and low-rank matrix constraint function in the adaptive low-rank adaptation module, a target low-rank matrix corresponding to the to-be-trained model parameters of the to-be-trained network layer in the pre-trained segmentation model is obtained; the improved sensitivity constraint function is obtained by performing local smoothing optimization on the original sensitivity constraint function; Obtaining an output result of the network layer to be trained according to the target low-rank matrix, the routing decision value of the hybrid expert system and the sample image; Obtaining a target loss value of the pre-trained segmentation model according to a segmentation loss value between an image segmentation result corresponding to an output result of the network layer to be trained and an image segmentation label of the sample image, and a balanced loss value corresponding to each expert module in the hybrid expert system; According to the target loss value, the parameters of the to-be-trained model of the to-be-trained network layer are iteratively fine-tuned to obtain an image segmentation model; the image segmentation model is used to obtain an image segmentation result of the to-be-segmented image according to the segmentation of the to-be-segmented image.

[0007] According to an efficient parameter fine-tuning method based on a hybrid expert system provided by the present invention, the improved sensitivity constraint function and low-rank matrix constraint function in the adaptive low-rank adaptation module are used to obtain a target low-rank matrix corresponding to the parameters of the to-be-trained model of the to-be-trained network layer in the pre-trained segmentation model, including: Calculating the importance of the triples corresponding to the parameters of the model to be trained of the network layer to be trained according to the improved sensitivity constraint function; According to the importance, obtaining the rank dimension of the target low-rank matrix; According to the rank dimension of the target low-rank matrix and the low-rank matrix constraint function, obtaining the target low-rank matrix; The low-rank matrix constraint function is used to constrain the orthogonality of the target low-rank matrix.

[0008] According to an efficient parameter fine-tuning method based on a hybrid expert system provided by the present invention, the importance of the triplet corresponding to the to-be-trained model parameter of the to-be-trained network layer is calculated according to the improved sensitivity constraint function, including: Obtaining local sensitivity smoothed values ​​and local uncertainty quantized smoothed values ​​of the to-be-trained model parameters of the to-be-trained network layer at multiple historical moments before the current moment; Based on the improved sensitivity constraint function, according to the sensitivity value of the to-be-trained model parameter of the to-be-trained network layer at the current moment, local smoothing processing is performed on the local sensitivity smoothing values ​​of the multiple historical moments to obtain the local sensitivity smoothing value of the to-be-trained model parameter of the to-be-trained network layer at the current moment; Determine the sensitivity change of the to-be-trained model parameter of the to-be-trained network layer at the current moment according to the difference between the sensitivity value at the current moment and the local sensitivity smoothing value at the current moment; According to the sensitivity change amount, local smoothing processing is performed on the local uncertainty quantization smoothing values ​​of the multiple historical moments to obtain the local uncertainty quantization smoothing value of the to-be-trained model parameter of the to-be-trained network layer at the current moment; According to the local uncertainty quantization smoothed value at the current moment and the local sensitivity smoothed value at the current moment, the importance of the triplet corresponding to the to-be-trained model parameters of the to-be-trained network layer is obtained.

[0009] According to a method for efficient parameter fine-tuning based on a hybrid expert system provided by the present invention, the output result of the network layer to be trained is obtained according to the target low-rank matrix, the routing decision value of the hybrid expert system and the sample image, including: According to the sample image, obtaining input information of the network layer to be trained; multiplying the target low-rank matrix, the routing decision value, and the input information; According to the multiplication result, the output result of the network layer to be trained is obtained.

[0010] According to a method for efficient parameter fine-tuning based on a hybrid expert system provided by the present invention, the equalization loss value corresponding to each expert module in the hybrid expert system is obtained based on the following steps: Calculating the correlation coefficient between the routing decision value of each of the expert modules and each of the sample images; Obtaining the importance coefficient of the correlation coefficient according to the matching status between the data type of the data processed by each of the expert modules and the data type of each of the sample images; The correlation coefficient and the importance coefficient are correspondingly multiplied to obtain the balanced loss value corresponding to each of the expert modules.

[0011] According to a method for efficient parameter fine-tuning based on a hybrid expert system provided by the present invention, the target loss value of the pre-trained segmentation model is obtained according to the segmentation loss value between the image segmentation result corresponding to the output result of the network layer to be trained and the image segmentation label of the sample image, and the balanced loss value corresponding to each expert module in the hybrid expert system, including: Calculate the variance and mean of the balanced loss values ​​corresponding to all the expert modules; calculating a ratio between the variance and the mean; The ratio and the segmentation loss value are weightedly added to obtain the target loss value.

[0012] According to an efficient parameter fine-tuning method based on a hybrid expert system provided by the present invention, the pre-trained segmentation model is constructed based on a feature embedding layer, a multi-layer feature extraction layer and a mask decoder layer; The network layer to be trained is at least one network layer among the feature embedding layer, multiple feature extraction layers and the mask decoder layer.

[0013] The present invention also provides an image segmentation method, comprising: Obtain the image to be segmented; Inputting the image to be segmented into an image segmentation model to obtain an image segmentation result of the image to be segmented; Wherein, the image segmentation model is trained based on the efficient parameter fine-tuning method based on the hybrid expert system as described in any of the above items.

[0014] The present invention also provides a parameter efficient fine-tuning device based on a hybrid expert system, comprising: A first acquisition module is used to acquire a target low-rank matrix corresponding to the to-be-trained model parameters of the to-be-trained network layer in the pre-trained segmentation model based on the improved sensitivity constraint function and the low-rank matrix constraint function in the adaptive low-rank adaptation module; the improved sensitivity constraint function is obtained by performing local smoothing optimization on the original sensitivity constraint function; A second acquisition module is used to acquire the output result of the network layer to be trained according to the target low-rank matrix, the routing decision value of the hybrid expert system and the sample image; A third acquisition module is used to obtain a target loss value of the pre-trained segmentation model according to a segmentation loss value between an image segmentation result corresponding to an output result of the network layer to be trained and an image segmentation label of the sample image, and a balanced loss value corresponding to each expert module in the hybrid expert system; The training module is used to iteratively and efficiently fine-tune the parameters of the to-be-trained model of the to-be-trained network layer according to the target loss value to obtain an image segmentation model; the image segmentation model is used to obtain an image segmentation result of the to-be-segmented image according to the to-be-segmented image segmentation.

[0015] The present invention also provides an image segmentation device, comprising: A fourth acquisition module, used to acquire the image to be segmented; A segmentation processing module, used for inputting the image to be segmented into an image segmentation model to obtain an image segmentation result of the image to be segmented; Wherein, the image segmentation model is trained based on the efficient parameter fine-tuning method based on the hybrid expert system as described in any of the above items.

[0016] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the efficient parameter fine-tuning method based on the hybrid expert system or the image segmentation method as described above is implemented.

[0017] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-mentioned efficient parameter fine-tuning methods based on a hybrid expert system or image segmentation methods.

[0018] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned efficient parameter fine-tuning methods based on a hybrid expert system or image segmentation methods.

[0019] The present invention provides a method and apparatus for efficient parameter fine-tuning based on a hybrid expert system. The method and apparatus for image segmentation obtain the target low-rank matrix corresponding to the parameters of the model to be trained of the network layer to be trained by smooth optimization based on the improved sensitivity constraint function and the low-rank matrix constraint function in the adaptive low-rank adaptation module, so as to obtain stronger robustness in the fine-tuning training process. The model parameters of the network layer to be trained are updated collaboratively according to the target low-rank matrix and the routing decision value of the hybrid expert system. In the process of updating the model parameters, the segmentation loss value between the image segmentation result corresponding to the output result of the network layer to be trained and the image segmentation label of the sample image, as well as the equalization loss value corresponding to each expert module in the hybrid expert system are introduced to fine-tune the parameters of the model to be trained, so as to achieve equalization processing of all expert modules while ensuring the image segmentation accuracy, thereby improving the processing performance of the trained image segmentation model on tasks in multiple fields, effectively solving the problems of limited data quality and interference between tasks in specific fields, significantly improving the robustness and task processing performance of the model in complex scenarios, and realizing the optimization of the image segmentation model trained by efficient parameter fine-tuning. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 This is one of the flow charts of the efficient parameter fine-tuning method based on the hybrid expert system provided by the present invention.

[0022] Figure 2 This is the second flow chart of the efficient parameter fine-tuning method based on the hybrid expert system provided by the present invention.

[0023] Figure 3 This is the third flow chart of the efficient parameter fine-tuning method based on the hybrid expert system provided by the present invention.

[0024] Figure 4 It is a simulation comparison diagram of the efficient parameter fine-tuning method based on the hybrid expert system provided by the present invention.

[0025] Figure 5 It is a flowchart of the image segmentation method provided by the present invention.

[0026] Figure 6 It is a structural schematic diagram of a parameter efficient fine-tuning device based on a hybrid expert system provided by the present invention.

[0027] Figure 7 It is a structural schematic diagram of the image segmentation device provided by the present invention.

[0028] Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] With the development of deep learning, pre-trained models, such as the Bidirectional Encoder Representations from Transformers (BERT) and Generative Pre-trained Transformer (GPT), have achieved remarkable results in natural language processing and computer vision tasks. These models learn general semantic and visual representations by pre-training on large-scale text or image data. Although pre-trained models have strong generalization capabilities, in order to best adapt to the needs of specific tasks, such as sentiment analysis, named entity recognition, image classification, etc., it is usually necessary to fine-tune the pre-trained model. Traditional fine-tuning often requires a large amount of labeled data and computing resources, especially when the model is large, which leads to long training time and high cost.

[0031] In this regard, relevant technicians have proposed a method of efficient parameter fine-tuning to fine-tune pre-trained models. Efficient parameter fine-tuning significantly reduces the computing resource requirements for training by fine-tuning only a small number of parameters. This enables rapid model iteration and improved experimental efficiency even when resources are limited. By selectively updating parameters in the pre-trained model, the model's generalization ability on new tasks can be better maintained. This means that even during the fine-tuning process of a specific task, the general knowledge learned by the pre-trained model can be fully utilized. Efficient parameter fine-tuning enables researchers and developers to quickly customize models for specific tasks. This flexibility and efficiency are critical to rapid iteration and experimental design.

[0032] At present, relevant technicians usually perform efficient parameter fine-tuning training on pre-trained models by adding multi-type data in specific fields, so as to achieve the task processing capability of the fine-tuned model in specific fields. However, on the one hand, these data in specific fields are inevitably subject to external interference during the collection process, making it difficult to ensure their quality, resulting in the difficulty of fine-tuning models based on the existing efficient parameter fine-tuning methods to cope with such complex scenarios and poor robustness; on the other hand, the increase in professional field data types inevitably brings about the problem of interference between tasks during the training process, resulting in poor task processing performance of models fine-tuned based on the existing efficient parameter fine-tuning methods.

[0033] Therefore, there is an urgent need for an efficient parameter fine-tuning method that is robust and minimizes interference between tasks. To this end, an embodiment of the present application provides an efficient parameter fine-tuning method based on a hybrid expert system. First, in order to overcome the defect of low robustness of existing efficient parameter fine-tuning methods, this method introduces local smoothing optimization processing to the original sensitivity constraint function in the existing adaptive low-rank adaptation (Adaptive Low-Rank Adaptation, AdaLora) module to obtain an improved AdaLora module, so that it can obtain stronger robustness during the training process. Furthermore, in order to minimize the problem of interference between tasks that arises in model fine-tuning training, this method integrates the improved AdaLora module as an expert module into the hybrid expert system and balances the utilization of all expert modules during the training phase. Therefore, by improving the AdaLora module algorithm, integrating the improved AdaLora module into the hybrid expert system, and balancing the utilization of expert modules in the hybrid expert system to perform efficient parameter fine-tuning, it is possible to effectively improve robustness while reducing interference between tasks to improve task processing performance.

[0034] Figure 1 This is one of the flow charts of the method for efficient parameter fine-tuning based on the hybrid expert system provided by the present invention; Figure 1 As shown, the method includes: step 110, step 120, step 130 and step 140.

[0035] Step 110, based on the improved sensitivity constraint function and low-rank matrix constraint function in the adaptive low-rank adaptation module, obtain the target low-rank matrix corresponding to the model parameters to be trained of the network layer to be trained in the pre-trained segmentation model; the improved sensitivity constraint function is obtained by locally smoothing and optimizing the original sensitivity constraint function.

[0036] Among them, the adaptive low-rank adaptation module reduces the redundant parameters of the model and improves the generalization ability of the model by performing low-rank constraints on the model parameters. The adaptive low-rank adaptation module here is an improved AdaLora module that introduces local smoothing optimization on the basis of the original AdaLora module. The adaptive low-rank adaptation module in the embodiment of the present application is also referred to as the improved AdaLora module below.

[0037] Figure 2 FIG. 2 is a flow chart of the efficient parameter fine-tuning method based on the hybrid expert system provided by the present invention; Figure 2 As shown, the improved AdaLora module consists of a random dropout unit (also called Dropout), a low-rank matrix , low-rank matrix Composition and singular matrices Composition; among them, is the rank dimension; and They are low-rank matrices The row dimension and low rank matrix The column dimension of Dropout. Dropout can effectively alleviate the occurrence of overfitting by ignoring some feature detectors and achieve the regularization effect to a certain extent. The low-rank matrix A is used to map the input vector to a low-dimensional latent space, and the low-rank matrix B is used to map the vector in the latent space back to the original high-dimensional space.

[0038] The improved sensitivity constraint function here is used to measure the sensitivity of the model parameters to the input data, so as to constrain the rank dimensions of the low-rank matrix A and the low-rank matrix B based on local smooth optimization, so that the rank dimension of the target low-rank matrix is ​​within a reasonable range; the low-rank matrix constraint function here is used to constrain the orthogonality of the low-rank matrix A and the low-rank matrix B.

[0039] The pre-trained segmentation model here can be constructed based on one or more neural network models.

[0040] Figure 3 FIG3 is a flow chart of the efficient parameter fine-tuning method based on the hybrid expert system provided by the present invention; illustratively, Figure 3 As shown, in some embodiments, the pre-trained segmentation model is constructed based on a feature embedding layer, a multi-layer feature extraction layer and a mask decoder layer; Correspondingly, the network layer to be trained is at least one network layer among the feature embedding layer (or Embedding), multiple layers of the feature extraction layers and the mask decoder layer.

[0041] The feature extraction layer here can be based on an encoder block, such as a transformer module (or TransformerBlock), which includes but is not limited to a linear layer (or Linear), multiple multi-head attention (MAH) modules, a first fusion layer, a multi-layer perception layer, and a second fusion layer. The fusion layer here is also called a normalization layer; wherein the output of the linear layer is connected to the input of each MAH module and to the input of the first fusion layer; the output of each MAH module is connected to the input of the first fusion layer; the output of the first fusion layer is connected to the input of the multi-layer perception layer and the input of the second fusion layer; the output of the multi-layer perception layer is connected to the input of the second fusion layer. The first fusion layer and the second fusion layer here can achieve feature fusion through feature addition and feature normalization.

[0042] The network layer to be trained may be at least one of the feature embedding layer, the feature extraction layer, and the mask decoder layer. For example, the network layer to be trained may be the feature extraction layer. Accordingly, the model parameters to be trained may be the model parameters of at least one multi-head attention module among the multiple multi-head attention modules. The following describes the method provided in the embodiment of the present application by taking the model parameters to be trained as the model parameters of at least one multi-head attention module in each feature extraction layer as an example.

[0043] Optionally, when efficiently fine-tuning parameters, the rank dimension of the target low-rank matrix corresponding to the parameters of the model to be trained can be first obtained based on the improved sensitivity constraint function, and the target low-rank matrix corresponding to the parameters of the model to be trained of the network layer to be trained can be optimized based on the rank dimension of the target low-rank matrix and the low-rank matrix constraint function, so that the target low-rank matrix can be used to update the original model parameters later, reduce the redundant parameters of the model, and improve the generalization ability of the model.

[0044] Step 120, obtaining the output result of the network layer to be trained according to the target low-rank matrix, the routing decision value of the hybrid expert system and the sample image.

[0045] The routing decision here is used to represent the processing weight of each expert module on the input data in the hybrid expert system.

[0046] Optionally, after obtaining the target low-rank matrix, the original output calculation model of the network layer to be trained can be updated based on the target low-rank matrix and the routing decision value of the hybrid expert system, and the input information of the network layer to be trained can be obtained based on the sample image. The input information is input into the updated output calculation model to calculate the output result of the network layer to be trained.

[0047] Step 130, obtaining the target loss value of the pre-trained segmentation model according to the segmentation loss value between the image segmentation result corresponding to the output result of the network layer to be trained and the image segmentation label of the sample image, and the balanced loss value corresponding to each expert module in the hybrid expert system.

[0048] Optionally, after obtaining the output result of the network layer to be trained, the output result of the network layer to be trained can be sequentially input into the network layer after the network layer to be trained in the pre-trained segmentation model until the output result of the last network layer in the pre-trained segmentation model is obtained, thereby obtaining the image segmentation result of the sample image.

[0049] And according to the error between the image segmentation result of the sample image and the image segmentation label of the sample image, the corresponding segmentation loss value is obtained.

[0050] In addition, in order to further avoid the interference between tasks during the fine-tuning process, the equilibrium loss value corresponding to each expert module in the hybrid expert system can be calculated to coordinate the utilization between the expert modules, thereby combating the phenomenon of unbalanced utilization and reducing the interference between tasks during the fine-tuning process.

[0051] Subsequently, the segmentation loss value and the balanced loss value corresponding to each expert module are fused to calculate the target loss value of the pre-trained segmentation model. The fusion here can be achieved by weighted addition of the segmentation loss value and the balanced loss value corresponding to each expert module, or by first fusing the balanced loss value corresponding to each expert module and then fusing it with the segmentation loss value, or by fusing it based on a fusion model, etc. This embodiment does not specifically limit this.

[0052] Step 140, according to the target loss value, iteratively and efficiently fine-tune the parameters of the to-be-trained model of the to-be-trained network layer to obtain an image segmentation model; the image segmentation model is used to obtain an image segmentation result of the to-be-segmented image according to the to-be-segmented image segmentation.

[0053] Optionally, after obtaining the target loss value, the parameters of the model to be trained of the network layer to be trained can be iteratively fine-tuned until the maximum number of training times is reached or the model converges, and then the training is stopped. According to the model performance of the pre-trained segmentation model after each iterative training, the pre-trained segmentation model with the best model performance is determined to construct an image segmentation model that can accurately segment the image to produce the corresponding image segmentation result.

[0054] The method provided in the embodiment of the present application smoothly optimizes and obtains the target low-rank matrix corresponding to the parameters of the model to be trained of the network layer to be trained based on the improved sensitivity constraint function and low-rank matrix constraint function in the adaptive low-rank adaptation module, so as to obtain stronger robustness in the fine-tuning training process, and collaboratively updates the model parameters of the network layer to be trained according to the target low-rank matrix and the routing decision value of the hybrid expert system. In the process of updating the model parameters, the segmentation loss value between the image segmentation result corresponding to the output result of the network layer to be trained and the image segmentation label of the sample image, as well as the equalization loss value corresponding to each expert module in the hybrid expert system are introduced to fine-tune the parameters of the model to be trained, so as to achieve equalization processing of all expert modules while ensuring the image segmentation accuracy, thereby improving the processing performance of the trained image segmentation model on tasks in multiple fields, effectively solving the problems of limited data quality and interference between tasks in specific fields, significantly improving the robustness and task processing performance of the model in complex scenarios, and realizing the optimization of image segmentation models with efficient parameter fine-tuning training.

[0055] In some embodiments, the step of calculating the target low-rank matrix in step 110 includes: Calculating the importance of the triples corresponding to the parameters of the model to be trained of the network layer to be trained according to the improved sensitivity constraint function; According to the importance, obtaining the rank dimension of the target low-rank matrix; According to the rank dimension of the target low-rank matrix and the low-rank matrix constraint function, obtaining the target low-rank matrix; The low-rank matrix constraint function is used to constrain the orthogonality of the target low-rank matrix.

[0056] Optionally, singular value decomposition is performed on the parameters of the model to be trained of the network layer to be trained to obtain triples corresponding to the parameters of the model to be trained of the network layer to be trained. The specific calculation formula is as follows: ; in, The model parameters to be trained for the network layer to be trained; and is a low-rank matrix; for The singular value matrix of ; The triplet corresponding to the model parameters to be trained is the one containing The triplet of singular values ​​and low-rank vectors can be specifically expressed as .

[0057] Furthermore, since the allocation area of ​​the rank dimension of the target low-rank matrix depends on the importance of the triples corresponding to the parameters of the model to be trained; and in the prior art, the importance of the triples is obtained by using the product of the sensitivity and the uncertainty quantization value, but this method only considers the information of the current time point, which is easily affected by external interference, and it is difficult to cope with such complex scenarios, and the robustness is poor. In this regard, the embodiment of the present application improves the original sensitivity constraint function by introducing local smoothing optimization, so that the improved original sensitivity constraint function does not only consider the information of the current time point, but introduces a recent time window M, which includes the M historical moments before the current moment, and is used to improve the sensitivity estimation calculation of the original sensitivity constraint function. The time window includes the sensitivity values ​​of the previous M moments, so as to calculate a smoother and more stable sensitivity estimate and uncertainty quantification, thereby obtaining the importance of the triples corresponding to the parameters of the model to be trained, so as to improve the robust performance of the model.

[0058] After obtaining the importance of the triplets corresponding to the parameters of the model to be trained, the rank dimension of the target low-rank matrix used to update and replace the parameters of the model to be trained can be determined.

[0059] After obtaining the rank dimension, the low-rank matrix A and low-rank matrix B in the singular value decomposition process can be updated in combination with the low-rank matrix constraint function to obtain the target low-rank matrix that satisfies both orthogonality and a specific rank dimension, so that the parameters can be efficiently fine-tuned during training, ensuring that the model parameters of the neural network are updated and optimized with lower computational complexity while maintaining the performance of the model.

[0060] Among them, the low-rank matrix constraint function is used to enforce the orthogonality of the low-rank matrix. The specific calculation formula is as follows: ; and is the low-rank matrix obtained by singular value decomposition. is the identity matrix; is the square of the F-norm function. is the regularization function.

[0061] Exemplarily, in some embodiments, the step of calculating the importance of the triples corresponding to the parameters of the model to be trained includes: Obtaining local sensitivity smoothed values ​​and local uncertainty quantized smoothed values ​​of the to-be-trained model parameters of the to-be-trained network layer at multiple historical moments before the current moment; Based on the improved sensitivity constraint function, according to the sensitivity value of the to-be-trained model parameter of the to-be-trained network layer at the current moment, local smoothing processing is performed on the local sensitivity smoothing values ​​of the multiple historical moments to obtain the local sensitivity smoothing value of the to-be-trained model parameter of the to-be-trained network layer at the current moment; Determine the sensitivity change of the to-be-trained model parameter of the to-be-trained network layer at the current moment according to the difference between the sensitivity value at the current moment and the local sensitivity smoothing value at the current moment; According to the sensitivity change amount, local smoothing processing is performed on the local uncertainty quantization smoothing values ​​of the multiple historical moments to obtain the local uncertainty quantization smoothing value of the to-be-trained model parameter of the to-be-trained network layer at the current moment; According to the local uncertainty quantization smoothed value at the current moment and the local sensitivity smoothed value at the current moment, the importance of the triplet corresponding to the to-be-trained model parameters of the to-be-trained network layer is obtained.

[0062] Optionally, the step of calculating the importance of the triplet corresponding to the parameters of the model to be trained includes: Obtain the local sensitivity smoothed values ​​and local uncertainty quantized smoothed values ​​of the to-be-trained model parameters of the to-be-trained network layer at multiple historical moments before the current moment.

[0063] In addition, the sensitivity value of the model parameters to be trained at the current moment is calculated. The specific calculation formula is as follows: ; Subsequently, based on the improved sensitivity constraint function, according to the sensitivity value of the to-be-trained model parameter of the to-be-trained network layer at the current moment, the local sensitivity smoothing values ​​of multiple historical moments are locally smoothed to obtain the local sensitivity smoothing value of the to-be-trained model parameter of the to-be-trained network layer at the current moment. The specific calculation formula is as follows: ; in, The parameters of the model to be trained exist The local sensitivity smoothing value at the moment; is the size of the sliding window, that is, the number of historical data points included in the smoothing calculation; The parameters of the model to be trained exist The local sensitivity smoothing value at the moment; is the smoothing factor, , used to control the influence of historically sensitive information; The parameters of the model to be trained exist The sensitivity value at the moment.

[0064] In addition, the difference between the sensitivity value at the current moment and the local sensitivity smoothing value at the current moment is calculated to obtain the sensitivity change of the model parameters to be trained at the current moment. The specific calculation formula is as follows: ; in, The parameters of the model to be trained exist The sensitivity change at each moment.

[0065] In addition, according to the sensitivity change, the local uncertainty quantization smoothing values ​​of multiple historical moments are locally smoothed to obtain the local uncertainty quantization smoothing values ​​of the model parameters to be trained at the current moment. The specific calculation formula is as follows: ; in, The parameters of the model to be trained exist The local uncertainty at the moment is quantified and smoothed; is the smoothing factor, , used for the influence of uncertainty quantification; The parameters of the model to be trained exist The local uncertainty at time instant is quantified as a smoothed value.

[0066] Then, the local uncertainty quantization smoothed value at the current moment is multiplied by the local sensitivity smoothed value at the current moment to obtain the importance of the triplet corresponding to the model parameters to be trained. The specific calculation formula is as follows: ; in, The parameters of the model to be trained exist The importance of the triplet corresponding to the moment.

[0067] In summary, the method provided in this embodiment, by introducing the sensitivity constraint function and low-rank matrix constraint function improved by local smoothing optimization, achieves efficient updating and optimization of the neural network model parameters with lower computational complexity while maintaining the performance of the model and improving the robustness of the model.

[0068] In some embodiments, step 120 further includes: According to the sample image, obtaining input information of the network layer to be trained; multiplying the target low-rank matrix, the routing decision value, and the input information; According to the multiplication result, the output result of the network layer to be trained is obtained.

[0069] Optionally, the improved AdaLora module can be introduced into the hybrid expert system as an expert module for parameter update to achieve the acquisition of the output results of the network layer to be trained. Specifically, the hybrid expert system is introduced, and the target low-rank matrix obtained by the improved AdaLora module is used to replace the parameter matrix of the expert module for parameter update, so that the expert modules can achieve efficient parameter fine-tuning through mutual cooperation. The specific steps are as follows: First, the sample image is processed in turn by the network layer before the network layer to be trained, until the output result of the network layer before the network layer to be trained is obtained, and the output result of the network layer before the network layer is used as the input information of the network layer to be trained.

[0070] In order to enable the hybrid expert system to effectively handle various types of tasks, the improved AdaLora module and the hybrid expert system can be used to update the forward propagation calculation of the network layer to be trained, that is, multiply the target low-rank matrix, routing decision value and input information, and calculate the output result of the network layer to be trained based on the multiplication result.

[0071] The following describes the process of obtaining the output results of the training network layer using a specific example.

[0072] Assume that the hybrid expert system layer containing N expert modules can be expressed as , then the forward propagation process of the network layer to be trained can be expressed as follows: ; in, and They represent the first The output of the expert module and the The routing decision value corresponding to the expert module is It is the matrix formed by the parameters of the model to be trained in the network layer to be trained, that is, the training parameter matrix; is the input information of the network layer to be trained. It is the matrix formed by the frozen model parameters in the network layer to be trained, that is, the non-trainable parameter matrix.

[0073] By updating the forward propagation calculation of the network layer to be trained through the improved AdaLora module and the hybrid expert system, experts and external experts can work together to enable experts to effectively handle various types of tasks. For example, the output result of the network layer to be trained can be expressed as follows: ; in, is the output result of the network layer to be trained; is the number of expert modules in the hybrid expert system; is the regularization parameter; α is the scaling factor, which is a hyperparameter approximately equal to the learning rate. Indicates The routing decision value (or weight value) of an expert module, and They are the parameters of the model to be trained for the network layer to be trained in the The corresponding target low-rank matrix under each expert module; is the input result of the network layer to be trained; The model parameters frozen in the network layer to be trained.

[0074] The method provided in the embodiment of the present application, by introducing a hybrid expert system and an improved AdaLora module, utilizes the target low-rank matrix and the routing decision value to collaboratively update the parameters of the network layer to be trained, so as to perform fine adjustments for different tasks and data types, thereby achieving efficient fine-tuning of the parameters, significantly enhancing the robustness of the model, and reducing interference between tasks, significantly improving the task processing performance, thereby effectively solving the problems of poor robustness and poor task processing performance faced by existing pre-trained models in the process of parameter fine-tuning.

[0075] In some embodiments, the equalization loss value corresponding to each expert module in the hybrid expert system in step 130 is obtained based on the following steps: Calculating the correlation coefficient between the routing decision value of each of the expert modules and each of the sample images; Obtaining the importance coefficient of the correlation coefficient according to the matching status between the data type of the data processed by each of the expert modules and the data type of each of the sample images; The correlation coefficient and the importance coefficient are correspondingly multiplied to obtain the balanced loss value corresponding to each of the expert modules.

[0076] Optionally, in order to further avoid the problem of inter-task interference during the fine-tuning process, the embodiment of the present application adopts a soft constraint strategy to enable all expert modules to be softly focused on a single category, thereby combating the imbalance of utilization and balancing the utilization of all experts in the hybrid expert system framework, thereby effectively reducing the inter-task interference during the fine-tuning process and improving the task processing performance. The specific implementation steps are as follows: Calculate the correlation coefficient between the routing decision value of each expert module and each sample image; illustratively, the calculation formula of the correlation coefficient is represented as follows: ; in, is the first The routing decision value of the first expert module is The correlation coefficient (or sum of correlation) between sample images; For the The number of tokens (i.e. image blocks) in a sample image; It is The implicit input of tokens, where For the The token is in Feature vectors under expert modules; is the smoothing coefficient, which is used to control the smoothness of the softmax function.

[0077] The importance coefficient of the correlation coefficient is obtained according to the matching status between the data type of the data processed by each expert module and the data type of each sample image.

[0078] For example, if The data type processed by the first expert module is the same as that of the If the data types of the sample images are consistent, the importance coefficient of the correlation coefficient between the two is determined as the first coefficient value; if the The data type processed by the first expert module is the same as that of the If the data types of the sample images are inconsistent, the importance coefficient of the correlation coefficient between the two is determined to be the second coefficient value, wherein the first coefficient value is obtained by adding the imbalance coefficient between the expert modules and 1, and the second coefficient value is obtained by subtracting the imbalance coefficient between the expert modules and 1. Exemplarily, the calculation formula of the importance coefficient is as follows: ; in, For the The routing decision value of the first expert module is The importance coefficient of the correlation coefficient between sample images; Indicates the first The data type processed by each expert module; For the The data type of the sample images; is the imbalance coefficient, which is used to control the imbalance degree between expert modules.

[0079] Furthermore, the routing decision value of each expert module is multiplied with the correlation coefficient between each sample image and the corresponding importance coefficient to obtain the balanced loss value corresponding to each expert module.

[0080] Furthermore, the following steps are performed to obtain the target loss value in step 130: Calculate the variance and mean of the balanced loss values ​​corresponding to all the expert modules; calculating a ratio between the variance and the mean; The ratio and the segmentation loss value are weightedly added to obtain the target loss value.

[0081] Optionally, after obtaining the balanced loss value corresponding to each expert module, the variance and mean of the balanced loss values ​​corresponding to all expert modules may be calculated, and the ratio between the variance and mean of the balanced loss values ​​corresponding to all expert modules may be calculated; illustratively, the calculation formula of the ratio is as follows: ; in, It is the ratio between the variance and mean of the balanced loss values ​​corresponding to all expert modules; A correlation coefficient matrix formed by the correlation coefficients corresponding to all expert modules; The importance coefficient matrix is ​​formed by the importance coefficients of the correlation coefficients corresponding to all expert modules. The balanced loss value matrix is ​​obtained by element-by-element multiplication of the correlation coefficient matrix and the importance coefficient matrix, and the balanced loss value matrix contains the balanced loss values ​​corresponding to all expert modules; is the variance function; is the mean function.

[0082] It should be noted that, in certain cases, some samples from data sets related to a specified category often appear in the importance coefficient matrix The enhanced weight of the expert module responsible for processing this category is reflected in , which enables these expert modules to extract richer information from these samples, thus prompting the router to assign higher routing decision values ​​to them. This strategy not only ensures a certain degree of balance between expert modules that handle similar tasks, but also implements constraints in a flexible way to promote collaboration between experts to maintain their generalization ability.

[0083] Then, the ratio and the segmentation loss value are weighted and added to get the target loss value. The specific calculation formula is: ; in, is the target loss value; is the segmentation loss value; is the ratio; is the weight coefficient, which is used to adjust the intensity of the equalization loss.

[0084] In summary, the embodiment of the present application improves the existing AdaLoRA module by introducing local smoothing optimization, so that the improved AdaLoRA module has stronger robust performance, and integrates the improved AdaLora module into the hybrid expert system as an expert module, so that only some parameters need to be adjusted during the model fine-tuning process to achieve efficient parameter fine-tuning training. At the same time, all expert modules are balanced, and the router network automatically assigns weights to experts, which can effectively improve the task processing performance of the model in multiple professional field tasks, thereby improving the robustness and task processing performance of efficient parameter fine-tuning in semi-fine-tuning of multi-category field data.

[0085] The following simulation of the efficient parameter fine-tuning method based on the hybrid expert system provided in the embodiment of the present application is carried out using a specific image segmentation data set, and is compared with existing models and methods to verify the effectiveness of the method provided in the embodiment of the present application.

[0086] Figure 4 is a simulation comparison diagram of the efficient parameter fine-tuning method based on the hybrid expert system provided by the present invention, such as Figure 4 As shown, by inputting sample images of different objects in different environmental scenes as models into the method provided in the embodiment of the present application (or AdaLoRA+, that is, an adaptive low-rank adaptation model that introduces a hybrid expert system), as well as existing models and methods, such as the Ground Truth (GT), the original Low-Rank Adaptation (LoRA) module, and the original AdaLoRA module (that is, an adaptive low-rank adaptation model that does not introduce a hybrid expert system), the image segmentation results output by different models and methods are obtained. According to the comparison of the image segmentation results output by different models and methods, compared with the existing models and methods, the method provided in the embodiment of the present application has a stronger robustness performance characteristic, which effectively verifies the effectiveness of the method provided in the embodiment of the present application in realizing efficient parameter fine-tuning.

[0087] In some embodiments, the present application also provides an image segmentation method. Figure 5 It is a flow chart of the image segmentation method provided by the present invention; Figure 5 As shown, the method includes: step 510 and step 520.

[0088] Step 510: Obtain an image to be segmented.

[0089] Here, the image to be segmented is an image required for image segmentation, and the image may be captured in real time by an image collector, loaded from an image database, or uploaded by a user, which is not specifically limited in this embodiment.

[0090] Step 520, input the image to be segmented into an image segmentation model to obtain an image segmentation result of the image to be segmented; wherein the image segmentation model is trained based on the efficient parameter fine-tuning method based on the hybrid expert system provided in the above embodiments.

[0091] Optionally, after obtaining the image to be segmented, an image segmentation model may be loaded. The image segmentation model is trained based on the efficient parameter fine-tuning method based on the hybrid expert system provided in the above embodiments. For specific training steps, see Figure 1 , I will not go into details here.

[0092] In addition, after obtaining the image to be segmented, in order to improve the image quality, the image to be segmented can be preprocessed, such as image scaling, normalization, denoising, etc., to ensure that the image quality meets the requirements of the model input, thereby improving the image segmentation performance.

[0093] Subsequently, the preprocessed image to be segmented is used as model input and input into the image segmentation model, so that the image segmentation model uses its internal feature embedding layer, multi-layer feature extraction layer and mask decoder layer to perform embedded feature extraction, deep feature extraction and segmentation mask processing on the preprocessed image to be segmented in sequence, thereby obtaining the image segmentation result of the image to be segmented.

[0094] The method provided in this embodiment smoothly optimizes and obtains the target low-rank matrix corresponding to the parameters of the model to be trained of the network layer to be trained based on the improved sensitivity constraint function and low-rank matrix constraint function in the adaptive low-rank adaptation module, so as to obtain stronger robustness in the fine-tuning training process, and collaboratively updates the model parameters of the network layer to be trained according to the target low-rank matrix and the routing decision value of the hybrid expert system. In the process of updating the model parameters, the segmentation loss value between the image segmentation result corresponding to the output result of the network layer to be trained and the image segmentation label of the sample image, as well as the equalization loss value corresponding to each expert module in the hybrid expert system are introduced to fine-tune the parameters of the model to be trained, so as to achieve equalization processing of all expert modules while ensuring the image segmentation accuracy, thereby improving the processing performance of the trained image segmentation model on tasks in multiple fields, effectively solving the problems of limited data quality and interference between tasks in specific fields, significantly improving the robustness and task processing performance of the model in complex scenarios, and realizing the optimization of image segmentation models with efficient parameter fine-tuning training.

[0095] The following is a description of the efficient parameter fine-tuning device based on the hybrid expert system provided by the present invention. The efficient parameter fine-tuning device based on the hybrid expert system described below and the efficient parameter fine-tuning method based on the hybrid expert system described above can correspond to each other.

[0096] Figure 6Schematic diagram of the structure of the parameter efficient fine-tuning device based on the hybrid expert system provided by the present invention; Figure 6 As shown, the device includes: a first acquisition module 610, a second acquisition module 620, a third acquisition module 630 and a training module 640.

[0097] The first acquisition module 610 is used to obtain the target low-rank matrix corresponding to the to-be-trained model parameters of the to-be-trained network layer in the pre-trained segmentation model based on the improved sensitivity constraint function and the low-rank matrix constraint function in the adaptive low-rank adaptation module; the improved sensitivity constraint function is obtained by performing local smoothing optimization on the original sensitivity constraint function; The second acquisition module 620 is used to obtain the output result of the network layer to be trained according to the target low-rank matrix, the routing decision value of the hybrid expert system and the sample image; The third acquisition module 630 is used to obtain the target loss value of the pre-trained segmentation model according to the segmentation loss value between the image segmentation result corresponding to the output result of the network layer to be trained and the image segmentation label of the sample image, and the balanced loss value corresponding to each expert module in the hybrid expert system; The training module 640 is used to iteratively and efficiently fine-tune the parameters of the trained model of the network layer to be trained according to the target loss value to obtain an image segmentation model; the image segmentation model is used to obtain an image segmentation result of the image to be segmented according to the image to be segmented.

[0098] The device provided in the embodiment of the present application smoothly optimizes and obtains the target low-rank matrix corresponding to the model parameters to be trained of the network layer to be trained based on the improved sensitivity constraint function and low-rank matrix constraint function in the adaptive low-rank adaptation module, so as to obtain stronger robustness in the fine-tuning training process, and collaboratively updates the model parameters of the network layer to be trained according to the target low-rank matrix and the routing decision value of the hybrid expert system. In the process of updating the model parameters, the segmentation loss value between the image segmentation result corresponding to the output result of the network layer to be trained and the image segmentation label of the sample image, as well as the equalization loss value corresponding to each expert module in the hybrid expert system are introduced to fine-tune the parameters of the model to be trained, so as to achieve equalization processing of all expert modules while ensuring the image segmentation accuracy, thereby improving the processing performance of the trained image segmentation model on tasks in multiple fields, effectively solving the problems of limited data quality and interference between tasks in specific fields, significantly improving the robustness and task processing performance of the model in complex scenarios, and realizing the optimization of image segmentation models with efficient parameter fine-tuning training.

[0099] The device provided by the present invention is used to execute the above-mentioned method embodiments. Please refer to the above-mentioned embodiments for the specific processes and detailed contents, which will not be repeated here.

[0100] In some embodiments, the present application also provides an image segmentation device. Figure 7 Schematic diagram of the structure of the image segmentation device provided by the present invention; Figure 7 As shown, the device comprises: The fourth acquisition module 710 is used to acquire the image to be segmented; The segmentation processing module 720 is used to input the image to be segmented into an image segmentation model to obtain an image segmentation result of the image to be segmented; The image segmentation model is trained based on the efficient parameter fine-tuning method based on the hybrid expert system provided in the above embodiments.

[0101] The device provided in the embodiment of the present application smoothly optimizes and obtains the target low-rank matrix corresponding to the model parameters to be trained of the network layer to be trained based on the improved sensitivity constraint function and low-rank matrix constraint function in the adaptive low-rank adaptation module, so as to obtain stronger robustness in the fine-tuning training process, and collaboratively updates the model parameters of the network layer to be trained according to the target low-rank matrix and the routing decision value of the hybrid expert system. In the process of updating the model parameters, the segmentation loss value between the image segmentation result corresponding to the output result of the network layer to be trained and the image segmentation label of the sample image, as well as the equalization loss value corresponding to each expert module in the hybrid expert system are introduced to fine-tune the parameters of the model to be trained, so as to achieve equalization processing of all expert modules while ensuring the image segmentation accuracy, thereby improving the processing performance of the trained image segmentation model on tasks in multiple fields, effectively solving the problems of limited data quality and interference between tasks in specific fields, significantly improving the robustness and task processing performance of the model in complex scenarios, and realizing the optimization of image segmentation models with efficient parameter fine-tuning training.

[0102] Figure 8 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 8As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the efficient parameter fine-tuning method based on the hybrid expert system, the method comprising: based on the improved sensitivity constraint function and the low-rank matrix constraint function in the adaptive low-rank adaptation module, obtaining the target low-rank matrix corresponding to the to-be-trained model parameters of the to-be-trained network layer in the pre-trained segmentation model; the improved sensitivity constraint function is obtained by performing local smoothing optimization on the original sensitivity constraint function; according to the target low-rank matrix, the routing decision value of the hybrid expert system and the sample image, obtaining the output result of the to-be-trained network layer; according to the image segmentation result corresponding to the output result of the to-be-trained network layer and the image segmentation label of the sample image; The pre-trained segmentation model is obtained by using a segmentation loss value and a balanced loss value corresponding to each expert module in the hybrid expert system; the target loss value of the pre-trained segmentation model is obtained; according to the target loss value, the parameters of the to-be-trained model of the to-be-trained network layer are iteratively and efficiently fine-tuned to obtain an image segmentation model; the image segmentation model is used to obtain an image segmentation result of the to-be-segmented image according to the segmentation of the to-be-segmented image, or to execute an image segmentation method, the method comprising: obtaining an image to be segmented; inputting the image to be segmented into the image segmentation model to obtain an image segmentation result of the to-be-segmented image; wherein the image segmentation model is trained based on the parameter efficient fine-tuning method based on the hybrid expert system as provided in the above embodiments.

[0103] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0104] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the efficient parameter fine-tuning method based on the hybrid expert system provided by the above methods, and the method includes: based on the improved sensitivity constraint function and the low-rank matrix constraint function in the adaptive low-rank adaptation module, obtaining the target low-rank matrix corresponding to the training model parameters of the network layer to be trained in the pre-trained segmentation model; the improved sensitivity constraint function is obtained by locally smoothing and optimizing the original sensitivity constraint function; according to the target low-rank matrix, the routing decision value of the hybrid expert system and the sample image, obtaining the output result of the network layer to be trained; according to the network layer to be trained The target loss value of the pre-trained segmentation model is obtained by using the segmentation loss value between the image segmentation result corresponding to the output result of the network layer and the image segmentation label of the sample image, and the balanced loss value corresponding to each expert module in the hybrid expert system; according to the target loss value, the parameters of the to-be-trained model of the to-be-trained network layer are iteratively and efficiently fine-tuned to obtain the image segmentation model; the image segmentation model is used to obtain the image segmentation result of the to-be-segmented image according to the segmentation of the to-be-segmented image, or to execute the image segmentation method, which method includes: obtaining the to-be-segmented image; inputting the to-be-segmented image into the image segmentation model to obtain the image segmentation result of the to-be-segmented image; wherein, the image segmentation model is trained based on the efficient parameter fine-tuning method based on the hybrid expert system provided in the above embodiments.

[0105] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the efficient parameter fine-tuning method based on the hybrid expert system provided by the above-mentioned methods, the method comprising: obtaining a target low-rank matrix corresponding to the model parameters of the network layer to be trained in the pre-trained segmentation model based on the improved sensitivity constraint function and the low-rank matrix constraint function in the adaptive low-rank adaptation module; the improved sensitivity constraint function is obtained by performing local smoothing optimization on the original sensitivity constraint function; according to the target low-rank matrix, the routing decision value of the hybrid expert system and the sample image, obtaining the output result of the network layer to be trained; and according to the image segmentation result corresponding to the output result of the network layer to be trained The target loss value of the pre-trained segmentation model is obtained by combining the segmentation loss value between the image segmentation label of the sample image and the balanced loss value corresponding to each expert module in the hybrid expert system; the parameters of the to-be-trained model of the to-be-trained network layer are iteratively and efficiently fine-tuned according to the target loss value to obtain an image segmentation model; the image segmentation model is used to obtain an image segmentation result of the to-be-segmented image according to the segmentation of the to-be-segmented image, or to execute an image segmentation method, which method includes: obtaining an image to be segmented; inputting the image to be segmented into the image segmentation model to obtain an image segmentation result of the to-be-segmented image; wherein the image segmentation model is trained based on the efficient parameter fine-tuning method based on the hybrid expert system provided in the above embodiments.

[0106] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0107] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0108] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An efficient parameter fine-tuning method based on a hybrid expert system, characterized in that: include: Based on the improved sensitivity constraint function and low-rank matrix constraint function in the adaptive low-rank adaptation module, a target low-rank matrix corresponding to the to-be-trained model parameters of the to-be-trained network layer in the pre-trained segmentation model is obtained; the improved sensitivity constraint function is obtained by performing local smoothing optimization on the original sensitivity constraint function; Obtaining an output result of the network layer to be trained according to the target low-rank matrix, the routing decision value of the hybrid expert system and the sample image; Obtaining a target loss value of the pre-trained segmentation model according to a segmentation loss value between an image segmentation result corresponding to an output result of the network layer to be trained and an image segmentation label of the sample image, and a balanced loss value corresponding to each expert module in the hybrid expert system; According to the target loss value, the parameters of the to-be-trained model of the to-be-trained network layer are iteratively fine-tuned to obtain an image segmentation model; the image segmentation model is used to obtain an image segmentation result of the to-be-segmented image according to the segmentation of the to-be-segmented image.

2. The method for efficient parameter fine-tuning based on a hybrid expert system according to claim 1, characterized in that: The method of obtaining the target low-rank matrix corresponding to the to-be-trained model parameters of the to-be-trained network layer in the pre-trained segmentation model based on the improved sensitivity constraint function and the low-rank matrix constraint function in the adaptive low-rank adaptation module includes: Calculating the importance of the triples corresponding to the parameters of the model to be trained of the network layer to be trained according to the improved sensitivity constraint function; According to the importance, obtaining the rank dimension of the target low-rank matrix; According to the rank dimension of the target low-rank matrix and the low-rank matrix constraint function, obtaining the target low-rank matrix; The low-rank matrix constraint function is used to constrain the orthogonality of the target low-rank matrix.

3. The method for efficient parameter fine-tuning based on a hybrid expert system according to claim 2, characterized in that: The step of calculating the importance of the triplet corresponding to the parameters of the model to be trained of the network layer to be trained according to the improved sensitivity constraint function includes: Obtaining local sensitivity smoothed values ​​and local uncertainty quantized smoothed values ​​of the to-be-trained model parameters of the to-be-trained network layer at multiple historical moments before the current moment; Based on the improved sensitivity constraint function, according to the sensitivity value of the to-be-trained model parameter of the to-be-trained network layer at the current moment, local smoothing processing is performed on the local sensitivity smoothing values ​​of the multiple historical moments to obtain the local sensitivity smoothing value of the to-be-trained model parameter of the to-be-trained network layer at the current moment; Determine the sensitivity change of the to-be-trained model parameter of the to-be-trained network layer at the current moment according to the difference between the sensitivity value at the current moment and the local sensitivity smoothing value at the current moment; According to the sensitivity change amount, local smoothing processing is performed on the local uncertainty quantization smoothing values ​​of the multiple historical moments to obtain the local uncertainty quantization smoothing value of the to-be-trained model parameter of the to-be-trained network layer at the current moment; According to the local uncertainty quantization smoothed value at the current moment and the local sensitivity smoothed value at the current moment, the importance of the triplet corresponding to the to-be-trained model parameters of the to-be-trained network layer is obtained.

4. The method for efficient parameter fine-tuning based on a hybrid expert system according to any one of claims 1 to 3, characterized in that: The step of obtaining the output result of the network layer to be trained according to the target low-rank matrix, the routing decision value of the hybrid expert system and the sample image includes: According to the sample image, obtaining input information of the network layer to be trained; multiplying the target low-rank matrix, the routing decision value, and the input information; According to the multiplication result, the output result of the network layer to be trained is obtained.

5. The method for efficient parameter fine-tuning based on a hybrid expert system according to any one of claims 1 to 3, characterized in that: The balanced loss value corresponding to each expert module in the hybrid expert system is obtained based on the following steps: Calculating the correlation coefficient between the routing decision value of each of the expert modules and each of the sample images; Obtaining the importance coefficient of the correlation coefficient according to the matching status between the data type of the data processed by each of the expert modules and the data type of each of the sample images; The correlation coefficient and the importance coefficient are correspondingly multiplied to obtain the balanced loss value corresponding to each of the expert modules.

6. The method for efficient parameter fine-tuning based on a hybrid expert system according to any one of claims 1 to 3, characterized in that: The step of obtaining the target loss value of the pre-trained segmentation model according to the segmentation loss value between the image segmentation result corresponding to the output result of the network layer to be trained and the image segmentation label of the sample image, and the balanced loss value corresponding to each expert module in the hybrid expert system, comprises: Calculate the variance and mean of the balanced loss values ​​corresponding to all the expert modules; calculating a ratio between the variance and the mean; The ratio and the segmentation loss value are weightedly added to obtain the target loss value.

7. The method for efficient parameter fine-tuning based on a hybrid expert system according to any one of claims 1 to 3, characterized in that: The pre-trained segmentation model is constructed based on a feature embedding layer, a multi-layer feature extraction layer and a mask decoder layer; The network layer to be trained is at least one network layer among the feature embedding layer, multiple feature extraction layers and the mask decoder layer.

8. An image segmentation method, characterized in that: include: Obtain the image to be segmented; Inputting the image to be segmented into an image segmentation model to obtain an image segmentation result of the image to be segmented; Wherein, the image segmentation model is trained based on the efficient parameter fine-tuning method based on the hybrid expert system as described in any one of claims 1 to 7.

9. A parameter efficient fine-tuning device based on a hybrid expert system, characterized in that: include: A first acquisition module is used to acquire a target low-rank matrix corresponding to the to-be-trained model parameters of the to-be-trained network layer in the pre-trained segmentation model based on the improved sensitivity constraint function and the low-rank matrix constraint function in the adaptive low-rank adaptation module; the improved sensitivity constraint function is obtained by performing local smoothing optimization on the original sensitivity constraint function; A second acquisition module is used to acquire the output result of the network layer to be trained according to the target low-rank matrix, the routing decision value of the hybrid expert system and the sample image; A third acquisition module is used to obtain a target loss value of the pre-trained segmentation model according to a segmentation loss value between an image segmentation result corresponding to an output result of the network layer to be trained and an image segmentation label of the sample image, and a balanced loss value corresponding to each expert module in the hybrid expert system; The training module is used to iteratively and efficiently fine-tune the parameters of the to-be-trained model of the to-be-trained network layer according to the target loss value to obtain an image segmentation model; the image segmentation model is used to obtain an image segmentation result of the to-be-segmented image according to the to-be-segmented image segmentation.

10. An image segmentation device, characterized in that: include: A fourth acquisition module, used to acquire the image to be segmented; A segmentation processing module, used for inputting the image to be segmented into an image segmentation model to obtain an image segmentation result of the image to be segmented; Wherein, the image segmentation model is trained based on the efficient parameter fine-tuning method based on the hybrid expert system as described in any one of claims 1 to 7.

Images