Lightening method, device and equipment for large government affair model and storage medium

Abstract

The invention discloses a lightweight method, device and equipment for a large government affair model and a storage medium, and relates to the field of artificial intelligence model compression, and the method comprises the steps: determining metadata of the large government affair model, constructing a semantic incidence matrix based on a preset government affair field knowledge graph, and determining target information based on the metadata and the semantic incidence matrix; performing credibility evaluation on weight parameters of the government affair large model based on the target information, and determining a pruning strategy based on an obtained evaluation result; performing sensitivity analysis on the hierarchy of the government affair large model based on the target information and the weight parameter to obtain an analysis result, determining a quantization interval of the weight parameter, and determining a quantization strategy by using the analysis result and the quantization interval; and lightening the government affair large model by using the pruning strategy and the quantification strategy to obtain a corresponding target lightweight model. Therefore, the lightweight effect of the large government affair model can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence model compression, in particular to a lightweight method, device and equipment for government affairs large model and a storage medium. BACKGROUND

[0002] With the in-depth application of artificial intelligence in the government field, the demand for lightweight deployment of large models is increasingly urgent. Its core technology focuses on model compression and dynamic adaptability optimization. The mainstream methods include pruning, quantization and knowledge distillation. However, these existing technologies still have many obvious shortcomings.

[0003] In terms of specific technical implementation, traditional pruning adopts a static strategy, which cannot flexibly cope with dynamic changes in the government scenario, resulting in a large fluctuation in model accuracy. The quantization technology is affected by the long-tail distribution characteristics of government data, which easily causes semantic distortion and destroys the logical coherence of data. In addition, although incremental learning and edge-cloud collaboration technology can dynamically adjust the model, it cannot run efficiently in actual government applications due to the contradiction between computing resources and energy efficiency. Moreover, existing lightweight technologies generally lack multi-level trusted verification mechanisms, making it difficult to meet the strict requirements of safety and compliance in the government field.

[0004] Therefore, in the government scenario, how to improve the effect of lightweight large government models and effectively reduce the information loss in the quantization process is a technical problem that needs to be solved at present. SUMMARY

[0005] Therefore, the purpose of the present application is to provide a lightweight method, device and equipment for government large models and a storage medium, which can improve the effect of lightweight large government models and effectively reduce the information loss in the quantization process. The specific scheme is as follows:

[0006] In a first aspect, the present application provides a lightweight method for government large models, comprising:

[0007] determining the metadata of the government large model, and constructing a semantic association matrix based on a preset government domain knowledge graph, to determine target information based on the metadata and the semantic association matrix;

[0008] conducting a trustworthiness evaluation on the weight parameters of the government large model based on the target information, and determining a pruning strategy based on the obtained evaluation results; wherein the trustworthiness is the importance of the weight parameters in the government task scenario;

[0009] conducting a sensitivity analysis on the levels of the government large model based on the target information and the weight parameters, obtaining an analysis result, and determining a quantization interval of the weight parameters, and determining a quantization strategy using the analysis result and the quantization interval; wherein the sensitivity is the contribution degree of each level of the government large model to the processing of government information;

[0010] Lighten the government big model by using the pruning strategy and the quantization strategy to obtain a corresponding target lightweight model.

[0011] Optionally, the metadata of the government big model is determined, and a semantic correlation matrix is constructed based on a preset government domain knowledge graph, which includes:

[0012] The distribution of each weight parameter of the government big model is determined, and the hierarchical features and inter-layer connection topology of each level of the government big model are determined; wherein the inter-layer connection topology is a data structure representing the connection relationship between layers in the government big model;

[0013] The metadata of the government big model is determined based on the distribution, the hierarchical features and the inter-layer connection topology;

[0014] A target policy and a target government case are determined, and a semantic correlation matrix is constructed based on the target policy, the target government case and a preset government domain knowledge graph through a graph attention network.

[0015] Optionally, the credibility of the weight parameters of the government big model is evaluated based on the target information, which includes:

[0016] Each weight parameter of the government big model is determined;

[0017] The gradient is determined based on a preset loss function and the weight parameter, the absolute value of the gradient is determined, the maximum absolute value is determined from the absolute value, the gradient sensitivity of the weight parameter is determined based on each absolute value and the maximum absolute value;

[0018] The mean and standard deviation of the weight parameter are determined, and the weight significance of the weight parameter is determined based on the mean and the standard deviation;

[0019] A preset perturbation information is applied to the weight parameter to obtain a perturbed parameter, a first model output result and a second model output result are determined based on the weight parameter and the perturbed parameter through the government big model respectively, the difference information is determined based on the first model output result and the second model output result, and the output stability of the weight parameter is determined using the difference information and the preset perturbation information;

[0020] The credibility of each weight parameter of the government big model is evaluated based on the gradient sensitivity, the weight significance, the output stability and the target information to obtain a corresponding evaluation result.

[0021] Optionally, the pruning strategy is determined based on the obtained evaluation result, which includes:

[0022] determine a credibility ranking of each of the weight parameters based on the obtained evaluation result, and determine a first target weight parameter that satisfies a preset low credibility condition in terms of credibility based on the credibility ranking;

[0023] determine an entropy value of input data of the government affair task, and determine a complexity of the government affair task based on the entropy value, and determine a pruning ratio based on the complexity and a hardware resource occupancy rate;

[0024] determine a corresponding pruning strategy based on the first target weight parameter, a preset sparsification rate, and the pruning ratio.

[0025] Optionally, the sensitivity of the level of the government affair large model is analyzed based on the target information and the weight parameter, an analysis result is obtained, a quantization interval of the weight parameter is determined, a quantization strategy is determined based on the analysis result and the quantization interval, and the quantization strategy comprises:

[0026] The sensitivity of the level of the government affair large model is analyzed based on the target information, the weight parameter, the level of the government affair large model, and a preset Gaussian disturbance, and a corresponding analysis result is obtained;

[0027] The quantization interval of the weight parameter is determined by using a preset quantile statistical method;

[0028] A first level with a sensitivity satisfying a preset high sensitivity condition and a second level with a sensitivity satisfying a preset low sensitivity condition are determined from the analysis result;

[0029] A mixed precision quantization strategy is adopted for the first level and a preset bit quantization strategy is adopted for the second level based on the quantization interval to determine a corresponding quantization strategy; the mixed precision quantization strategy is a quantization strategy comprising a preset precision fixed point number and a preset precision floating point number.

[0030] Optionally, the government affair large model is lightened by using the pruning strategy and the quantization strategy to obtain a corresponding target lightened model, which comprises:

[0031] The government affair large model is pruned by using the pruning strategy to obtain a sparse model, and the sparse model is quantized by using the quantization strategy to obtain an initial lightened model;

[0032] The initial lightened model is iteratively optimized based on a preset government affair data set until the size of the lightened model in iteration is reduced to a preset model size or the accuracy of the government affair task is reduced to a preset accuracy; the government affair task is a task of processing input government affair data by the lightened model in iteration;

[0033] The lightened model at the end of iteration is determined as the target lightened model.

[0034] Optionally, the pruning the government large model by using the pruning strategy further includes:

[0035] determining a second target weight parameter reserved in the government large model, and determining data loss in a pruning process based on the second target weight parameter;

[0036] determining a constraint condition in a data supplement process based on a proximal gradient method and a current weight parameter of the current government large model, and supplementing the data loss by using the constraint condition, the preset government domain knowledge graph, and a knowledge injection intensity coefficient.

[0037] In a second aspect, the present application provides a lightweight device of a government large model, comprising:

[0038] an information determination module configured to determine metadata of the government large model, construct a semantic correlation matrix based on a preset government domain knowledge graph, and determine target information based on the metadata and the semantic correlation matrix;

[0039] a pruning strategy determination module configured to perform credibility evaluation on weight parameters of the government large model based on the target information, and determine a pruning strategy based on an evaluation result obtained; wherein the credibility is the importance of the weight parameters in a government task scenario;

[0040] a quantization strategy determination module configured to perform sensitivity analysis on levels of the government large model based on the target information and the weight parameters, obtain an analysis result, determine a quantization interval of the weight parameters, and determine a quantization strategy by using the analysis result and the quantization interval; wherein the sensitivity is the contribution degree of each level of the government large model to government information processing;

[0041] a model lightweight module configured to perform lightweight on the government large model by using the pruning strategy and the quantization strategy, and obtain a corresponding target lightweight model.

[0042] In a third aspect, the present application provides an electronic device, comprising:

[0043] a memory configured to save a computer program;

[0044] a processor configured to execute the computer program to implement the lightweight method of the government large model described above.

[0045] In a fourth aspect, the present application provides a computer readable storage medium configured to save a computer program; wherein the computer program is executed by a processor to implement the lightweight method of the government large model described above.

[0046] In this application, the metadata of the government affairs model is determined, and a semantic association matrix is ​​constructed based on a preset government affairs domain knowledge graph to determine the target information based on the metadata and the semantic association matrix; the weight parameters of the government affairs model are evaluated for credibility based on the target information, and a pruning strategy is determined based on the obtained evaluation results; wherein, credibility is the importance of the weight parameters in the government affairs task scenario; based on the target information and the weight parameters, a sensitivity analysis is performed on the levels of the government affairs model to obtain analysis results, and the quantization interval of the weight parameters is determined, and the quantization strategy is determined using the analysis results and the quantization interval; wherein, sensitivity is the contribution of each level of the government affairs model to government affairs information processing; the government affairs model is lightweighted using the pruning strategy and the quantization strategy to obtain the corresponding target lightweight model. As can be seen from the above, this application first clarifies the metadata of the government affairs model, constructs a semantic association matrix based on a preset government affairs domain knowledge graph, and then determines the target information through the metadata and the semantic association matrix. Then, a credibility evaluation is performed on the weight parameters of the government affairs model based on the target information, and a pruning strategy is determined based on the evaluation results. Then, with the help of target information and weight parameters, a sensitivity analysis is performed on the hierarchy of the government affairs model to obtain the analysis results, and the quantization interval of the weight parameters is determined at the same time. The quantization strategy is then determined using the analysis results and quantization interval. Finally, the government affairs model is lightweighted using the pruning strategy and the quantization strategy to obtain the corresponding target lightweight model. In this way, this application can improve the lightweight effect of the government affairs model and effectively reduce the information loss in the quantization process, thereby improving the application efficiency of the government affairs model in actual government affairs scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0048] Figure 1 A flowchart of a lightweight method for a large government affairs model disclosed in this application;

[0049] Figure 2 This is a schematic diagram of the lightweight device structure of a large government model disclosed in this application;

[0050] Figure 3 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0051] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described, obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0052] The conventional pruning adopts a static strategy, which cannot flexibly cope with dynamic changes in the government affair scene, resulting in a large fluctuation in model accuracy; the quantization technology is easily affected by the long-tail distribution characteristics of government affair data, which easily causes semantic distortion and destroys the logical coherence of data. In addition, although the incremental learning and edge-cloud collaboration technology can dynamically adjust the model, in actual government affair applications, it is limited by the contradiction between computing power resources and energy efficiency, and cannot be efficiently operated. Moreover, the existing lightweight technology generally lacks a multi-level trusted verification mechanism, and it is difficult to meet the strict requirements of safety and compliance in the government affair field. Therefore, the present application provides a lightweight method, device and equipment for a government affair large model, which can improve the effect of lightweight of the government affair large model and effectively reduce the information loss in the quantization process.

[0053] Referring to Figure 1 The embodiments of the present application disclose a lightweight method for a government affair large model, comprising:

[0054] Step S11, determining the metadata of the government affair large model, and constructing a semantic association matrix based on a preset government affair domain knowledge graph, to determine target information based on the metadata and the semantic association matrix.

[0055] In this embodiment, specifically, first, the basic parameter information of the government affair large model needs to be obtained, the distribution of each weight parameter is determined by analyzing the model architecture, for example, the sparsity of convolution layer weights, the numerical range of fully connected layer parameters, etc., and the hierarchical features of each level of the model are extracted, including the number of neurons, the type of activation function, the size of feature maps, and other attributes of each layer, as well as the inter-layer connection topology structure, which represents the input-output connection relationship between layers in the form of a directed graph, such as the full connection relationship between adjacent layers in a feedforward neural network, the hierarchical connection relationship between convolution layers and pooling layers in a convolutional neural network, etc.

[0056] Based on the above-mentioned distribution, hierarchical features and inter-layer connection topology, the metadata set of the government affair large model is determined, which can completely represent the basic architecture features of the government affair large model.

[0057] Further, target policy and target government case data are determined, wherein the target policy can select the latest regulatory provisions in a specific field, and the target government case is a representative historical processing instance in the field. The target policy and the target government case data are processed by a graph attention network, and a preset government domain knowledge graph is used as a basis. The knowledge graph can include government entities and semantic relationships between entities. By using a graph attention mechanism, the correlation weight of each semantic unit in the target policy and the target government case in the knowledge graph is calculated, and a semantic correlation matrix reflecting the correlation strength between the text semantic units and the knowledge graph entities is constructed. Thus, the target information is determined based on the metadata and the semantic correlation matrix.

[0058] In step S12, the weight parameters of the government large model are evaluated based on the target information, and a pruning strategy is determined based on the obtained evaluation results. The credibility is the importance of the weight parameters in the government task scenario.

[0059] In this embodiment, the full weight parameters of the government large model are first obtained. For each weight parameter, a credibility evaluation operation is performed from multiple dimensions. First, forward propagation and backward propagation calculations are performed based on a preset loss function and the weight parameter to obtain corresponding gradient values. The absolute values of the gradients are taken and the maximum absolute value is selected. The gradient sensitivity index of the weight parameter is determined by the ratio of each absolute value to the maximum absolute value. The index reflects the response degree of the weight parameter to the change of the loss function in the model training process. The higher the gradient sensitivity, the more significant the influence of the weight parameter on the model optimization.

[0060] Secondly, the mean and standard deviation of the weight parameter are calculated to construct a weight significance index. The mean reflects the overall level of the weight parameter, and the standard deviation reflects the dispersion degree of the parameter value. When the mean of the weight parameter approaches zero and the standard deviation is small, it means that the weight parameter may be in a secondary position in the model and has a relatively low contribution to the expression ability of the model. On the contrary, the weight parameter with a larger mean deviation from zero or a larger standard deviation often plays a more important role in the model.

[0061] Thirdly, a preset disturbance information is applied to the weight parameter to obtain a disturbed parameter. The original weight parameter and the disturbed parameter are input into the government large model to obtain first and second model output results. By calculating the difference information of the two, and combining the strength of the preset disturbance information, the output stability index of the weight parameter is determined. The weight parameter with high output stability has a small change in model output when subjected to a small disturbance, indicating that it has high robustness and importance in the model. On the contrary, the parameter with low output stability may have a relatively small impact on the model performance.

[0062] After obtaining the gradient sensitivity, weight significance, and output stability, the credibility of each weight parameter of the government affair large model is evaluated based on the gradient sensitivity, weight significance, output stability, and target information to obtain a corresponding evaluation result.

[0063] Based on the evaluation result, the credibility of each weight parameter is sorted. By setting a preset low credibility condition, first target weight parameters that meet the preset low credibility condition are screened out, which are the initially determined pruning candidate parameters.

[0064] Next, the characteristics of the government affair task are analyzed to determine a reasonable pruning strategy. The entropy value of the government affair task input data is calculated. The greater the entropy value, the higher the uncertainty and complexity of the input data, and the more difficult the corresponding government affair task. Based on the entropy value, the complexity level of the government affair task is determined, and the pruning proportion is comprehensively determined in combination with the resource occupancy rate of the hardware device. For complex government affair tasks and sufficient hardware resources, a smaller pruning proportion can be used to retain more model capabilities; for simple government affair tasks or hardware resource limited scenarios, the pruning proportion is appropriately increased to achieve more efficient lightweight under the premise of ensuring model performance.

[0065] Finally, the specific pruning strategy is determined in combination with the first target weight parameter, the preset sparsification rate, and the pruning proportion. For example, the first target weight parameter is pruned layer by layer according to the credibility sorting, the pruning proportion of each layer is controlled to be not more than the threshold value determined according to the task complexity and hardware resources, and the pruning process is gradually promoted in accordance with the preset sparsification rate, avoiding excessive pruning at one time and causing a sharp drop in model performance. In this way, the government affair large model weight parameters are accurately selected and reasonably pruned, effectively reducing the model complexity and resource occupancy under the premise of ensuring the performance of the model in the government affair task scenario.

[0066] In step S13, based on the target information and the weight parameter, a sensitivity analysis is performed on the levels of the government affair large model to obtain an analysis result, and a quantization interval of the weight parameter is determined, and a quantization strategy is determined using the analysis result and the quantization interval. The sensitivity is the contribution of each level of the government affair large model to the processing of government information.

[0067] In this embodiment, the semantic association matrix in the target information is combined with the weight parameter, and a systematic disturbance experiment is performed on each level of the government affair large model by a preset Gaussian disturbance. For each level, random noise conforming to a Gaussian distribution is added to the input data, and by comparing the change degree of the model output result before and after the disturbance, the contribution of the level to the processing of government information is quantified, thereby obtaining the corresponding analysis result about the sensitivity.

[0068] In determining the quantization interval of the weight parameter, a preset quantile statistical method is adopted. Statistical analysis is performed on all weight parameters, the probability distribution function is calculated, and then the parameter value corresponding to a specific quantile is selected as the upper and lower bounds of the quantization interval. This method can effectively exclude the interference of abnormal values and ensure that the quantization interval can cover most of the effective parameter information.

[0069] Based on the analysis results of sensitivity, hierarchical classification is performed. The preset high sensitivity condition and the preset low sensitivity condition are set to divide the hierarchical level of the government large model into the first level and the second level. The first level, or the high sensitivity level, usually corresponds to the layers in the government large model that play an important role in semantic understanding and key feature extraction of government information, such as attention mechanism layer, high-level semantic feature fusion layer, etc. The low sensitivity level includes layers that perform preliminary processing on input data, such as early convolutional layers or simple feature extraction layers.

[0070] According to the characteristics of different levels, a differentiated quantization strategy is formulated. For the first level, a mixed precision quantization strategy is adopted, combining preset precision fixed-point numbers and preset precision floating-point numbers for quantization. In this mixed precision quantization strategy, the weight parameters are divided into two parts: one part uses higher precision floating-point numbers, and the other part uses lower precision fixed-point numbers. This mixed approach effectively reduces storage and computing overhead while ensuring the accuracy of the government large model.

[0071] For the second level, a preset bit quantization strategy is adopted to uniformly quantize the weight parameters to fixed-bit fixed-point numbers. Since these levels have less impact on the overall performance of the model, reducing the quantization precision has limited impact on the government task processing effect, but can significantly reduce the model size and computational complexity.

[0072] Step S14, using the pruning strategy and the quantization strategy to perform lightweight processing on the government large model to obtain a corresponding target lightweight model.

[0073] In this embodiment, the government large model is subjected to a structured pruning operation according to the pruning strategy. Based on the determined first target weight parameter and pruning ratio, a progressive pruning method is adopted to filter the model weights layer by layer. During the pruning process, the basic architecture of the model is kept unchanged as much as possible, and only redundant connections are removed to form a sparse model. This sparse model maintains the integrity of the original model's functions while significantly reducing the number of parameters and computational complexity.

[0074] And, in the pruning process, the second target weight parameter retained in the government big model is analyzed, and the data loss is determined by comparing the output difference of the original model and the pruned model on the government task. To make up for this data loss, a data supplement mechanism based on the proximal gradient method can be used. Combined with the current weight parameter state of the government big model, the constraint condition in the data supplement process is determined to ensure that the supplement operation will not destroy the overall structure of the model and the existing knowledge expression. At the same time, the preset government domain knowledge graph is used to convert the entity relationship and semantic information contained in the graph into prior knowledge for data supplement. The proportion of supplemented information is controlled by the knowledge injection intensity coefficient, which neither over-reliance on external knowledge leads the model to deviate from the original training direction, nor effectively makes up for the information loss caused by pruning.

[0075] Next, the sparse model is quantized based on the determined quantization strategy. For the first level with high sensitivity, a mixed precision quantization strategy is used; for the second level with low sensitivity, a preset bit quantization strategy is used for quantization. Through this differentiated quantization method, the storage requirements and computational complexity of the government big model are further reduced under the premise of ensuring the government big model's ability to process government information, and an initial lightweight model is obtained.

[0076] Further, the initial lightweight model is iteratively optimized using a preset government data set. This data set contains typical cases and policy texts in various government scenarios, covering common government processing tasks. In each iteration, the government big model processes the input government data, calculates the task accuracy and model size. When the model size is reduced to the preset model size, or the accuracy of the government task is reduced to the preset accuracy, the iteration is stopped.

[0077] Finally, the lightweight model at the end of iteration is determined as the target lightweight model. This model significantly reduces the model size and computational complexity while ensuring the accuracy of government task processing, and can be better deployed on resource-constrained government terminal devices such as grassroots government service all-in-one machines, improving the response speed and convenience of government services, while maintaining the ability to understand and process complex government scenarios.

[0078] As can be seen from the above, this application first clarifies the metadata of the government affairs big model, constructs a semantic association matrix based on the preset government affairs field knowledge graph, and then determines the target information through the metadata and the semantic association matrix. Next, a credibility assessment is conducted on the weight parameters of the government affairs big model based on the target information, and a pruning strategy is determined based on the assessment results. Then, a sensitivity analysis is performed on the hierarchy of the government affairs big model with the help of the target information and weight parameters to obtain the analysis results, and the quantization interval of the weight parameters is determined at the same time, and the quantization strategy is determined using the analysis results and quantization interval. Finally, the pruning strategy and quantization strategy are used to lightweight the government affairs big model to obtain the corresponding target lightweight model. In this way, this application can improve the lightweight effect of the government affairs big model and effectively reduce the information loss in the quantization process, thereby improving the application efficiency of the government affairs big model in actual government affairs scenarios.

[0079] The technical solution of the embodiment of the present application is described in detail below in conjunction with the specific implementation process.

[0080] Specifically, first, for the original government affairs big model, the model structure metadata of the original government affairs big model is extracted, such as layer type, parameter distribution, connection topology, etc. The formula for extracting model structure features is: , where is the layer node feature, is the inter-layer connection topology, and Represents different layers of the government affairs model, Indicates the The mean of the layer weight parameters, Indicates the The variance of the layer weight parameters is then calculated. The government knowledge graph is then loaded to construct a semantic association matrix for credibility assessment. This is achieved by fusing the policy text embeddings and case embeddings determined from the government knowledge graph using a graph attention network (GAT) to obtain the corresponding semantic association matrix. Furthermore, the metadata and semantic association matrix are identified as target information.

[0081] Next, determine the pruning strategy. Specifically, the credibility of the weight parameters in the government affairs model needs to be evaluated based on the established parameter credibility evaluation mechanism. This parameter credibility evaluation mechanism includes multi-dimensional evaluation indicators, including gradient sensitivity, weight significance, and output stability. In addition, this parameter credibility evaluation mechanism can be expressed as the following formula:

[0082] ;

[0083] Where, are the weight parameters in the government affairs model, represents the gradient sensitivity of the weight parameter, is a loss function, is a weight significance of a weight parameter, is a standard deviation of a weight parameter, is a mean value of a weight parameter, is an output stability of a weight parameter, is a perturbation to a weight parameter, is a model output before the perturbation, is a model output after the perturbation, is data input into the government big model, , and are weight coefficients for controlling gradient sensitivity, weight significance and output stability, respectively, and .

[0084] According to the parameter credibility evaluation mechanism, a corresponding credibility score ranking is determined, and the target information and the credibility score ranking are used to gradually sparsify the low-credibility parameter (i.e., the first target weight parameter), and some redundant parameters are retained to deal with sudden government data fluctuations.

[0085] In addition, a task-aware threshold adjustment algorithm is used to determine the pruning ratio for different government sub-tasks. That is, the entropy value of the input data of the current government task can be determined, and the complexity of the current government task can be determined according to the entropy value. Moreover, the current hardware resource occupation ratio is determined, and the complexity of the current government task and the current hardware resource occupation ratio are used to determine the pruning ratio, so that the corresponding pruning strategy is obtained according to the pruning ratio, the first target weight parameter and the preset sparsification rate.

[0086] At the same time, the sensitivity of the levels of the government big model is analyzed. The level sensitivity analysis quantifies the sensitivity of each layer weight in the government big model to the output error through layer-by-layer perturbation experiments, and is divided into a high-sensitivity layer (i.e., a first level) and a low-sensitivity layer (i.e., a second level). Moreover, for the long-tail distribution characteristics of government data, the sliding quantile statistical method is used to determine the upper and lower bounds of the quantization interval. The high-sensitivity layer adopts mixed precision quantization (i.e., a mixed precision quantization strategy), and the low-sensitivity layer adopts extremely low bit quantization (i.e., a preset bit quantization strategy). Thus, the corresponding quantization strategy is determined according to the mixed precision quantization and the extremely low bit quantization. The mixed precision quantization can refer to 8-bit fixed-point numbers and 16-bit floating-point numbers, and the extremely low bit quantization can refer to 4-bit binary encoding.

[0087] After determining the pruning strategy and the quantization strategy, first, dynamic trusted pruning is performed to generate a sparse sub-model, then the sparse model is quantized in layers to generate a low-bit coding model (i.e., an initial lightweight model), and finally, the low-bit coding model is jointly fine-tuned based on the government verification set to update the corresponding pruning parameters and quantization parameters. The termination condition of iteration is that the size of the government large model is reduced to the target threshold (i.e., the preset model size) and the task accuracy decreases to the threshold (i.e., the preset accuracy). When the iteration ends, the current government large model is determined as the target lightweight model.

[0088] In addition, after pruning the government large model, the domain features are injected using the government domain knowledge graph to compensate for the information loss caused by pruning. That is, the weight parameters (i.e., the second target weight parameters) retained in the government large model are determined, and the data loss in the pruning process is determined based on the weight parameters retained in the model. Then, the constraint condition in the data supplement process is determined based on the proximal gradient method and the current weight parameters of the current government large model, and the data loss is supplemented accordingly using the constraint condition, the preset government domain knowledge graph, and the knowledge injection intensity coefficient.

[0089] Correspondingly, referring to Figure 2 The embodiment of the present application provides a lightweight device of a government large model, which comprises:

[0090] An information determination module 11 is configured to determine the metadata of the government large model, construct a semantic association matrix based on a preset government domain knowledge graph, and determine target information based on the metadata and the semantic association matrix.

[0091] A pruning strategy determination module 12 is configured to perform credibility evaluation on the weight parameters of the government large model based on the target information, and determine a pruning strategy based on the obtained evaluation result. The credibility is the importance of the weight parameters in the government task scenario.

[0092] A quantization strategy determination module 13 is configured to perform sensitivity analysis on the levels of the government large model based on the target information and the weight parameters, obtain an analysis result, determine a quantization interval of the weight parameters, and determine a quantization strategy using the analysis result and the quantization interval. The sensitivity is the contribution degree of each level of the government large model to the government information processing.

[0093] A model lightweight module 14 is configured to perform lightweight on the government large model using the pruning strategy and the quantization strategy, and obtain a corresponding target lightweight model.

[0094] From the above, the present application first determines the metadata of the government big model, constructs a semantic correlation matrix according to the preset government field knowledge graph, and then determines the target information through the metadata and the semantic correlation matrix. Next, the weight parameters of the government big model are evaluated based on the target information, and the pruning strategy is determined according to the evaluation result. Then, the sensitivity of the level of the government big model is analyzed with the help of the target information and the weight parameters, and the analysis result is obtained, and the quantization interval of the weight parameters is determined, and then the quantization strategy is determined by using the analysis result and the quantization interval. Finally, the pruning strategy and the quantization strategy are used to perform lightweight processing on the government big model, and the corresponding target lightweight model is obtained. In this way, the present application can improve the lightweight effect of the government big model and effectively reduce the information loss in the quantization process, thereby improving the application efficiency of the government big model in the actual government scenario.

[0095] In some embodiments, the information determination module 11 specifically includes:

[0096] A model information determination unit is configured to determine the distribution of each weight parameter of the government big model, and determine the level features of each level of the government big model and the inter-layer connection topology. The inter-layer connection topology is a data structure representing the connection relationship between layers in the government big model.

[0097] A metadata determination unit is configured to determine the metadata of the government big model based on the distribution, the level features, and the inter-layer connection topology.

[0098] A matrix construction unit is configured to determine a target policy and a target government case, and construct a semantic correlation matrix based on the target policy, the target government case, and a preset government field knowledge graph through a graph attention network.

[0099] In some embodiments, the pruning strategy determination module 12 specifically includes:

[0100] A first parameter determination unit is configured to determine each weight parameter of the government big model.

[0101] A sensitivity determination unit is configured to determine a gradient based on a preset loss function and the weight parameters, determine the absolute values of the gradient, determine the maximum absolute value from the absolute values, and determine the gradient sensitivity of the weight parameters based on each absolute value and the maximum absolute value.

[0102] A significance determination unit is configured to determine the mean and standard deviation of the weight parameters, and determine the weight significance of the weight parameters based on the mean and the standard deviation.

[0103] The stability determination unit is configured to apply preset disturbance information to the weight parameter to obtain a disturbed parameter, determine a first model output result and a second model output result based on the weight parameter and the disturbed parameter respectively by using the government affair large model, determine difference information based on the first model output result and the second model output result, and determine output stability of the weight parameter by using the difference information and the preset disturbance information.

[0104] The parameter evaluation unit is configured to perform credibility evaluation on each weight parameter of the government affair large model based on the gradient sensitivity, the weight significance, the output stability and the target information to obtain a corresponding evaluation result.

[0105] In some embodiments, the pruning strategy determination module 12 specifically includes:

[0106] The second parameter determination unit is configured to determine a credibility ranking of each weight parameter based on the obtained evaluation result, and determine a first target weight parameter that satisfies a preset low credibility condition based on the credibility ranking.

[0107] The proportion determination unit is configured to determine an entropy value of input data of a government affair task, determine complexity of the government affair task based on the entropy value, and determine a pruning proportion based on the complexity and a hardware resource occupancy rate.

[0108] The first strategy determination unit is configured to determine a corresponding pruning strategy based on the first target weight parameter, a preset sparsification rate and the pruning proportion.

[0109] In some embodiments, the quantization strategy determination module 13 specifically includes:

[0110] The sensitivity analysis unit is configured to perform sensitivity analysis on a level of the government affair large model based on the target information, the weight parameter, the level of the government affair large model and a preset Gaussian disturbance, and obtain a corresponding analysis result.

[0111] The interval determination unit is configured to determine a quantization interval of the weight parameter by using a preset quantile statistical method.

[0112] The level determination unit is configured to determine a first level that satisfies a preset high sensitivity condition and a second level that satisfies a preset low sensitivity condition from the analysis result.

[0113] The second strategy determination unit is configured to adopt a mixed precision quantization strategy for the first level and a preset bit quantization strategy for the second level based on the quantization interval to determine a corresponding quantization strategy; the mixed precision quantization strategy is a quantization strategy containing a preset precision fixed point number and a preset precision floating point number.

[0114] In some embodiments, the model lightening module 14 specifically includes:

[0115] a model lightening unit, configured to prune the government affair large model by using the pruning strategy to obtain a sparse model, and quantize the sparse model by using the quantization strategy to obtain an initial lightened model;

[0116] a model optimization unit, configured to iteratively optimize the initial lightened model based on a preset government affair dataset until a size of a lightened model in iteration is reduced to a preset model size or an accuracy of a government affair task is reduced to a preset accuracy, the government affair task being a task of processing input government affair data by the lightened model in iteration;

[0117] a model determination unit, configured to determine the lightened model at the end of iteration as a target lightened model.

[0118] In some embodiments, the model lightening unit specifically further includes:

[0119] a loss determination subunit, configured to determine a second target weight parameter reserved in the government affair large model, and determine a data loss in a pruning process based on the second target weight parameter;

[0120] a data supplement subunit, configured to determine a constraint condition in a data supplement process based on a proximal gradient method and a current weight parameter of the current government affair large model, and supplement the data loss correspondingly by using the constraint condition, the preset government affair domain knowledge graph and a knowledge injection intensity coefficient.

[0121] Further, the present application also discloses an electronic device, Figure 3 is a structural diagram of an electronic device 20 according to an exemplary embodiment, and the contents in the figure cannot be considered as any limitation on the use range of the present application. The electronic device 20 specifically can include at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25 and a communication bus 26. Wherein, the memory 22 is used to store a computer program, the computer program is loaded and executed by the processor 21 to realize the related steps in the government affair large model lightening method disclosed in any of the preceding embodiments. In addition, the electronic device 20 in the present embodiment can be specifically an electronic computer.

[0122] In this embodiment, the power supply 23 is configured to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 is configured to create a data transmission channel between the electronic device 20 and external devices, and the communication protocol followed by the communication interface 24 can be any communication protocol applicable to the technical solution of the present application, which will not be specifically limited herein; the input and output interface 25 is configured to obtain external input data or output data to the outside, and the specific interface type can be selected according to the specific application needs, which will not be specifically limited herein.

[0123] In addition, the memory 22 as a carrier for storing resources can be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc., and the resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage mode can be temporary storage or permanent storage.

[0124] The operating system 221 is configured to manage and control each hardware device on the electronic device 20 and the computer program 222, and can be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program capable of completing the lightweight method of the government big model executed by the electronic device 20 disclosed in any of the preceding embodiments, the computer program 222 can further include a computer program capable of completing other specific work.

[0125] Further, the present application also discloses a computer readable storage medium for storing a computer program; wherein the computer program is executed by a processor to implement the lightweight method of the government big model disclosed above. For the specific steps of the method, please refer to the corresponding content disclosed in the preceding embodiments, which will not be described here.

[0126] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts of each embodiment can be referred to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the related parts can be referred to the method part.

[0127] The skilled person can further realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized by electronic hardware, computer software or a combination of the two. In order to clearly show the interchangeability of hardware and software, the components and steps of each example have been described in the above description. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0128] The steps of a method or algorithm described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in RAM, flash memory, ROM, electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. The processor can be configured to execute the software module.

[0129] Finally, it should be noted that the terms "first", "second", and the like, herein do not denote any order, quantity, combination, or importance, but rather are used to distinguish one element from another, and are not necessarily intended to denote the temporal or chronological sequence of the execution of the respective steps. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without further constraints, exclude the presence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0130] The above detailed description of the technical solutions provided by the present application has been described in detail, and the principles and implementation modes of the present application have been described by applying specific examples; the above description of the embodiments is only for the purpose of helping to understand the method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range will be changed; in view of the above, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A lightweight method for a large government model, characterized by: include: Determine metadata of the government affairs model, and construct a semantic association matrix based on a preset government affairs domain knowledge graph, so as to determine target information based on the metadata and the semantic association matrix; Performing a credibility evaluation on the weight parameters of the government affairs model based on the target information, and determining a pruning strategy based on the obtained evaluation results; wherein the credibility is the importance of the weight parameters in the government affairs task scenario; Based on the target information and the weight parameters, a sensitivity analysis is performed on the layers of the government affairs model to obtain analysis results, and a quantization interval of the weight parameters is determined. A quantization strategy is determined using the analysis results and the quantization interval; wherein the sensitivity is the contribution of each layer of the government affairs model to government affairs information processing; The large government affairs model is lightweighted using the pruning strategy and the quantization strategy to obtain a corresponding target lightweight model.

2. The lightweight method for the government affairs large model according to claim 1 is characterized in that: Determining the metadata of the government affairs model and constructing a semantic association matrix based on a preset government affairs domain knowledge graph includes: Determining the distribution of weight parameters of the government affairs model, and determining the hierarchical characteristics and inter-layer connection topology of each layer of the government affairs model; wherein the inter-layer connection topology is a data structure representing the connection relationship between each layer in the government affairs model; Determining metadata of the government affairs big model based on the distribution situation, the hierarchical characteristics, and the inter-layer connection topology; Determine the target policy and target government affairs case, and construct a semantic association matrix based on the target policy, the target government affairs case and the preset government affairs field knowledge graph through the graph attention network.

3. The lightweight method for the government affairs large model according to claim 1 is characterized in that: The credibility evaluation of the weight parameters of the government affairs model based on the target information includes: Determining the weight parameters of the government affairs model; Determining a gradient based on a preset loss function and the weight parameter, determining an absolute value of the gradient, determining a maximum absolute value from the absolute values, and determining a gradient sensitivity of the weight parameter based on each of the absolute values ​​and the maximum absolute value; Determining a mean and a standard deviation of the weight parameter, and determining a weight significance of the weight parameter based on the mean and the standard deviation; Applying preset disturbance information to the weight parameters to obtain disturbed parameters, determining a first model output result and a second model output result based on the weight parameters and the disturbed parameters, respectively, by the government affairs big model, determining corresponding difference information based on the first model output result and the second model output result, and determining the output stability of the weight parameters using the difference information and the preset disturbance information; Based on the gradient sensitivity, the weight significance, the output stability and the target information, a credibility evaluation is performed on each weight parameter of the government affairs model to obtain corresponding evaluation results.

4. The lightweight method for the government affairs large model according to claim 1 is characterized in that: Determining a pruning strategy based on the obtained evaluation results includes: Determining a credibility ranking of each of the weight parameters based on the obtained evaluation results, and using the credibility ranking to determine a first target weight parameter whose credibility meets a preset low credibility condition; Determining an entropy value of input data of a government task, determining a complexity of the government task based on the entropy value, and determining a pruning ratio using the complexity and hardware resource occupancy rate; A corresponding pruning strategy is determined based on the first target weight parameter, a preset thinning rate, and the pruning ratio.

5. The lightweight method for the government affairs large model according to claim 1 is characterized in that: The step of performing a sensitivity analysis on the hierarchy of the government affairs model based on the target information and the weight parameter to obtain an analysis result, and determining a quantization interval of the weight parameter, and determining a quantization strategy using the analysis result and the quantization interval includes: Performing a sensitivity analysis on the hierarchy of the government affairs model based on the target information, the weight parameter, the hierarchy of the government affairs model, and a preset Gaussian perturbation, and obtaining corresponding analysis results; Determining the quantization interval of the weight parameter using a preset quantile statistics method; Determining, from the analysis results, a first level whose sensitivity satisfies a preset high sensitivity condition and a second level whose sensitivity satisfies a preset low sensitivity condition; Based on the quantization interval, a mixed precision quantization strategy is adopted for the first level, and a preset bit quantization strategy is adopted for the second level to determine a corresponding quantization strategy; the mixed precision quantization strategy is a quantization strategy that includes preset precision fixed-point numbers and preset precision floating-point numbers.

6. The lightweight method for a large government affairs model according to any one of claims 1 to 5, characterized in that: The lightening of the government affairs large model by using the pruning strategy and the quantization strategy to obtain a corresponding target lightening model includes: Pruning the large government affairs model using the pruning strategy to obtain a sparse model, and quantizing the sparse model using the quantization strategy to obtain an initial lightweight model; Iteratively optimizing the initial lightweight model based on a preset government data set until the size of the lightweight model in the iteration is reduced to a preset model size or the accuracy of the government task is reduced to a preset accuracy; the government task is a task in which the lightweight model in the iteration processes the input government data; The lightweight model at the end of the iteration is determined as the target lightweight model.

7. The lightweight method for the government affairs large model according to claim 6 is characterized in that: After pruning the government affairs model using the pruning strategy, the method further includes: Determining a second target weight parameter to be retained in the government affairs model, and determining data loss during the pruning process based on the second target weight parameter; Based on the proximal gradient method and the current weight parameters of the current government affairs model, the constraints in the data supplement process are determined, and the data loss is supplemented accordingly using the constraints, the preset government affairs field knowledge graph and the knowledge injection intensity coefficient.

8. A lightweight device for a large government model, characterized in that: include: An information determination module is used to determine metadata of the government affairs model and construct a semantic association matrix based on a preset government affairs domain knowledge graph to determine target information based on the metadata and the semantic association matrix; a pruning strategy determination module, configured to perform a credibility assessment on the weight parameters of the government affairs model based on the target information, and determine a pruning strategy based on the obtained assessment results; wherein the credibility is the importance of the weight parameters in the government affairs task scenario; a quantification strategy determination module, configured to perform a sensitivity analysis on the hierarchical levels of the government affairs model based on the target information and the weight parameters, obtain analysis results, determine a quantization interval for the weight parameters, and determine a quantization strategy using the analysis results and the quantization interval; wherein sensitivity refers to the contribution of each hierarchical level of the government affairs model to government affairs information processing; The model lightweight module is used to lightweight the government affairs model by using the pruning strategy and the quantization strategy to obtain a corresponding target lightweight model.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor is used to execute the computer program to implement the lightweight method of the government affairs big model as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that Used to store computer programs; wherein, when the computer program is executed by a processor, it implements the lightweight method of the government affairs big model as described in any one of claims 1 to 7.

Images