Sensitive layer searching method and device in deep learning model and computer equipment

Abstract

The invention discloses a sensitive layer searching method and device in a deep learning model and computer equipment. The method comprises the steps that configuration data adopted by a target model in the training process is collected, and the configuration data at least comprises one of the following components: a weight value, an activation value and a gradient value of each layer in the target model; acquiring data distribution characteristics of the configuration data, and extracting structural characteristics of each network layer in the target model; and analyzing the data distribution characteristics and the structural characteristics by adopting a pre-trained evaluation model to obtain the sensitivity of each layer in the target model, and determining the layer of which the sensitivity is higher than a preset threshold value in each layer as a sensitive layer. The method at least solves the technical problem of low accuracy of the search result of the sensitive layer in the related art.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and more specifically, to a method, apparatus, and computer device for searching sensitive layers in a deep learning model. Background Art

[0002] In deep learning model quantization, sensitive layer search is the process of identifying layers within the model that are sensitive to quantization. Currently, common sensitive layer search methods fall into the following categories: Empirical methods: Based on empirical experience with different model structures and tasks, certain layers are artificially designated as sensitive layers. For example, in convolutional neural networks (CNNs), convolutional layers near the input are generally considered to be more sensitive to quantization because these layers process raw image features, and quantization may result in the loss of important information. However, this method lacks versatility and accuracy. The distribution of sensitive layers may vary across different models and datasets, making it difficult to accurately identify the true sensitive layers. Layer-by-layer quantization testing: Each layer of the model is quantized sequentially and the performance change after quantization is evaluated. By comparing performance metrics (such as precision, recall, and mean squared error) after quantization, layers with the largest performance degradation are identified as sensitive layers. While this method can accurately identify sensitive layers, it is computationally complex and requires multiple quantization and evaluation cycles, resulting in significant time and computational resource consumption. This computational burden is particularly prohibitive for large-scale deep learning models. Methods based on sensitivity index calculations: These methods define sensitivity indices to measure each layer's sensitivity to quantization. For example, they calculate the variance of each layer's weights and gradients. The larger the variance, the more sensitive the layer is to quantization. Other methods utilize concepts from information theory, such as calculating the entropy of each layer's output to measure information richness. Layers with higher entropy are considered more sensitive to quantization. While these methods can quickly assess layer sensitivity to a certain extent, the relationship between sensitivity indices and post-quantization model performance is not completely linear. They may not accurately reflect the actual impact of quantization on model performance, leading to inaccurate search results. Summary of the Invention

[0003] The embodiments of the present application provide a method, apparatus, and computer device for searching sensitive layers in a deep learning model to at least solve the technical problem of low accuracy of sensitive layer search results in related technologies.

[0004] According to one aspect of an embodiment of the present application, a method for searching for sensitive layers in a deep learning model is provided, comprising: collecting configuration data used by a target model during training, wherein the configuration data includes at least one of the following: weight values, activation values, and gradient values of each layer in the target model; obtaining data distribution characteristics of the configuration data, and extracting structural characteristics of each network layer in the target model; using a pre-trained evaluation model to analyze the data distribution characteristics and the structural characteristics to obtain the sensitivity of each layer in the target model, and determining the layer whose sensitivity is higher than a preset threshold in each layer as a sensitive layer.

[0005] Optionally, data distribution characteristics of the configuration data are obtained, and structural characteristics of each layer in the target model are extracted, including: normalizing the configuration data to obtain normalized configuration data; respectively obtaining the mean, variance and data entropy of each type of data in the normalized configuration data, and determining the mean, variance and data entropy of each type of data as the data distribution characteristics; obtaining the model type of the target model, and obtaining the structural characteristics of the target model according to the model type.

[0006] Optionally, the structural characteristics of the target model are obtained according to the model type, including: when the target model is a first type of model, respectively obtaining the structural characteristics of the convolutional layer and the structural characteristics of the fully connected layer in the target model, the structural characteristics of the convolutional layer include at least one of the following: convolution kernel size, step size, and padding method, and the structural characteristics of the fully connected layer include at least: the number of neurons; when the target model is a second type of model, obtaining the structural characteristics of the encoder layer in the target model, the structural characteristics of the encoder layer include at least one of the following: the number of attentions and the dimension of the hidden layer in the encoder layer, wherein the first type of model includes a convolutional layer and a fully connected layer; the second type of model only includes an encoder layer.

[0007] Optionally, a pre-trained evaluation model is used to analyze the data distribution characteristics and the structural characteristics, including: constructing a feature vector for each layer in the target model based on the data distribution characteristics and the structural characteristics; using the pre-trained evaluation model to analyze the feature vector to obtain the prediction result, which is used to indicate the sensitivity of each layer in the target model.

[0008] Optionally, the method further includes: constructing an initial model of the pre-trained evaluation model, the initial model including: an input layer, multiple hidden layers and an output layer; obtaining a training data set, the training data set including: model parameters of multiple known quantization results; using the training data set to train the initial model to obtain the pre-trained evaluation model.

[0009] Optionally, after determining the sensitive layer in the target model, the method further includes: quantizing the sensitive layer in the target model using a first type of quantization strategy; quantizing the insensitive layer in the target model using a second type of quantization strategy to obtain a quantized target model, wherein the quantization accuracy of the first type of quantization strategy is higher than that of the second type of quantization strategy.

[0010] Optionally, the method further includes: obtaining multiple candidate thresholds; screening out the sensitive layer in the target model according to each candidate threshold; quantizing the sensitive layer and the non-sensitive layer using the first type of quantization strategy and the second type of quantization strategy respectively to obtain the target model after quantization; using a validation set to evaluate the performance degradation of the target model after quantization and the target model; and selecting the candidate threshold with the smallest performance degradation from the multiple candidate thresholds as the preset threshold.

[0011] According to another aspect of an embodiment of the present application, a device for searching for sensitive layers in a deep learning model is also provided, including: an acquisition module for acquiring configuration data adopted by a target model during training, wherein the configuration data includes at least one of the following: weight values, activation values, and gradient values of each layer in the target model; a feature module for obtaining data distribution characteristics of the configuration data and extracting structural characteristics of each network layer in the target model; a determination module for analyzing the data distribution characteristics and the structural characteristics using a pre-trained evaluation model to obtain the sensitivity of each layer in the target model, and determining the layer whose sensitivity is higher than a preset threshold in each layer as a sensitive layer.

[0012] According to another aspect of an embodiment of the present application, a computer device is also provided, including: a memory and a processor, wherein the memory is used to store program instructions; the processor is connected to the memory and is used to execute the sensitive layer search method in the above-mentioned deep learning model.

[0013] According to another aspect of an embodiment of the present application, a non-volatile storage medium is also provided, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the sensitive layer search method in the above-mentioned deep learning model by running the computer program.

[0014] According to another aspect of the embodiments of the present application, a computer program product is also provided, including computer instructions, which, when executed by a processor, implement the sensitive layer search method in the above-mentioned deep learning model.

[0015] In an embodiment of the present application, configuration data used by the target model during the training process is collected, wherein the configuration data includes at least one of the following: weight values, activation values, and gradient values of each layer in the target model; data distribution characteristics of the configuration data are obtained, and structural characteristics of each network layer in the target model are extracted; a pre-trained evaluation model is used to analyze the data distribution characteristics and the structural characteristics to obtain the sensitivity of each layer in the target model, and the layer with a sensitivity higher than a preset threshold in each layer is determined as a sensitive layer, thereby achieving the purpose of determining the sensitive layer according to the distribution characteristics of the configuration data of the target model and the structural characteristics of the target model, thereby achieving the technical effect of improving the accuracy of sensitive layer determination, and further solving the technical problem of low accuracy of sensitive layer search results in related technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0017] Figure 1 This is a hardware structure block diagram of a computer terminal for implementing a sensitive layer search method in a deep learning model according to an embodiment of the present application;

[0018] Figure 2 is a flowchart of a sensitive layer search method in a deep learning model according to an embodiment of the present application;

[0019] Figure 3 is a flowchart of another sensitive layer search method in a deep learning model according to an embodiment of the present application;

[0020] Figure 4 is a flow chart of a model quantization method according to an embodiment of the present application;

[0021] Figure 5 This is a structural diagram of a sensitive layer search device in a deep learning model according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0023] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0024] In order to better understand the embodiments of the present application, the technical terms involved in the embodiments of the present application are explained as follows:

[0025] Deep learning model: A machine learning model based on artificial neural networks. By constructing a multi-layered neural network structure, it can automatically learn complex patterns and feature representations from large amounts of data, thereby achieving tasks such as data classification, prediction, and generation. For example, convolutional neural networks (CNNs) are used for image recognition, recurrent neural networks (RNNs) and their variants, such as LSTM and GRU, are used to process sequential data, and the Transformer model has performed well in the field of natural language processing.

[0026] Model quantization: The process of converting parameters and activation values in a deep learning model, originally represented by higher precision (such as 32-bit floating-point numbers), to lower precision (such as 8-bit integers, 16-bit floating-point numbers, etc.). Quantization can reduce the model's storage space, lower the memory bandwidth requirements during the calculation process, and speed up inference. It is particularly suitable for deploying deep learning models on resource-constrained devices (such as mobile devices and embedded devices). Quantization can be divided into linear quantization and nonlinear quantization based on the quantization method. It can be divided into layer-by-layer quantization, group-by-group quantization, and channel-by-channel quantization based on the quantization granularity. It can be divided into online quantization and offline quantization based on the activation value quantization method.

[0027] Sensitive layers: In deep learning models, these layers are particularly sensitive to quantization. Improper quantization of these layers can significantly impact the overall accuracy of the model. These layers typically contain important features or decision-making components of the model. For example, in an image recognition model, early convolutional layers may be crucial for extracting basic image features. Therefore, quantization requires careful handling, as failure to do so can significantly reduce the accuracy of subsequent tasks such as classification.

[0028] Quantization table: A table used to define quantization rules during the model quantization process. It contains key parameters in the quantization process, such as the scale factor and offset (zero point). Different quantization methods and application scenarios may use different quantization tables. For example, in uniform quantization, the quantization table defines how to divide the continuous floating-point value range into a finite number of discrete quantization levels; in non-uniform quantization, the quantization table assigns different quantization step sizes to data in different ranges based on the data distribution characteristics to better preserve important data information. The quantization table plays a guiding role in the quantization process, through which the original high-precision data can be accurately converted to a low-precision representation.

[0029] The information collected in the embodiments of the present application is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with the relevant laws, regulations and standards of the relevant regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or reject the automated decision results; if the user chooses to reject, the expert decision-making process will be entered.

[0030] In order to solve the problems existing in the related art, the embodiment of the present application provides a method for searching sensitive layers in a deep learning model. Figure 1 In the computer terminal shown, the computer terminal is explained below.

[0031] The embodiment of the sensitive layer search method in the deep learning model provided in the embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 The hardware structure block diagram of a computer terminal for implementing a sensitive layer search method in a deep learning model is shown. Figure 1 As shown, the computer terminal 10 may include one or more (illustrated by 102a, 102b, ..., 102n in the figure) processors (the processor may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions connected via a wired and / or wireless network. In addition, it may also include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, and a BUS bus. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components than shown, or with Figure 1Different configurations shown.

[0032] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry." The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuitry may be a single, independent processing module, or may be incorporated in whole or in part into any of the other components of the computer terminal 10. As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0033] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the sensitive layer search method in the deep learning model in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, implementing the sensitive layer search method in the deep learning model mentioned above. The memory 104 may include a high-speed random access memory and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely located relative to the processor, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0034] The transmission module 106 is configured to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by the communications provider of the computer terminal 10. In one embodiment, the transmission module 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission module 106 may be a radio frequency (RF) module, which is configured to communicate with the Internet wirelessly.

[0035] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 .

[0036] It should be noted that, in some optional embodiments, the above Figure 1 The computer terminal shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of hardware elements and software elements. Figure 1This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the computer terminal described above.

[0037] In the above-mentioned operating environment, an embodiment of the present application provides an embodiment of a sensitive layer search method in a deep learning model. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0038] Figure 2 This is a flowchart of a sensitive layer search method in a deep learning model according to an embodiment of the present application. Figure 2 As shown, the method includes the following steps:

[0039] Step S202: collecting configuration data used by the target model during training, wherein the configuration data includes at least one of the following: weight values, activation values, and gradient values of each layer in the target model;

[0040] Step S204, obtaining data distribution characteristics of the configuration data, and extracting structural characteristics of each network layer in the target model;

[0041] In step S206 , a pre-trained evaluation model is used to analyze the data distribution characteristics and the structural characteristics to obtain the sensitivity of each layer in the target model, and a layer having a sensitivity higher than a preset threshold is determined as a sensitive layer.

[0042] Through the above steps S202 to S206, the configuration data used by the target model during the training process is collected, wherein the configuration data includes at least one of the following: weight values, activation values, and gradient values of each layer in the target model; data distribution characteristics of the configuration data are obtained, and structural characteristics of each network layer in the target model are extracted; the data distribution characteristics and the structural characteristics are analyzed using a pre-trained evaluation model to obtain the sensitivity of each layer in the target model, and the layers with a sensitivity higher than a preset threshold are determined as sensitive layers, thereby achieving the purpose of determining the sensitive layer based on the distribution characteristics of the configuration data of the target model and the structural characteristics of the target model, thereby achieving the technical effect of improving the accuracy of sensitive layer determination, and further solving the technical problem of low accuracy of sensitive layer search results in related technologies. The following is a detailed description.

[0043] In some embodiments of the present application, the specific steps of obtaining the data distribution characteristics of the configuration data and extracting the structural characteristics of each layer in the target model are as follows: normalizing the configuration data to obtain the normalized configuration data; respectively obtaining the mean, variance and data entropy of each type of data in the normalized configuration data, and determining the mean, variance and data entropy of each type of data as the data distribution characteristics; obtaining the model type of the target model, and obtaining the structural characteristics of the target model according to the model type.

[0044] Among them, the process of obtaining the structural characteristics of the target model according to the model type is as follows: when the target model is a first type of model, the structural characteristics of the convolutional layer and the structural characteristics of the fully connected layer in the target model are respectively obtained, and the structural characteristics of the convolutional layer include at least one of the following: convolution kernel size, step size, and filling method, and the structural characteristics of the fully connected layer include at least: the number of neurons; when the target model is a second type of model, the structural characteristics of the encoder layer in the target model are obtained, and the structural characteristics of the encoder layer include at least one of the following: the number of attentions and the dimension of the hidden layer in the encoder layer, wherein the first type of model includes a convolutional layer and a fully connected layer; the second type of model only includes an encoder layer.

[0045] Taking the calculation of data entropy as an example, it can be determined by the following formula:

[0046]

[0047] In the formula, p(x i ) represents the target type data x i The probability of occurrence, n represents the number of target type data.

[0048] In some embodiments of the present application, the specific steps of using a pre-trained evaluation model to analyze the data distribution characteristics and the structural characteristics are as follows: constructing a feature vector for each layer in the target model based on the data distribution characteristics and the structural characteristics; using the pre-trained evaluation model to analyze the feature vector to obtain the prediction result, which is used to indicate the sensitivity of each layer in the target model.

[0049] It should be noted that each feature vector corresponds to a layer in the model. For example, for a convolutional layer, its feature vector can be expressed as [k h ,k w ,s h ,s w ,μ,σ 2 ,H,...], where k h and k w Indicates the size of the convolution kernel, sh and s w represents the step size, μ represents the mean of the target type data, σ 2 represents the variance of the target type data, and H represents the data entropy of the target type data, where the target type data includes but is not limited to activation values, weight values, and gradient values.

[0050] Taking the target model as a CNN-based image recognition model as an example, the target model is a first-class model. The process of predicting the sensitive layer of the target model is as follows:

[0051] Step 1, Data Preprocessing: Collect data from a convolutional neural network (CNN) model used for image recognition on the image recognition training set, including weights and activation values for the 10 convolutional layers and 3 fully connected layers. Extract the structural features of the convolutional layers. For the first convolutional layer, specify the kernel size as (3,3), stride as (1,1), and padding as "same." Calculate the data distribution characteristics of the activation values, such as data entropy, and obtain the feature vector: [(3,3), (1,1), "same", activation value data entropy].

[0052] Step 2: Predict the sensitivity of each layer: Input the feature vector corresponding to each layer in the model into the trained sensitivity assessment model to obtain the sensitivity prediction value for each layer. Rank the sensitivity prediction values for all layers. Through cross-validation, determine the sensitivity threshold T as 0.6. Layers with sensitivities greater than the threshold T are identified as sensitive layers. Ultimately, the third and seventh convolutional layers in the model are identified as sensitive layers.

[0053] In some embodiments of the present application, an initial model of the pre-trained evaluation model is constructed, wherein the initial model includes: an input layer, multiple hidden layers and an output layer; a training data set is obtained, wherein the training data set includes: model parameters of multiple known quantization results; the initial model is trained using the training data set to obtain the pre-trained evaluation model.

[0054] The extracted structural features and data distribution characteristics of each layer are constructed into a multidimensional feature vector. An MLP neural network is built as a sensitivity assessment model. This model consists of an input layer, multiple hidden layers, and an output layer to determine the sensitivity of each layer of the model to be quantized. The feature vectors of each layer are input into the MLP neural network for training. The model parameters are adjusted by minimizing the error (such as mean squared error) between the predicted sensitivity and the actual change in model performance after quantization.

[0055] Taking the target model as a Transformer-based natural language processing model, step 1, data preprocessing: Collect data such as weights and activations at each layer generated by running the Transformer model for text classification on the text classification training set. The model contains six encoder layers. For the encoder layers, extract structural features, such as the number of attention heads and hidden layer dimensions. Calculate the data distribution characteristics of the activation values at each layer.

[0056] Step 2: Build a sensitivity assessment model: The extracted structural features and data distribution features are transformed into a multidimensional feature vector. For example, the feature vector for a particular encoder layer is [number of attention heads, hidden layer dimensions, activation value data entropy]. A sensitivity assessment model is constructed using an MLP neural network, consisting of an input layer, hidden layers, and an output layer. The MLP neural network is trained using model data with known quantization results to optimize model parameters by minimizing the error between predicted sensitivity and actual post-quantization model performance.

[0057] Step 3: Search strategy execution: Input the feature vectors of each layer into the trained sensitivity evaluation model to obtain sensitivity predictions. Rank the sensitivity predictions. Set the sensitivity threshold T to 0.5 and identify layers with sensitivities greater than the threshold T as sensitive layers. Finally, the second and fifth encoder layers are identified as sensitive layers.

[0058] After determining the sensitive layer in the target model, the sensitive layer in the target model is quantized using a first type of quantization strategy; the non-sensitive layer in the target model is quantized using a second type of quantization strategy to obtain a quantized target model, wherein the quantization accuracy of the first type of quantization strategy is higher than that of the second type of quantization strategy.

[0059] Specifically, after determining the sensitive layer, during the model quantization process, special quantization strategies are adopted for the sensitive layer, such as customizing a more refined quantization table or implementing quantization-aware training; while a general quantization strategy is adopted for the non-sensitive layer, which improves the efficiency and effectiveness of model quantization while ensuring model performance.

[0060] In some embodiments of the present application, a method for determining a preset threshold is also provided, which is as follows: obtaining multiple candidate thresholds; screening out the sensitive layer in the target model according to each candidate threshold; quantizing the sensitive layer and the non-sensitive layer using the first type of quantization strategy and the second type of quantization strategy, respectively, to obtain the target model after quantization; using a validation set to evaluate the performance degradation of the target model after quantization and the target model; and selecting the candidate threshold with the smallest performance degradation from the multiple candidate thresholds as the preset threshold.

[0061] The embodiment of the present application also provides another method for searching sensitive layers in a deep learning model, such as Figure 3 As shown, it includes: collecting model operation data (configuration data); normalizing the operation data; extracting data distribution characteristics and model structure characteristics respectively; constructing feature vectors; predicting sensitivity based on the feature vectors, and determining the sensitive layer according to the sensitivity.

[0062] The present application also provides a model quantization method, such as Figure 4 Shown, including:

[0063] The sensitive layer of the target model is determined through a deep learning model. A special quantization strategy (second type quantization strategy) is used for quantization of the sensitive layer, such as non-uniform quantization and adding quantization-aware training during the model training process. A general quantization strategy (first type quantization strategy) is used for quantization of the non-sensitive layer, such as uniform quantization.

[0064] It's important to note that uniform quantization uniformly maps data to a finite set of discrete values. For example, a given numerical range is divided into several equal intervals, each corresponding to a quantized value. Taking 8-bit integer quantization as an example, the data range is mapped to the 256 integers [-128, 127]. Its advantages are computational simplicity and ease of implementation; however, its disadvantage is that it can lead to large quantization errors when the data distribution is uneven. Non-uniform quantization considers the data distribution characteristics and uses different quantization step sizes for different data ranges. For example, a smaller quantization step size is used for densely distributed data to improve quantization accuracy, while a larger quantization step size is used for sparsely distributed data to reduce quantization error. This method better adapts to the actual data distribution, but has relatively high computational complexity. Quantization-aware training introduces quantization during model training, simulating the effects of low precision. This allows the model to adapt to the information loss caused by quantization during training, thereby maintaining good performance after quantization. This method generally achieves better accuracy than post-training quantization, but the training process is more complex and computationally expensive.

[0065] The specific quantization process is as follows: Sensitive layers typically employ a more refined quantization strategy (Type II quantization strategy), for example, using higher-bit quantization (e.g., 16-bit floating-point quantization) rather than standard low-precision quantization (e.g., 8-bit integer quantization). Dynamic quantization strategies are also considered, meaning that quantization parameters (e.g., scaling factors and zero points) can be dynamically adjusted based on the data to reduce quantization error. Special quantization tables are designed for sensitive layers to ensure that the quantized values more closely match the distribution of the original values, minimizing information loss.

[0066] General quantization strategy for non-sensitive layers (Type 1 quantization strategy): Non-sensitive layers often use static uniform quantization, which is the most commonly used quantization strategy. Weights and activation values are mapped to pre-defined quantization levels. Using a fixed set of quantization tables for non-sensitive layers simplifies the quantization process.

[0067] Based on the above strategy, each layer in the model is quantized. Sensitive layers: Use finer quantization levels or dynamically adjust quantization parameters based on specific quantization strategies. Insensitive layers: Use the general quantization strategy to convert weights and activations to low-precision representations. The quantization results of all layers are aggregated to generate the overall quantized model.

[0068] Figure 5 A device for searching sensitive layers in a deep learning model according to an embodiment of the present application includes:

[0069] An acquisition module 50 is configured to acquire configuration data used by the target model during training, wherein the configuration data includes at least one of the following: weight values, activation values, and gradient values of each layer in the target model;

[0070] A feature module 52 is used to obtain data distribution features of the configuration data and extract structural features of each network layer in the target model;

[0071] The determination module 54 is used to analyze the data distribution characteristics and the structural characteristics using a pre-trained evaluation model to obtain the sensitivity of each layer in the target model, and determine the layer with a sensitivity higher than a preset threshold as a sensitive layer.

[0072] Through the sensitive layer search device in the above-mentioned deep learning model, the configuration data used by the target model in the training process is collected, wherein the configuration data includes at least one of the following: the weight value, activation value and gradient value of each layer in the target model; the data distribution characteristics of the configuration data are obtained, and the structural characteristics of each network layer in the target model are extracted; the data distribution characteristics and the structural characteristics are analyzed by using a pre-trained evaluation model to obtain the sensitivity of each layer in the target model, and the layer with a sensitivity higher than a preset threshold in each layer is determined as a sensitive layer, thereby achieving the purpose of determining the sensitive layer according to the distribution characteristics of the configuration data of the target model and the structural characteristics of the target model, thereby achieving the technical effect of improving the accuracy of sensitive layer determination, and then solving the technical problem of low accuracy of sensitive layer search results in related technologies.

[0073] The feature module 52 includes: a feature submodule, which is used to obtain the data distribution characteristics of the configuration data and extract the structural characteristics of each layer in the target model, including: normalizing the configuration data to obtain the normalized configuration data; respectively obtaining the mean, variance and data entropy of each type of data in the normalized configuration data, and determining the mean, variance and data entropy of each type of data as the data distribution characteristics; obtaining the model type of the target model, and obtaining the structural characteristics of the target model according to the model type.

[0074] The feature submodule includes: a first feature unit and a second feature unit, wherein the first feature unit is used to obtain the structural features of the target model according to the model type, including: when the target model is a first type of model, respectively obtaining the structural features of the convolutional layer and the structural features of the fully connected layer in the target model, the structural features of the convolutional layer include at least one of the following: convolution kernel size, step size, and padding method, and the structural features of the fully connected layer include at least: the number of neurons; when the target model is a second type of model, obtaining the structural features of the encoder layer in the target model, the structural features of the encoder layer include at least one of the following: the number of attentions and the dimension of the hidden layer in the encoder layer, wherein the first type of model contains convolutional layers and fully connected layers; the second type of model only contains encoder layers.

[0075] The second feature unit is used to analyze the data distribution characteristics and the structural characteristics using a pre-trained evaluation model, including: constructing a feature vector for each layer in the target model based on the data distribution characteristics and the structural characteristics; using the pre-trained evaluation model to analyze the feature vector to obtain the prediction result, which is used to indicate the sensitivity of each layer in the target model.

[0076] The determination module 54 includes: a training submodule, which is used to construct an initial model of the pre-trained evaluation model, wherein the initial model includes: an input layer, multiple hidden layers and an output layer; obtain a training data set, wherein the training data set includes: model parameters of multiple known quantization results; and use the training data set to train the initial model to obtain the pre-trained evaluation model.

[0077] The sensitive layer search device in the above-mentioned deep learning model also includes: a quantization submodule, which is used to, after determining the sensitive layer in the target model, quantize the sensitive layer in the target model using a first type of quantization strategy; and quantize the non-sensitive layer in the target model using a second type of quantization strategy to obtain a quantized target model, wherein the quantization accuracy of the first type of quantization strategy is higher than that of the second type of quantization strategy.

[0078] The quantization submodule includes: a threshold unit for obtaining multiple candidate thresholds; screening out the sensitive layer in the target model according to each candidate threshold; quantizing the sensitive layer and the non-sensitive layer using the first type of quantization strategy and the second type of quantization strategy respectively to obtain the target model after quantization; using a validation set to evaluate the performance degradation of the target model after quantization and the target model; selecting the candidate threshold with the smallest performance degradation from the multiple candidate thresholds as the preset threshold.

[0079] It should be noted that Figure 5 The sensitive layer search device in the deep learning model shown is used to perform Figure 2 The sensitive layer search method in the deep learning model shown, therefore the relevant explanations in the above-mentioned sensitive layer search method in the deep learning model are also applicable to the sensitive layer search device in the deep learning model, and will not be repeated here.

[0080] An embodiment of the present application also provides a computer device, including: a memory and a processor, wherein the memory is used to store program instructions; the processor is connected to the memory and is used to execute the sensitive layer search method in the above-mentioned deep learning model.

[0081] The sensitive layer search method in the deep learning model executed by the above-mentioned computer device collects the configuration data used by the target model during the training process, wherein the configuration data includes at least one of the following: the weight value, activation value and gradient value of each layer in the target model; obtains the data distribution characteristics of the configuration data, and extracts the structural characteristics of each network layer in the target model; uses a pre-trained evaluation model to analyze the data distribution characteristics and the structural characteristics to obtain the sensitivity of each layer in the target model, and determines the layer with a sensitivity higher than a preset threshold in each layer as the sensitive layer, thereby achieving the purpose of determining the sensitive layer according to the distribution characteristics of the configuration data of the target model and the structural characteristics of the target model, thereby achieving the technical effect of improving the accuracy of sensitive layer determination, and thus solving the technical problem of low accuracy of sensitive layer search results in related technologies.

[0082] An embodiment of the present application also provides a non-volatile storage medium, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the sensitive layer search method in the above-mentioned deep learning model by running the computer program.

[0083] The sensitive layer search method in the deep learning model stored in the above-mentioned non-volatile storage medium collects the configuration data used by the target model during the training process, wherein the configuration data includes at least one of the following: the weight value, activation value and gradient value of each layer in the target model; obtains the data distribution characteristics of the configuration data, and extracts the structural characteristics of each network layer in the target model; uses a pre-trained evaluation model to analyze the data distribution characteristics and the structural characteristics to obtain the sensitivity of each layer in the target model, and determines the layer with a sensitivity higher than a preset threshold in each layer as the sensitive layer, thereby achieving the purpose of determining the sensitive layer according to the distribution characteristics of the configuration data of the target model and the structural characteristics of the target model, thereby achieving the technical effect of improving the accuracy of sensitive layer determination, and then solving the technical problem of low accuracy of sensitive layer search results in related technologies.

[0084] An embodiment of the present application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps of the sensitive layer search method in the deep learning model in the present application.

[0085] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0086] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0087] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0088] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0089] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0090] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0091] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for searching sensitive layers in a deep learning model, characterized in that: include: Collecting configuration data used by the target model during training, wherein the configuration data includes at least one of the following: weight values, activation values, and gradient values of each layer in the target model; Obtaining data distribution characteristics of the configuration data and extracting structural characteristics of each network layer in the target model; The data distribution characteristics and the structural characteristics are analyzed using a pre-trained evaluation model to obtain the sensitivity of each layer in the target model, and the layer with a sensitivity higher than a preset threshold is determined as a sensitive layer.

2. The method according to claim 1, characterized in that Obtaining data distribution characteristics of the configuration data and extracting structural characteristics of each layer in the target model, including: Normalizing the configuration data to obtain normalized configuration data; Respectively obtaining the mean, variance, and data entropy of each type of data in the normalized configuration data, and determining the mean, variance, and data entropy of each type of data as the data distribution feature; The model type of the target model is obtained, and the structural features of the target model are obtained according to the model type.

3. The method according to claim 2, characterized in that Acquiring structural features of the target model according to the model type includes: When the target model is a first type model, respectively obtaining structural features of a convolutional layer and a fully connected layer in the target model, wherein the structural features of the convolutional layer include at least one of the following: convolution kernel size, step size, and padding mode, and the structural features of the fully connected layer include at least the number of neurons; In the case where the target model is a second type of model, structural features of the encoder layer in the target model are obtained, and the structural features of the encoder layer include at least one of the following: the number of attentions and the dimension of the hidden layer in the encoder layer, wherein the first type of model includes a convolutional layer and a fully connected layer; the second type of model only includes an encoder layer.

4. The method according to claim 2, characterized in that The data distribution characteristics and the structural characteristics are analyzed using a pre-trained evaluation model, including: Constructing a feature vector for each layer in the target model according to the data distribution characteristics and the structural characteristics; The pre-trained evaluation model is used to analyze the feature vector to obtain a prediction result, which is used to indicate the sensitivity of each layer in the target model.

5. The method according to claim 1, wherein The method further comprises: Constructing an initial model of the pre-trained evaluation model, the initial model comprising: an input layer, a plurality of hidden layers, and an output layer; Acquire a training data set, wherein the training data set includes: model parameters of a plurality of known quantization results; The initial model is trained using the training data set to obtain the pre-trained evaluation model.

6. The method according to claim 1, characterized in that After determining the sensitive layer in the target model, the method further includes: quantizing the sensitive layer in the target model using a first type of quantization strategy; The non-sensitive layer in the target model is quantized using a second type of quantization strategy to obtain a quantized target model, wherein the quantization accuracy of the first type of quantization strategy is higher than that of the second type of quantization strategy.

7. The method according to claim 6, characterized in that The method further comprises: Obtain multiple candidate thresholds; Filtering out the sensitive layers in the target model according to each candidate threshold respectively; quantizing the sensitive layer and the non-sensitive layer using the first type of quantization strategy and the second type of quantization strategy respectively to obtain a quantized target model; Using a validation set to evaluate the performance degradation of the target model after the quantization process and the target model; A candidate threshold with the smallest performance degradation amount is selected from the multiple candidate thresholds and determined as the preset threshold.

8. A sensitive layer search device in a deep learning model, characterized in that: include: An acquisition module is used to acquire configuration data used by the target model during training, wherein the configuration data includes at least one of the following: weight values, activation values, and gradient values of each layer in the target model; A feature module, configured to obtain data distribution features of the configuration data and extract structural features of each network layer in the target model; The determination module is used to analyze the data distribution characteristics and the structural characteristics using a pre-trained evaluation model to obtain the sensitivity of each layer in the target model, and determine the layer whose sensitivity is higher than a preset threshold as a sensitive layer.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory is used to store program instructions; The processor is connected to the memory and is used to execute the sensitive layer search method in the deep learning model described in any one of claims 1 to 7.

10. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the sensitive layer search method in the deep learning model described in any one of claims 1 to 7 is implemented.

Images