Model optimization method and related device

Abstract

The invention relates to the field of data processing, in particular to a model optimization method and a related device. The method comprises the steps of obtaining a to-be-optimized first model; the first model is a neural network model meeting a preset optimization condition; the preset optimization condition is determined based on the type of the first model; performing adaptive quantization processing on a to-be-optimized structure in the first model to obtain a second model; wherein different quantization factors are adopted for different to-be-optimized structures in the adaptive quantization processing, and the quantization factors are obtained through adaptive calculation of characteristic values contained in the to-be-optimized structures; performing mixed precision decomposition on the second model to obtain a first decomposition model containing abnormal characteristic values and a second decomposition model containing non-abnormal characteristic values; and respectively optimizing the first decomposition model and the second decomposition model according to a preset strategy, and fusing optimization results into a final output third model. The method can improve the model quantification precision, reduce the occupied space of the model and improve the model processing efficiency.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a model optimization method and related devices. Background Art

[0002] In the field of modern artificial intelligence, large-scale language models, such as the BLOOM model, have attracted widespread attention for their excellent generation and understanding capabilities. Such models usually have a large number of parameters, which leads to a significant increase in the storage and reasoning costs of the models. Traditional floating-point representation methods (such as FP32) take up a lot of memory and computing resources. In order to effectively reduce these costs, quantization technology is widely used in the field of model compression and acceleration.

[0003] In the related technologies, asymmetric (zero-point) quantization and absolute maximum (absmax) quantization technologies have attracted much attention because they can significantly reduce resource consumption while maintaining high accuracy. However, the activation value distribution in large language models is complex, and there are a large number of abnormal eigenvalues ​​(i.e., outliers), and these abnormal eigenvalues ​​will cause the quantization accuracy to further decrease during the quantization process. In addition, although the quantization technology in the related technologies can reduce the memory usage and computational complexity of the model, in some application scenarios, the additional overhead generated by the quantization process (such as quantization / dequantization operations) will also cause the model inference speed to decrease, and it is impossible to achieve the purpose of improving model performance and improving model processing efficiency.

[0004] Therefore, it is urgent to design a technical solution to solve at least one of the above technical problems. Summary of the invention

[0005] In response to the technical problems existing in the prior art, the present application provides a model optimization method and related devices to solve the problems of decreased quantization accuracy due to complex activation value distribution and difficulty in eliminating outliers during the quantization process, as well as poor model performance caused by quantization overhead.

[0006] In a first aspect, an embodiment of the present application provides a model optimization method, the method comprising: Acquire a first model to be optimized; the first model is a neural network model that meets a preset optimization condition; the preset optimization condition is determined based on the type of the first model; The structure to be optimized in the first model is subjected to adaptive quantization processing to obtain a second model; wherein different quantization factors are used for different structures to be optimized in the adaptive quantization processing: the quantization factors are obtained by adaptively calculating the eigenvalues ​​contained in the structure to be optimized; Performing mixed precision decomposition on the second model to obtain a first decomposition model including abnormal eigenvalues ​​and a second decomposition model including non-abnormal eigenvalues; The first decomposition model and the second decomposition model are optimized respectively according to a preset strategy, and the optimization results are integrated into a third model for final output.

[0007] In a second aspect, the present application provides a model optimization device, wherein An acquisition unit is configured to acquire a first model to be optimized; the first model is a neural network model that meets a preset optimization condition; the preset optimization condition is determined based on the type of the first model; A quantization unit is configured to perform adaptive quantization processing on the structure to be optimized in the first model to obtain a second model; wherein different quantization factors are used for different structures to be optimized in the adaptive quantization processing: the quantization factors are adaptively calculated from the eigenvalues ​​contained in the structure to be optimized; a decomposition unit configured to perform mixed precision decomposition on the second model to obtain a first decomposition model including abnormal eigenvalues ​​and a second decomposition model including non-abnormal eigenvalues; The optimization unit is configured to optimize the first decomposition model and the second decomposition model respectively according to a preset strategy, and merge the optimization results into a third model that is finally output.

[0008] In a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising: at least one processor, memory, and input-output unit; The memory is used to store a computer program, and the processor is used to call the computer program stored in the memory to execute the model optimization method of the first aspect.

[0009] In a fourth aspect, a computer-readable storage medium is provided, which includes instructions, and when the instructions are executed on a computer, the computer executes the model optimization method of the first aspect.

[0010] The beneficial effect of the present application is: a model optimization method and related devices are provided. In this technical solution, first, a first model to be optimized is obtained; the first model is a neural network model that meets the preset optimization conditions; the preset optimization conditions are determined based on the type of the first model. Then, the structure to be optimized in the first model is adaptively quantized to obtain a second model. In the adaptive quantization process, different quantization factors are used for different structures to be optimized: the quantization factors are adaptively calculated from the eigenvalues ​​contained in the structure to be optimized. Then, a mixed precision decomposition is performed on the second model to obtain a first decomposition model containing abnormal eigenvalues, and a second decomposition model containing non-abnormal eigenvalues. Finally, the first decomposition model and the second decomposition model are optimized respectively according to the preset strategy, and the optimization results are fused into the third model for the final output.

[0011] The technical solution of this application, by adaptively processing different structures in the neural network model and separating outliers and non-outliers for quantization processing, not only helps to retain the original data information of the model, but also can effectively avoid the decrease in quantization accuracy caused by the complex distribution of outliers, further improve the model quantization accuracy, reduce the space occupied by the model, and improve the model processing efficiency. In addition, using different strategies to process outliers and non-outliers can also save model quantization overhead, improve model performance, and further improve model processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a flow chart of a model optimization method according to an embodiment of the present application; Figure 2 It is a schematic diagram of the effect of a model optimization method in an embodiment of the present application; Figure 3 It is a structural schematic diagram of a model optimization method device according to an embodiment of the present application; Figure 4 is a schematic diagram of the structure of an electronic device according to an embodiment of the present application; Figure 5 It is a structural schematic diagram of a medium device according to an embodiment of the present application. DETAILED DESCRIPTION

[0013] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0014] In the field of modern artificial intelligence, especially in the application of large-scale language models such as BLOOM, quantization technology is of great significance as an effective means to reduce storage and computing costs. BLOOM is a powerful deep learning model specialized for natural language processing tasks such as text generation, sentiment analysis, and machine translation.

[0015] In related technologies, the activation value distribution of large-scale language models is usually very complex and contains a large number of abnormal values ​​(outliers). These abnormal values ​​will significantly affect the quantization accuracy during the quantization process. Traditional quantization methods often do not handle abnormal values ​​sufficiently, which affects the quantization effect and makes the model accuracy unstable. Outliers may cause the quantized representation to fail to accurately capture the characteristics of the original data, thereby causing instability in the calculation process and affecting the overall performance of the model. In addition, although quantization technology can reduce storage requirements and computational complexity, in some cases, the additional overhead of quantization and dequantization operations may cause the inference speed to not increase as expected, or even decrease. The additional calculations of the quantization process may offset the performance improvement it brings. Coupled with the differences in the level of support for low-precision data by different hardware, some hardware is not efficient in processing these data, and ultimately fails to significantly improve the inference speed.

[0016] Therefore, it is urgent to design a technical solution to solve at least one of the above technical problems.

[0017] In order to solve at least one technical problem in the related art, an embodiment of the present application provides a model optimization method and related devices.

[0018] In the technical solution provided by the present application, first, a first model to be optimized is obtained; the first model is a neural network model that meets the preset optimization conditions; the preset optimization conditions are determined based on the type of the first model. Then, the structure to be optimized in the first model is subjected to adaptive quantization processing to obtain a second model. In the adaptive quantization processing, different quantization factors are used for different structures to be optimized: the quantization factors are adaptively calculated by the eigenvalues ​​contained in the structure to be optimized. By adaptively calculating the quantization factors, the method can more accurately process the eigenvalues ​​in different structures to be optimized, especially effectively deal with complex activation value distributions and outliers. This dynamic adjustment can reduce information loss in the quantization process and improve model accuracy. Different quantization factors are used according to the specific levels and features, so that each layer can reduce the computational complexity and storage requirements as much as possible while maintaining accuracy. Then, mixed precision decomposition is performed on the second model to obtain a first decomposition model containing abnormal eigenvalues ​​and a second decomposition model containing non-abnormal eigenvalues. Abnormal eigenvalues ​​(outliers) and non-abnormal values ​​are processed separately, and the abnormal value part is processed with higher precision (such as FP16) to ensure that these abnormal values ​​will not have a negative impact on the quantization effect. Processing non-abnormal eigenvalues ​​with lower precision (such as INT8) can significantly speed up the calculation process, thereby reducing resource consumption without significantly losing precision. Finally, the first decomposition model and the second decomposition model are optimized according to the preset strategy, and the optimization results are merged into the third model for the final output. Optimization strategies are applied to different decomposed models. For example, a more accurate calculation method can be used for the abnormal eigenvalue part to reduce errors. Compression technology can be further applied to the non-abnormal part to improve processing efficiency. By merging the optimization results of the two decomposition models, the final model generated can comprehensively utilize the advantages of the two precision strategies, which not only reduces the computing and storage burden, but also ensures model performance.

[0019] The technical solution of this application, by adaptively processing different structures in the neural network model and separating outliers and non-outliers for quantization processing, not only helps to retain the original data information of the model, but also can effectively avoid the decrease in quantization accuracy caused by the complex distribution of outliers, further improve the model quantization accuracy, reduce the space occupied by the model, and improve the model processing efficiency. In addition, using different strategies to process outliers and non-outliers can also save model quantization overhead, improve model performance, and further improve model processing efficiency.

[0020] The model optimization method scheme provided in the embodiment of the present application can also be executed by an electronic device, which can be a server, a server cluster, or a cloud server. The electronic device can also be a terminal device such as a mobile phone, a computer, a tablet computer, a wearable device, or a dedicated device (such as a dedicated terminal device with a model optimization method system). These electronic devices can also be equipped with the chips introduced in the above embodiments. Alternatively, these electronic devices can also be installed with a service program for executing the model optimization method scheme.

[0021] Figure 1 A flow chart of a model optimization method provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the method comprises the following steps: 101, obtaining a first model to be optimized; 102, performing adaptive quantization processing on the structure to be optimized in the first model to obtain a second model; 103, performing mixed precision decomposition on the second model to obtain a first decomposition model including abnormal eigenvalues ​​and a second decomposition model including non-abnormal eigenvalues; 104. Optimize the first decomposition model and the second decomposition model respectively according to a preset strategy, and merge the optimization results into a third model that is finally output.

[0022] In the embodiment of the present application, the first model is a neural network model that reaches the preset optimization conditions. Specifically, these preset optimization conditions may include the following aspects: First, from the performance perspective, the model has achieved the minimum requirements of the existing usage scenarios in certain key performance indicators (such as accuracy, recall rate, F1 score, etc.). This means that the model has stable operation performance and eliminates models that are not suitable for basic tasks. Second, from the perspective of resource consumption, the resource consumption of the initial model (such as memory usage, calculation time) is high, reaching a certain threshold that needs to be optimized. This threshold may be set due to production requirements or hardware limitations. Third, from the perspective of structural complexity, the structural complexity of the model, such as the number of layers and the number of parameters, has reached the level that needs to be optimized. Generally, this means that the model has sufficiently complex feature expression capabilities, but there is also redundancy. Fourth, from the perspective of application scenarios, according to the application scenarios and requirements of the model, the type and structure of the model require specific tuning. For example, some real-time applications require the model to be optimized to meet latency constraints. Fifth, from the perspective of data scenarios, due to the special distribution or characteristics of the input data (such as containing a large number of outliers), more precise processing is required through optimization techniques.

[0023] By ensuring that the model meets these preset optimization conditions, the pertinence of the query to the model to be optimized can be effectively improved, ensuring that the subsequent adaptive quantization and precision decomposition process can play its maximum role, and ultimately achieving fine-tuning and performance improvement of the target task. In this way, optimization efforts and resources can be focused on models that really have optimization potential and needs.

[0024] Further, the preset optimization condition is determined based on the type of the first model. This means that for different types of neural network models, the preset optimization conditions will be different. This differentiated setting is because different types of models have different characteristics and requirements in structure and application. Specifically, for convolutional neural networks (CNNs) used for image or video processing, the optimization conditions may include the performance of the model in feature extraction efficiency, such as the required computing resources and processing speed. Limits on the number of parameters may be preset to maintain a certain computing and storage efficiency. For recurrent neural networks (RNNs) and variants (such as LSTM, GRU), these networks are often used to process time series data, so the optimization conditions may emphasize the performance of the model on long sequence data, such as short delay and efficient sequence state management. The optimization condition may be to ensure stable gradient updates to avoid gradient vanishing or exploding problems. For Transformers, the optimization condition may focus on the computational efficiency of the self-attention mechanism because its computational complexity is usually high. Upper limits on network parameters and the number of layers may be set to balance the high precision requirements and memory consumption of the model. For generative adversarial networks (GANs), the optimization conditions may focus on the balanced development of the generator and the discriminator to ensure generation quality and training stability. Conditions may need to be set to detect and avoid mode collapse. Optimization conditions for graph neural networks (GNNs) may involve processing efficiency for different graph structures (such as number of nodes and edge density). It may emphasize the speed of training convergence under different graph topologies.

[0025] By specifying the corresponding preset optimization conditions according to the type of model, the model can be optimized more accurately to maximize the performance and resource utilization efficiency of the model in its specific application field. This not only solves the inherent challenges of the model, but also effectively improves its performance in practical applications.

[0026] In another scenario, the preset optimization conditions of large-scale language models (such as BLOOM) are usually set according to their structural characteristics and application requirements. These conditions include several key aspects: First, in terms of computing resource constraints, memory usage needs to be considered, because large-scale language models have a large number of parameters, so the optimization conditions may require inference to be completed under limited hardware resources (such as GPU / TPU memory). In addition, the maximum computing time for processing each input data needs to be set to meet the needs of real-time or near-real-time applications. Second, in terms of model accuracy and performance, the optimization conditions may include ensuring performance indicators (such as accuracy and robustness) on specific tasks to maintain a high level of performance in the main application scenarios, while keeping the language generation quality and semantic understanding ability from significantly decreasing, especially during optimization processes such as quantization and pruning. In terms of model compression and acceleration, set the parameter pruning ratio or strategy to reduce the model size while maintaining model accuracy, and select appropriate quantization strategies, such as mixed precision, to reduce computational complexity and deployment costs. In terms of training and inference efficiency, the optimization conditions may involve improving the convergence speed of model training and reducing training time, especially in resource-limited environments, while setting the maximum allowable delay in the inference phase to ensure response speed. Finally, in terms of robustness and scalability, the model needs to remain robust in the face of noisy data or changes in input distribution. Optimization conditions may include verification through robustness testing, while considering the scalability of the model on different platforms or devices. These preset optimization conditions guide the design and optimization of large-scale language models, making them more efficient and effective in practical applications, not only helping to achieve excellent performance under limited resource conditions, but also supporting flexible deployment in diverse application scenarios.

[0027] In addition to the above preset optimization conditions, the neural network model can also be directly selected as the first model to be optimized. Alternatively, step 101 can also be implemented in other ways, which will not be described here.

[0028] As an optional embodiment, it is assumed that the structure to be optimized includes at least: the input matrix and / or model weight in the first model. Based on the above assumption, in 102, first, the quantization factors corresponding to the input matrix and / or model weight in the first model are adaptively calculated. The quantization factor is calculated based on the absolute value in the row or column to which the structure to be processed in the first model belongs. Then, the input matrix and / or model weight in the first model is vector-level quantized by the calculated quantization factor to obtain the second model.

[0029] In this optional embodiment, the structure to be optimized includes an input matrix and / or model weights in a neural network model. By adaptively calculating the quantization factors corresponding to these structures to be optimized and quantizing based on the absolute values ​​of rows or columns, significant optimization in terms of calculation and storage can be achieved.

[0030] Quantizing the input matrices and weights by quantization factors simplifies the original floating-point operations into fixed-point operations. This change significantly reduces computational complexity because fixed-point operations require fewer hardware resources than floating-point operations and are faster to execute. The storage space required for quantized data is usually smaller than that for unquantized data because quantization usually maps high-bit floating-point numbers to low-bit integer representations. This reduction is medically interpreted as reduced memory requirements and enhanced deployment capabilities on devices or platforms. Vector-level quantization can improve processing efficiency, especially in resource-constrained environments. This improvement comes from the direct improvement of simplified matrix operations (such as multiplication and addition) due to the fast integer operations. Since quantization is performed while ensuring that the information loss in the mapping process from the original model to the quantized model is minimal, in some cases, the generalization performance and robustness of the quantized model can be maintained at a high level. Quantized image or text processing models usually have faster response times in the inference phase because integer operations and reduced data usage speed up the entire data flow. Despite the many benefits, quantization may bring a slight loss in model accuracy. This mainly depends on the quantization strategy and quantization level. Therefore, the relationship between accuracy and performance needs to be weighed during implementation.

[0031] In general, by using adaptive quantization factors to quantize the input matrix and model weights at the vector level, an optimized second model can be constructed, which improves computing and storage efficiency without significantly compromising model performance. This makes it suitable for deployment in many application scenarios where resources are limited or require fast response.

[0032] Further optionally, in the above steps, the quantization factors corresponding to the input matrices and / or model weights in the first model are adaptively calculated, which can be implemented as follows: for the input matrices and / or model weights in the first model, the target eigenvalue with the largest absolute value in each row or column of the input matrix and / or model weight is selected, and the absolute value of the target eigenvalue is used as the quantization factor.

[0033] In this implementation process, by adaptively calculating the input matrix and / or model weight quantization factor in the first model, it is implemented as selecting the eigenvalue with the largest absolute value in each row or column, and using the absolute value of the eigenvalue as the quantization factor. This method can bring many beneficial effects. Specifically, by selecting the eigenvalue with the largest absolute value in each row or column as the quantization factor, the error that may be caused in the quantization process can be effectively reduced. This is because the maximum absolute value can better represent the data distribution characteristics of the row or column, and retain important information to a greater extent during quantization. When each row or column of the input matrix and weight is scaled by its maximum eigenvalue, the relative amplitude relationship between the values ​​in the data is maintained. This helps to maintain the original semantics and relationship of the data in the calculation of the quantized model. Through this quantization factor selection strategy, when there are outliers in the input matrix or weight, the quantization process will naturally tend to protect the important features with larger absolute values ​​in the data set. Therefore, in the face of input data disturbance or noise, the model may be more robust. Since the adjustment is based on the maximum feature of the row or column, the dynamic range of the target value can be fully utilized during quantization. This means that even in low-bit width representation, the available numerical range can be effectively used to represent data. The quantization factors determined in this way can often be efficiently implemented in hardware, especially for matrix multiplication accelerators, where larger values ​​can often better focus resources for optimization and acceleration.

[0034] For example, assume the matrix is ​​as follows:

[0035] For the first row, the element with the largest absolute value is 0.3. Therefore, each element in this row will use 0.3 as the quantization factor. For the second row, it is 0.6, and for the third row, it is 0.4. By selecting quantization factors in this way, each row of data is quantized accordingly, which can better preserve the feature information in the row and improve the overall efficiency and robustness of the model. In short, this strategy of adaptively selecting quantization factors can significantly enhance the quantized model to achieve a good balance between computational efficiency, storage savings, and performance stability.

[0036] In the above steps, the input matrix and / or model weight in the first model are vector-level quantized by the calculated quantization factor to obtain the second model, which can be implemented as follows: Each eigenvalue in each row or column of the input matrix and / or model weight is divided by the corresponding quantization factor, and the division result of each eigenvalue is multiplied by the preset parameter to obtain a mapping vector corresponding to each eigenvalue; based on the mapping vector corresponding to each eigenvalue, the corresponding input matrix and / or model weight in the second model is constructed.

[0037] In this quantization method, vector-level quantization can be effectively performed to build an optimized model by first calculating the quantization factor, then dividing each eigenvalue of each row or column of the input matrix and / or model weights by the corresponding quantization factor, and then multiplying it by a preset parameter. By dividing by the corresponding quantization factor, the proportional relationship between each feature and the largest eigenvalue is preserved. After multiplying with the preset parameter, all eigenvalues ​​are remapped to a new range, maintaining the relative strength and relationship between features.

[0038] This vector-level quantization method can greatly simplify the computational complexity in actual calculations by simplifying the numerical range (through a combination of division and multiplication). This significantly improves the computing efficiency, especially in hardware acceleration, because the numerical range of the processed data becomes easier to manage. The mapping vector obtained after division and multiplication usually has a smaller numerical range and bit width requirement, which can reduce the memory required for the model to be stored while still maintaining the integrity of important information. The introduction of preset parameters makes the adjustment of the quantization level more flexible. This adjustability allows for optimization of performance and resource usage in different applications and hardware environments. Through this quantization process, the model's acceptance and adaptability to data features are enhanced, and it can maintain a certain robustness when processing inputs with large variability.

[0039] For example, assume that the weight matrix is ​​as follows:

[0040] Assume that for the first row, the quantization factor is 0.3, for the second row, it is 0.6, and for the third row, it is 0.4, and the preset parameter is set to 10. Specifically, in the first step, for the first row [0.2, -0.1, 0.3], divide it by the quantization factor 0.3, and the result is [0.67, -0.33, 1.0]. Then multiply it by the preset parameter 10 to get the mapping vector [6.7, -3.3, 10]. In the second step, do the same for the other rows. The second row [-0.4, 0.6, -0.5] is quantized by 0.6 and multiplied by 10 to get [-6.67, 10, -8.33]. The third row [0.1, -0.2, 0.4] is quantized by 0.4 and multiplied by 10 to get [2.5, -5, 10].

[0041] The mapped matrix is ​​finally constructed in the above manner to form the second model. Through this step, not only a concise and efficient second model is constructed, but also its efficiency and reliability in application are ensured through a flexible quantization strategy.

[0042] In the embodiment of the present application, the strategy of dynamically adjusting the quantization factor optimizes the selection of the quantization factor according to the distribution characteristics of the input data and the weight matrix. This strategy can be differentiated in different network layers of the deep learning model or in each data batch, thereby improving the accuracy of the quantized model.

[0043] Specifically, the input data and weight distribution characteristics of each layer are different. By analyzing these characteristics, the appropriate quantization factor is dynamically selected instead of using a fixed quantization factor. This dynamic adjustment can better adapt to data changes and improve quantization accuracy. Different network layers have different sensitivities to quantization. Some layers, such as convolutional layers, may have higher quantization sensitivity, while fully connected layers may be relatively low. Therefore, different quantization strategies can be used for different layers. For example, when quantizing a convolutional layer, a finer quantization level may be required, while a more aggressive quantization strategy can be selected for a fully connected layer.

[0044] Set variable quantization thresholds based on the characteristics and importance of each layer. For example, a lower quantization threshold can be selected for key layers to retain more details, while the threshold can be increased for secondary layers to achieve better compression. Customize quantization strategies for specific characteristics of different network layers. For example, the most appropriate quantization factor and strategy can be selected based on gradient information, activation distribution, or sparsity of important features. This optimization can reduce unnecessary precision loss. Adjust the quantization strategy in real time for differences in data distribution within a batch, especially for input data that changes dramatically in real time, which can improve overall processing accuracy.

[0045] Exemplarily, when dynamically selecting the quantization factor, it can be adaptively adjusted based on the distribution of the activation function output, even in different input batches, to avoid precision loss when the feature map changes greatly. Due to the huge number of parameters and the different importance of specific parameters, the important part of the parameters of a specific layer can be selected to use a finer quantization level, while the rest can be quantized more coarsely. In different data batches, if the standard deviation and mean of the data are detected to have changed significantly, the quantization factor can be adjusted in real time to adapt to the new data statistical characteristics.

[0046] This dynamic adjustment and differentiated processing strategy can not only significantly improve the performance of the quantized model, but also minimize the loss of accuracy while achieving storage and computing compression. This allows deep learning models to maintain efficient operation and respond quickly in resource-constrained environments.

[0047] Further optionally, after adaptively calculating the quantization factors corresponding to the input matrix and / or model weights in the first model, the quantization sensitivity corresponding to each row or column of the input matrix and / or model weights can also be obtained. The quantization sensitivity is obtained based on the identification of the quantization sensing layer added during the model training process. The quantization sensing layer is constructed based on the data distribution law of each model structure in different types of neural network models. Furthermore, based on the quantization sensitivity, the sensitivity levels of each row or column of the input matrix and / or model weights are divided. Finally, according to the sensitivity level of each row or column, the quantization factors corresponding to the input matrix and / or model weights are dynamically adjusted.

[0048] In a further optimization step, the quantization factor can be finely tuned by introducing a quantization sensing layer to obtain the quantization sensitivity of the input matrix and / or model weights. The quantization sensing layer is a special layer added during the model training process to identify and evaluate the sensitivity of different parts of the model to quantization. It uses the data distribution law and gradient information of each layer to evaluate the impact of different rows or columns on the overall performance of the model during the quantization process. During the training process, the quantization sensing layer calculates the quantization sensitivity of each row or column by monitoring and recording the changes of each feature and the impact on the loss function. These sensitivities can reflect the relative importance of certain features to the accuracy and performance of the model after quantization. Based on the calculated quantization sensitivity, each row or column in the model can be divided into different sensitivity levels. It is usually divided into three levels: high, medium, and low. A row or column with high sensitivity means that accurate quantization is critical to the performance of the model, and vice versa. The quantization factor of each row or column is dynamically adjusted according to the sensitivity level. For parts with high sensitivity, a finer quantization factor is selected to retain key information. For parts with medium or low sensitivity, a coarser quantization factor can be used for a wider range of compression.

[0049] For example, assume a neural network model in which the weight matrix of a convolutional layer is subjected to quantization-sensitive analysis. Some convolution kernels are very sensitive to quantization when detecting important features such as edges. After identifying these convolution kernels (i.e., specific rows or columns corresponding to each convolution), a smaller range of quantization factors are applied to them to reduce quantization errors. For most convolution kernels for conventional feature detection, a medium range of quantization factors can be used, which not only maintains a certain compression efficiency but also retains sufficient performance. For parts with more edges or redundant features, a larger range of quantization factors is used to significantly improve the compression rate and computational efficiency without significantly affecting the overall model accuracy.

[0050] This strategy ensures that important information is preserved while achieving high compression efficiency. By dynamically adjusting the quantization factor based on quantization sensitivity, the model can show good performance stability and less accuracy loss in various application scenarios. This method is particularly suitable for resource-constrained environments such as mobile devices and embedded systems.

[0051] As an optional embodiment, in 103, first, by setting a threshold, the abnormal feature values ​​in the second model are screened to obtain the first decomposition model. The model structure of the first decomposition model is consistent with the model structure of the second model. Furthermore, in 103, the non-abnormal feature values ​​in the second model that have not been screened are stored in the second decomposition model that has the same model structure as the first decomposition model.

[0052] In this optional embodiment, first, abnormal eigenvalues ​​are screened out by setting a threshold, and a first decomposition model is constructed. Then, the non-abnormal eigenvalues ​​in the second model are stored in a second decomposition model with the same model structure. By screening abnormal eigenvalues ​​(such as those outliers that are too large or too small), the negative impact of noise and abnormal values ​​on model performance can be reduced. This helps to improve the stability and robustness of the model under different input conditions. The screened abnormal eigenvalues ​​are removed so that subsequent calculations are focused on more representative data. This can reduce unnecessary calculations, improve execution speed, and reduce storage requirements. Saving and using non-abnormal eigenvalues ​​can maintain the original structure and most of the precise information of the model, ensuring that the model has a high accuracy performance even after decomposition. This can retain key feature information to the greatest extent and improve the final reasoning ability. Because the structure of the decomposed model is the same as the original model, the verification and debugging process is simpler. This method allows potential problems or bottlenecks to be quickly detected and eliminated after the model is decomposed and compressed. The independent processing of abnormal features allows the focus to be placed on the eigenvalues ​​that need to be adjusted when performing model updates or adjustments in the future, without having to fully update all eigenvalue information of the model. This improves the efficiency of subsequent model improvements. Separating abnormal features from normal features helps to better understand the decision-making process of the model when processing certain inputs. This can help developers analyze model performance more clearly, especially in certain scenarios with strong feature dependencies.

[0053] This processing mechanism of eigenvalue screening and decomposition can more effectively manage model features, improve overall model performance, and provide an infrastructure that is easy to maintain and improve.

[0054] For example, the threshold is set to 6. This means that all feature values ​​greater than or less than this threshold are considered abnormal. This method is easy to implement, does not require complex calculations and adjustments, and is very suitable for data sets with known and relatively consistent data distribution.

[0055] In some applications, a fixed threshold can quickly filter out obvious abnormal feature values. For example, in a specific data domain after normalization or standardization, 6 represents an outlier that is far from the mean.

[0056] Exemplarily, an outlier detection algorithm can also be used to dynamically select a threshold based on a statistical distribution instead of fixing it to 6, so that the real outliers can be better captured and the computational efficiency and accuracy of the FP16 branch can be improved. Specifically, using an outlier detection algorithm, such as a statistical distribution (such as mean and standard deviation), the threshold can be dynamically adjusted according to the actual distribution of the data. This is more effective for situations where the data distribution is uneven or variable, and can better capture the real outliers. It can be adjusted according to the actual situation of the data. For example, the use of the three sigma principle (3σ principle) can automatically select a suitable threshold, so as to more accurately capture anomalies without being restricted by a single fixed value. More accurate anomaly detection is conducive to reducing false positives and false negatives, which is particularly important in the floating point 16-bit (FP16) computing precision environment. By dynamically adjusting the threshold, it is possible to ensure that the model accuracy is maintained while improving the computational efficiency.

[0057] For example, suppose a neural network model processes normalized input data in a range of approximately 0 to 10. Select 6 as the threshold, mark some extreme values ​​as anomalies and perform special processing, thereby reducing the interference of these values ​​in the calculation process and simplifying the calculation process. In this model, if the input data distribution may change over time or scenario, by calculating the mean and standard deviation in real time, the system can find that the distribution center of the data has shifted over a period of time, and thus automatically adjust the threshold of anomaly detection (for example, from 6 to 8 or 5), so that the detection mechanism is more adapted to the current data characteristics and maintains high efficiency and accuracy of the operation.

[0058] Through the above method, it is possible to simplify the computational complexity while maintaining a high model accuracy and adapt to different data distribution characteristics, so it has broad application prospects and flexibility.

[0059] As an optional embodiment, in 104, first, the first decomposition model is converted to high precision using a first preset precision to obtain a first optimization result. Further optionally, the abnormal eigenvalues ​​in the first decomposition model are quantized according to a preset floating point format to obtain the first optimization result. Then, the second decomposition model is converted to low precision using a second preset precision to obtain a second optimization result. The processing precision of the second preset precision is lower than the processing precision of the second preset precision. Further optionally, the non-abnormal eigenvalues ​​in the second decomposition model are quantized using an 8-bit integer format to obtain the second optimization result. Finally, the first optimization result and the second optimization result are merged into the third model.

[0060] In this embodiment, the decomposition model is converted to high precision and low precision respectively, combined with quantization processing, and finally fused to obtain a third model. The first decomposition model adopts a high-precision conversion strategy (such as FP32), especially the floating-point format quantization processing of abnormal eigenvalues, which can ensure that the model maintains high precision when processing key and sensitive information, thereby maximizing the recognition ability and accuracy of the model. High-precision processing of abnormal eigenvalues ​​can reduce the information loss caused by quantization. This is particularly important for detecting key patterns or processing complex data, and helps to improve the robustness of the model to inputs in boundary conditions.

[0061] The second decomposition model is converted and quantized using low precision (such as FP16 or 8-bit integer format), which can effectively reduce the computational burden and improve the processing speed, making it suitable for real-time or resource-constrained application scenarios. For example, 8-bit integer quantization can significantly reduce the model size and reduce computational complexity. The use of low-precision numerical representation reduces the memory footprint of the model and significantly reduces the need for storage and data transmission. At the same time, low-precision calculations can usually reduce power consumption, which is especially important for mobile devices and embedded systems.

[0062] Finally, the high-precision and low-precision optimization results are combined to build a third model, which can reduce some computing and storage requirements while maintaining overall performance. Outliers are processed with high precision to ensure accuracy, while non-outliers are processed with low precision to optimize performance, achieving a balance. The fusion model can respond quickly to a variety of data and application scenarios, maintaining good universality. The precision optimization of different parts reaches a dynamic balance, and the model can maintain excellent performance in different scenarios.

[0063] For example, in a convolutional neural network, the first decomposition model, such as the abnormal feature values ​​of the important convolution layer, is processed with FP32 precision to ensure accurate capture of details. The second decomposition model, such as the fully connected layer, uses 8-bit integer quantization to improve the overall reasoning speed and efficiency. Finally, the results of these different precision processing are fused to retain the high accuracy and fast response characteristics in image recognition tasks.

[0064] This piecewise precision optimization method fully utilizes the advantages of high- and low-precision processing, adapts to diverse application needs, and becomes an important means of model compression and acceleration.

[0065] As an optional embodiment, the first decomposition model is converted to high precision using a first preset precision, and before obtaining the first optimization result, the abnormal distribution characteristics corresponding to the abnormal feature values ​​in the first decomposition model can also be predicted through an abnormal perception model. Furthermore, after obtaining the abnormal distribution characteristics, a preset floating point format matching the abnormal distribution characteristics is determined to dynamically construct a variable quantization factor for the first decomposition model. The floating point format of the variable quantization factor changes with the change of the abnormal distribution characteristics.

[0066] The abnormality perception model is obtained by training abnormal feature values ​​corresponding to different types of neural network models. Further optionally, the abnormality perception model is a multi-branch structure, and each branch structure is used to identify the distribution law of abnormal feature values ​​corresponding to one or more types.

[0067] The anomaly perception model is a machine learning model used to detect and analyze the abnormal distribution of feature values ​​in the model. The anomaly perception model can identify possible abnormal feature values ​​based on the characteristics of the input data and predict the distribution characteristics of these abnormal feature values. This model is based on the statistical characteristics of existing data and machine learning training, and can be implemented in the following ways: The anomaly perception model can identify specific patterns of abnormal feature values ​​by training a large amount of historical data. For example, different types of neural network models such as convolutional neural networks (CNN), recurrent neural networks (RNN) or fully connected networks (MLP) are used to obtain abnormal patterns in specific fields. The multi-branch structure means that the model can handle multiple abnormal feature values ​​at the same time. Each branch can focus on different types of anomalies, such as spelling errors in text data, noise or missing information in image data, etc. This design enables the model to have higher generalization ability and can handle diverse abnormal patterns. The processing strategy of the model runtime is dynamically adjusted by analyzing the abnormal distribution characteristics. For example, according to the distribution law of the discreteness and skewness of the abnormal feature values, a matching floating point format (such as FP32, FP16 or others) is selected, and then the setting of the variable quantization factor is adjusted to optimize the processing.

[0068] In this specific embodiment, the anomaly perception model is used as a pre-step of the high-precision conversion and quantization process: the anomaly perception model is used to identify abnormal eigenvalues ​​in the first decomposition model and predict their distribution characteristics. This step can be performed by analyzing the statistical distribution, mean, variance and other characteristics of the eigenvalues. According to the identified abnormal distribution characteristics, a suitable floating-point format is selected (for example, eigenvalues ​​with large entropy are represented with higher precision) and applied to subsequent high-precision conversions. This dynamic selection makes model processing more targeted, ensuring performance while avoiding excessive waste of resources. Combined with the distribution law of abnormal eigenvalues, variable quantization factors suitable for these features are constructed, that is, the calculation and storage are optimized by adjusting the floating-point format. The flexibility of the variable quantization factor allows suitable calculation accuracy and storage requirements to be provided under various abnormal feature distributions.

[0069] Through this process, the embodiment can more effectively handle outliers in the model, making the overall optimization result more reliable and efficient. In general, the use of the anomaly-aware model significantly enhances the adaptability and robustness of the model under different data distribution conditions.

[0070] As an optional embodiment, in 104, the first optimization result and the second optimization result are merged into the third model, which can be implemented as follows: according to their respective corresponding quantization processing accuracy, the first optimization result and the second optimization result are dequantized to obtain corresponding first dequantization result and second dequantization result; and the first dequantization result and the second dequantization result are merged to obtain the third model.

[0071] The following describes, in conjunction with an embodiment, a process of fusing the first optimization result and the second optimization result into a third model through dequantization and merging operations.

[0072] Dequantization is the process of restoring quantized data to a format close to its original floating point format. This unifies the numerical format before merging results of different precision or different numerical formats. For the first optimized result that has been quantized (using a higher precision floating point format, such as FP32), dequantization is performed according to the factors and offsets (scale and zero-point) used when it was quantized. Restore to the original or close to the original high-precision floating point representation. Perform a similar dequantization operation on the second optimized result (using a lower precision floating point or integer format, such as FP16 or 8-bit integer). Use the same conversion factor and offset values ​​as before quantization to restore it to a floating point representation.

[0073] Once both the first and second results have been dequantized, it is now necessary to combine them into a unified model. Ensure that the two dequantized results are on the same data scale, and if necessary, further scale them (e.g., multiply by a uniform scaling factor) to facilitate element-wise addition. Ensure that the two results are compatible in structure and dimensions. This may require padding, clipping, or concatenation of the results along certain dimensions. Perform an additive or weighted combination of the two dequantized results (e.g., via a linear combination). This form of merging may adjust weights depending on the specific model architecture and fusion strategy. When dealing with deep learning models, this is often achieved by summing up network parameters or concatenating neuron layers.

[0074] Through the above merging steps, a fused model (third model) is obtained, which contains a comprehensive expression of information from both high-precision and low-precision parts. In practical applications, different weights are assigned to different parts according to actual needs to control the impact of each part on the final model behavior. Use the validation set to fine-tune and calibrate the third model to ensure that the overall performance of the fused model reaches the expected goal. After generating the third model, it may also be necessary to verify the model performance through a certain training or adjustment process, and make necessary fine-tuning based on the verification results to ensure that the third model performs well in different tasks.

[0075] Through the above steps, the optimization results obtained by two different precision processing can be merged into a complete and efficient third model. The advantage of this method is that it can fully utilize the efficiency and accuracy advantages of different processing precisions to create an optimized model system.

[0076] As an optional embodiment, in the above steps, after the first optimization result and the second optimization result are dequantized according to their respective corresponding quantization processing accuracy to obtain the corresponding first dequantization result and the second dequantization result, the first dequantization result and the second dequantization result can also be subjected to unidirectional and bidirectional bias correction to remove or compensate for the systematic error caused by the dequantization process.

[0077] Single and double bias correction is an error compensation technique performed after the dequantization result, which aims to reduce the systematic errors introduced in the quantization and dequantization process and ensure the accuracy of the data and the performance of the model. Before performing bias correction, it is necessary to identify the source of the systematic error. During the quantization process, due to the limited representation range, the data is usually truncated or rounded, resulting in loss of information. During the dequantization process, these rounding errors may accumulate, causing bias.

[0078] Correct in one direction, usually by applying a simple bias correction after dequantization. Compare the mean or median before and after dequantization. Calculate the deviation of the expected value (ideally the original value) from the actual dequantized result. Add the calculated deviation as a constant to all dequantized results to adjust the overall bias.

[0079] For bidirectional bias correction, it involves error compensation in two directions to improve the errors of multiple layers and their interactions: the direction and degree of the error are analyzed, considering both global (overall) and local (stratified) biases. On the basis of overall correction, differentiated adjustments are made according to the specific conditions of each layer or data block to ensure that each component tends to mean correction. The bias values ​​are detected and updated through multiple iterative steps to gradually reduce the bias in all directions. In each iteration, the errors are re-evaluated based on the corrected results and the parameters are adjusted until the convergence condition of a certain error limit is met.

[0080] In specific operations, models such as linear regression or neural networks are used to learn and apply error compensation, which can perform complex corrections for the deviations of specific features. The overall deviation and local deviation coefficients are obtained through statistical analysis, and adjustments are made based on this. The calculated correction amount is applied to the first dequantization result and the second dequantization result to ensure that all data points are closer to the original unquantized data after correction. After correction, the performance of the adjusted model is evaluated through a set of test data. Observe changes in indicators such as accuracy and precision to ensure that the compensation effect is ideal. Through the above correction techniques, the influence of systematic errors in the dequantization process can be effectively eliminated or reduced, and the stability and accuracy of the fused third model in practical applications can be guaranteed.

[0081] In the embodiment of the present application, by adaptively processing different structures in the neural network model and separating outliers and non-outliers for quantization processing, it is helpful to retain the original data information of the model and effectively avoid the decrease in quantization accuracy caused by the complex distribution of outliers, further improve the model quantization accuracy, reduce the space occupied by the model, and improve the model processing efficiency. In addition, using different strategies to process outliers and non-outliers can also save model quantization overhead, improve model performance, and further improve model processing efficiency.

[0082] For example, Large Language Model 8-bit Quantization (LLM.int8) is a quantization strategy designed specifically for large language models. Its core is to significantly reduce the model's storage requirements and speed up inference by reducing the precision of model weights and activation values ​​(converting from common half-floating point or full-floating point precision to 8-bit integer precision), while effectively maintaining the model's original performance and avoiding performance loss caused by model quantization. Obviously, applying the 8-bit quantization strategy in large language models can significantly reduce storage requirements and speed up calculations, especially in core calculations such as matrix multiplication (Matmul).

[0083] For example, in the optimization practice of the large BLOOM model, this optimization strategy is adopted specifically for the matrix multiplication (Matmul), a core calculation relief. Before quantization, the maximum and minimum values ​​of the model weights and activation values ​​are determined in order to calculate the scale factor and zero-point required for quantization. A portion of the training data is used to statistically analyze the distribution characteristics of the weights and activation values ​​through forward propagation to set a suitable range for subsequent quantization. When optimizing large language models using 8-bit quantization, the key steps include calculating the scale factor and zero-point, quantizing and dequantizing data, and compensating for quantization errors during inference. First, the scale factor and zero-point are calculated for each layer or each operation unit (such as a vector) based on the selected range (maximum and minimum values) of the floating-point data. The purpose of this step is to compress the floating-point values ​​with a wide distribution into a representable 8-bit integer range. Next, the quantization formula is applied to convert the floating-point numbers to 8-bit integers. Through this formula, each floating-point value is mapped to an integer value, which significantly reduces memory usage and improves computational efficiency. In the model inference phase, these quantized integers need to be decoded back to approximate floating-point numbers for accurate calculations. This reverse process restores the approximate original data accuracy, allowing the model to perform floating-point calculations. In fact, efficient matrix multiplication is the use of specific hardware instruction sets (such as AVX, NEON, or Tensor Cores) to perform fast multiplication and addition operations on quantized values. These instructions greatly improve the computational throughput by making full use of hardware parallel capabilities. Finally, for the errors introduced in the quantization process, calibration data is combined for error compensation. Through a small amount of floating-point operations, the deviation introduced by quantization can be reduced. In addition, the use of single and bidirectional bias correction methods can further reduce systematic errors and ensure that the quantized model is close to the original model in performance. This strategy effectively improves the operating efficiency and resource utilization of the model while retaining the original accuracy.

[0084] Specifically, the obtained input (hidden state) and model weight (Weight) are processed by vector wise quantization technology, and different quantization factors are used for multiplication calculations for different rows and columns to ensure that the values ​​can be mapped to the valid range of [-128, 127]. For the input matrix, first find the maximum absolute value of each row, then divide each element by this and multiply it by 127 to convert it to an 8-bit integer format (Int8). The model weight matrix is ​​processed in a similar way, but the operation is performed on each column. The specific conversion process uses the following formula:

[0085] The above formula shows two matrices and The relationship between is a matrix in floating point format, is a matrix in integer format. The subscript f16 indicates the floating point format, that is, the half-precision floating point number FP16, which ranges from -65504 to +65504. The subscript int8 indicates the integer format, which is an unsigned integer with a range of 0 to 255. In the above formula, Q() represents the quantization operation, S is the scaling factor, A and B represent the two matrices involved in the multiplication operation, respectively. Representation Matrix FP16 form, , Then they represent the matrices Decompose the two sub-matrices obtained by transformation, Represents the int32 format of matrix C, , Represents the int8 format of matrices A and B respectively, , Then it represents the FP16 format of matrices A and B.

[0086] After quantization, a custom threshold is used to identify and handle outliers, and the matrix is ​​decomposed into two parts. For example, a threshold of 6 is used for outlier extraction to fully restore the inference performance of the model. Subsequently, mixed precision decomposition is performed to separate the outliers (i.e., outlier features) in the matrix and calculate them using floating point half precision (FP16) to avoid excessive impact on quantization accuracy. For the remaining non-outlier parts, 8-bit integers (INT8) are used for matrix multiplication to further accelerate the inference process. Finally, the outlier results using floating point half precision (FP16) and the results of non-outlier dequantization are merged to obtain the output value.

[0087] For example, taking vector-wise quantization as an example, first, find the maximum absolute value. Assume that the input is a matrix X with a size of Cin×H×W. Based on this, calculate the maximum absolute value in the matrix X, recorded as cmax. Then, implement the quantization step. That is, use the maximum absolute value cmax found to quantize the matrix X to obtain the quantized matrix Xq. The element range of the matrix Xq is limited to [-127, 127]. Next, mixed precision decomposition. Here, the quantized matrix Xq is decomposed into two parts: the first part O (X, 8) is the part containing normal values, with a size of Cin×H×W. The second part O (X, 16) contains the part of outliers, with a size of Cin×H×W. Then, FFPS Matrix. For the outlier part O (X, 16), use the fixed-point and floating-point quantization scheme (FFPS) method to process it and obtain a new matrix Xffps. That is, use a higher number of bits (such as 16-bit) to represent the matrix multiplication. Then, perform the Combine step. Finally, add the normal value part O(X,8) and the abnormal value part to get the final result Out16.

[0088] In practical applications, the performance of the 8-bit quantization strategy for large language models is optimized by using assembly instructions based on high-performance chip hardware simulator (Veloce). Figure 2In the figure, mean zero-shot accuracy refers to the performance indicator of the average correct rate of the model in the zero-shot learning task. It measures the accuracy of the model in classifying on the new category test set. The horizontal axis (X-axis) represents the change in the amount of parameter data, which gradually increases from left to right. The vertical axis (Y-axis) is used to show the average zero-shot accuracy, that is, how the model performs on new categories. Higher values ​​indicate better performance. The quantization method icon (Method), that is, different methods here refer to several different quantization schemes. Specifically, the quantization method proposed in this application, the corresponding icon is LLM.intB(), and the corresponding curve of the icon is used to represent the result curve function obtained by the above quantization process. The 8-bit quantization method, the corresponding icon is 8-bit baseline, and the corresponding curve of the icon is used to indicate the use of the standard 8-bit quantization method as the baseline. Similarly, the 16-bit quantization method, the corresponding icon is 16-bit baseline, and the corresponding curve of the icon is used to indicate the use of the 16-bit quantization method as another baseline. The outlier feature value (Emergence of outlier features) is used to indicate the location of the outlier feature value that appears as the parameter changes. In the above example, it means that the 8-bit baseline cannot maintain the same accuracy as the original 16-bit after the amount of parameter data increases.

[0089] refer to Figure 2 As shown in the figure, the test results based on high-performance chip hardware simulators show that the performance of the optimized floodlight model at different parameter values ​​is very similar to that of the initial model, and the accuracy rate steadily increases with the increase in the number of parameters. In contrast, the accuracy of the traditional quantization method drops sharply after reaching a certain number of parameters. This comparison fully proves that the optimization strategy not only effectively saves computing resources in large-scale language models, but also ensures the excellent performance of the model and significantly reduces the quantization error.

[0090] In the embodiment of the present application, an 8-bit quantization strategy for a large language model is applied to a large flood model implemented on assembly instructions, which can significantly reduce the storage requirements of the model and speed up the inference speed while maintaining the original performance of the model, effectively reducing the quantization error, and providing strong support for running larger parameter models on high-performance chip hardware simulator hardware.

[0091] In another embodiment of the present application, a model optimization device is also provided. Figure 3 The device comprises the following units: An acquisition unit is configured to acquire a first model to be optimized; the first model is a neural network model that meets a preset optimization condition; the preset optimization condition is determined based on the type of the first model; A quantization unit is configured to perform adaptive quantization processing on the structure to be optimized in the first model to obtain a second model; wherein different quantization factors are used for different structures to be optimized in the adaptive quantization processing: the quantization factors are adaptively calculated from the eigenvalues ​​contained in the structure to be optimized; a decomposition unit configured to perform mixed precision decomposition on the second model to obtain a first decomposition model including abnormal eigenvalues ​​and a second decomposition model including non-abnormal eigenvalues; The optimization unit is configured to optimize the first decomposition model and the second decomposition model respectively according to a preset strategy, and merge the optimization results into a third model that is finally output.

[0092] Further optionally, the acquisition unit acquires the working status information of each node from the on-chip network and is configured to: For a first node in the network on chip, receiving a first working state signal of the first node; the first working state signal is associated with a functional attribute of the first node itself; Converting the first working state signal into first working state information in a single-bit form; The first working state information is used to indicate the working state type of the first node.

[0093] Further optionally, each working status type of the first node corresponds one-to-one to a plurality of data bits in the first working status information.

[0094] Further optionally, the acquiring unit converts the first working state signal into the first working state information in a single-bit form, and is configured to: Parsing the first working state signal to obtain a target working state type currently located at the first node; A target data bit corresponding to a target working state type is acquired, and the target data bit is set as a corresponding working state identifier to obtain the first working state information.

[0095] Further optionally, the multiple nodes in the on-chip network include at least: functional modules for implementing different logical functions, and transmission nodes between the multiple functional modules.

[0096] Further optionally, the statistical unit, based on the working status information of each node, uses a counter module to classify and count each node to obtain the working status statistical information of each node, and is configured as follows: Using a counter module to receive the first working state information in a single-bit form; According to the first working state information, a target working state type to be updated in the first node is determined, and counting processing is performed on the target working state type to obtain working state statistical information of the first node.

[0097] Further optionally, the counter module includes at least an adder and a trigger; and the statistical unit determines the target working state type to be updated in the first node according to the first working state information, and performs counting processing on the target working state type to obtain the working state statistical information of the first node, and is configured as follows: Splitting the first working state information into a plurality of single-bit data bits, and selecting a target single-bit data bit to be updated in the first node according to a current value in each single-bit data bit; wherein the target single-bit data bit is set to a valid value corresponding to the target working state type; Determine a target trigger corresponding to the target single-bit data bit; wherein each single-bit data bit corresponds to a trigger in a counter module; The target trigger is started, and the count value stored in the target trigger is accumulated through an adder to obtain an updated target count value.

[0098] Further optionally, the counter module further includes a timing module; a storage unit, which stores the working status statistical information of each node in the side storage module respectively, and is configured as follows: According to the transmission cycle set in the timing module, the target count value stored in the target trigger is sent to the side storage module; The target count value is stored in the storage space corresponding to the first node through the side storage module.

[0099] Further optionally, after the statistical unit starts the target trigger and accumulates the count value stored in the target trigger through the adder to obtain the updated target count value, it is further configured to: The timing module is used to count the number of clock cycles of each trigger updating the count value within the target monitoring period; Based on the number of clock cycles of the updated count value, the monitoring indicators corresponding to different working states in each node are obtained.

[0100] Further optionally, the monitoring indicators include: the ratio of the number of clock cycles in which different working states of each node occur within the target monitoring period to the total number of clock cycles within the target monitoring period; the ratio is used for at least one of load balancing, performance bottlenecks, traffic transmission characteristics, state evaluation parameters, change trend analysis, resource allocation, and long / short cycle change trends within the target monitoring period.

[0101] Further optionally, the statistical unit, before using the timing module to count the number of clock cycles for each trigger to update the count value within the target monitoring period, is further configured to: Adjust the duration of the target monitoring period to change the monitoring sparsity of each node within the target monitoring period; Aggregate the status data of each node within the target monitoring period to optimize the number of clock cycles of the count value.

[0102] Further optionally, the number of storage space addresses of the side storage module is greater than or equal to the number of signal lines of the access counter module, and the signal lines are used to carry single-bit data bits of the working status information.

[0103] Further optionally, the bit width of the data stored in the side storage module is greater than or equal to the bit width of the trigger used to store the value in the counter module.

[0104] Further optionally, the number of triggers provided in the counter module is the same as the number of signal lines connected to the counter module.

[0105] The system can implement various steps in the above method embodiment, which will not be expanded here.

[0106] In the embodiment of the present application, a model optimization device is used to adaptively process different structures in the neural network model and separate outliers and non-outliers to perform quantization processing, which not only helps to retain the original data information of the model, but also can effectively avoid the decrease in quantization accuracy caused by the complex distribution of outliers, further improve the model quantization accuracy, reduce the space occupied by the model, and improve the model processing efficiency. In addition, the model optimization device uses different strategies to process outliers and non-outliers, which can also save model quantization overhead, improve model performance, and further improve model processing efficiency.

[0107] See also Figure 4 , Figure 4 This is a schematic diagram of an electronic device provided in an embodiment of the present application. Figure 4As shown, an embodiment of the present application provides an electronic device 500, including a memory 510, a processor 520 and a computer program 511 stored in the memory 510 and executable on the processor 520, wherein the processor 520 implements the following steps when executing the computer program 511: obtaining a first model to be optimized; the first model is a neural network model that meets preset optimization conditions; the preset optimization conditions are determined based on the type to which the first model belongs; the structure to be optimized in the first model is adaptively quantized to obtain a second model; wherein different quantization factors are used for different structures to be optimized in the adaptive quantization process: the quantization factors are adaptively calculated from the eigenvalues ​​contained in the structure to be optimized; mixed precision decomposition is performed on the second model to obtain a first decomposition model containing abnormal eigenvalues, and a second decomposition model containing non-abnormal eigenvalues; the first decomposition model and the second decomposition model are optimized respectively according to a preset strategy, and the optimization results are merged into a third model as the final output.

[0108] See also Figure 5 , Figure 5 A schematic diagram of an embodiment of a computer-readable storage medium provided in an embodiment of the present application. Figure 5 As shown, this embodiment provides a computer-readable storage medium 600, on which a computer program 611 is stored, and when the computer program 611 is executed by a processor, the following steps are implemented: obtaining a first model to be optimized; the first model is a neural network model that meets preset optimization conditions; the preset optimization conditions are determined based on the type to which the first model belongs; the structure to be optimized in the first model is adaptively quantized to obtain a second model; wherein different quantization factors are used for different structures to be optimized in the adaptive quantization process: the quantization factors are adaptively calculated from the eigenvalues ​​contained in the structure to be optimized; performing mixed precision decomposition on the second model to obtain a first decomposition model containing abnormal eigenvalues, and a second decomposition model containing non-abnormal eigenvalues; optimizing the first decomposition model and the second decomposition model according to a preset strategy, and merging the optimization results into a third model as a final output.

[0109] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0110] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0111] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0112] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A model optimization method, characterized in that: The method comprises: Acquire a first model to be optimized; the first model is a neural network model that meets a preset optimization condition; the preset optimization condition is determined based on the type of the first model; The structure to be optimized in the first model is subjected to adaptive quantization processing to obtain a second model; wherein different quantization factors are used for different structures to be optimized in the adaptive quantization processing: the quantization factors are obtained by adaptively calculating the eigenvalues ​​contained in the structure to be optimized; Performing mixed precision decomposition on the second model to obtain a first decomposition model including abnormal eigenvalues ​​and a second decomposition model including non-abnormal eigenvalues; The first decomposition model and the second decomposition model are optimized respectively according to a preset strategy, and the optimization results are integrated into a third model for final output.

2. The model optimization method according to claim 1, characterized in that: The structure to be optimized includes at least: the input matrix and / or model weights in the first model; The step of performing adaptive quantization processing on the structure to be optimized in the first model to obtain a second model comprises: Adaptively calculating the quantization factors corresponding to the input matrix and / or model weights in the first model; the quantization factors are calculated based on the absolute values ​​in the rows or columns to which the structure to be processed in the first model belongs; The second model is obtained by performing vector-level quantization on the input matrix and / or model weight in the first model using the calculated quantization factor.

3. The model optimization method according to claim 2, characterized in that: The adaptively calculating the quantization factors corresponding to the input matrix and / or the model weights in the first model respectively includes: For the input matrix and / or model weight in the first model, select the target eigenvalue with the largest absolute value in each row or column of the input matrix and / or model weight, and use the absolute value of the target eigenvalue as the quantization factor; The quantization factor obtained by calculation is used to perform vector-level quantization on the input matrix and / or model weight in the first model to obtain the second model, including: Divide each eigenvalue in each row or column of the input matrix and / or model weight by the corresponding quantization factor, and multiply the division result of each eigenvalue by a preset parameter to obtain a mapping vector corresponding to each eigenvalue; Based on the mapping vector corresponding to each eigenvalue, a corresponding input matrix and / or model weight in the second model is constructed.

4. The model optimization method according to claim 2, characterized in that: After the adaptive calculation of the quantization factors corresponding to the input matrix and / or the model weights in the first model, the method further includes: Obtaining the quantization sensitivity corresponding to each row or column of the input matrix and / or model weight; wherein the quantization sensitivity is obtained based on the identification of the quantization sensing layer added during the model training process; the quantization sensing layer is constructed based on the data distribution law of each model structure in different types of neural network models; Based on the quantized sensitivity, dividing each row or column of the input matrix and / or the model weight into sensitivity levels; According to the sensitivity level of each row or column, the quantization factors corresponding to the input matrix and / or model weights are dynamically adjusted.

5. The model optimization method according to claim 1, characterized in that: The performing mixed precision decomposition on the second model to obtain a first decomposition model including abnormal eigenvalues ​​and a second decomposition model including non-abnormal eigenvalues ​​includes: By setting a threshold, abnormal feature values ​​in the second model are screened to obtain the first decomposition model; the model structure of the first decomposition model is consistent with the model structure of the second model; The non-abnormal feature values ​​that have not been screened in the second model are stored in the second decomposition model that has the same model structure as the first decomposition model.

6. The model optimization method according to claim 5, characterized in that: The set threshold is 6.

7. The model optimization method according to claim 1, characterized in that: The step of optimizing the first decomposition model and the second decomposition model according to a preset strategy, and fusing the optimization results into a third model that is finally output, includes: Performing high-precision conversion on the first decomposition model using a first preset precision to obtain a first optimization result; The second decomposition model is converted into a low-precision model using a second preset precision to obtain a second optimization result; wherein the processing precision of the second preset precision is lower than the processing precision of the second preset precision; The first optimization result and the second optimization result are merged into the third model.

8. The model optimization method according to claim 7, characterized in that: The step of converting the first decomposition model to a high-precision state using a first preset precision to obtain a first optimization result includes: Quantizing the abnormal characteristic values ​​in the first decomposition model according to a preset floating point number format to obtain the first optimization result; The step of converting the second decomposition model into a low-precision model using a second preset precision to obtain a second optimization result includes: The non-abnormal eigenvalues ​​in the second decomposition model are quantized in an 8-bit integer format to obtain the second optimization result.

9. The model optimization method according to claim 8, characterized in that: Before the first decomposition model is converted to a high precision using a first preset precision to obtain a first optimization result, the method further includes: The abnormal distribution characteristics corresponding to the abnormal feature values ​​in the first decomposition model are predicted by an abnormal perception model; the abnormal perception model is trained based on the abnormal feature values ​​corresponding to different types of neural network models; the abnormal perception model is a multi-branch structure; each branch structure is used to identify the distribution law of abnormal feature values ​​corresponding to one or more types; A preset floating point number format matching the abnormal distribution feature is determined to dynamically construct a variable quantization factor for the first decomposition model; wherein the floating point number format of the variable quantization factor changes with changes in the abnormal distribution feature.

10. The model optimization method according to claim 7, characterized in that: The step of fusing the first optimization result and the second optimization result into the third model includes: According to the respective corresponding quantization processing precisions, the first optimization result and the second optimization result are dequantized to obtain the corresponding first dequantization result and second dequantization result; The first dequantization result and the second dequantization result are combined to obtain the third model.

11. The model optimization method according to claim 10, characterized in that: After performing inverse quantization processing on the first optimization result and the second optimization result according to the respective corresponding quantization processing precisions to obtain the corresponding first inverse quantization result and the second inverse quantization result, the method further includes: Single and double-direction offset correction is performed on the first dequantization result and the second dequantization result to remove or compensate for the systematic error caused by the dequantization process.

12. A model optimization device, characterized in that: The device comprises the following units, wherein: An acquisition unit is configured to acquire a first model to be optimized; the first model is a neural network model that meets a preset optimization condition; the preset optimization condition is determined based on the type of the first model; A quantization unit is configured to perform adaptive quantization processing on the structure to be optimized in the first model to obtain a second model; wherein different quantization factors are used for different structures to be optimized in the adaptive quantization processing: the quantization factors are adaptively calculated from the eigenvalues ​​contained in the structure to be optimized; a decomposition unit configured to perform mixed precision decomposition on the second model to obtain a first decomposition model including abnormal eigenvalues ​​and a second decomposition model including non-abnormal eigenvalues; The optimization unit is configured to optimize the first decomposition model and the second decomposition model respectively according to a preset strategy, and merge the optimization results into a third model that is finally output.

13. An electronic device, characterized in that: include: Memory for storing computer software programs; A processor is used to read and execute the computer software program, thereby implementing the model optimization method described in any one of claims 1-11.

14. A computer-readable storage medium, characterized in that: The storage medium stores a computer software program, and when the computer software program is executed by the processor, the model optimization method according to any one of claims 1 to 11 is implemented.

15. A chip, characterized in that: The chip is loaded with a computer software program and / or a hardware unit, and the computer software program and / or the hardware unit are used to implement the model optimization method as described in any one of claims 1-11.

Images