Emotion sensitivity-based end-side large model mixed precision quantization method and device

Abstract

The application discloses an end-side large model mixed precision quantization method and device based on emotional sensitivity, relates to the field of model parameter quantization, and comprises the following steps: constructing an emotional weighting loss function based on a calibration sample set and an end-side large model to be quantized, and calculating a gradient sensitivity score according to the emotional weighting loss function; applying low-bit quantization disturbance to each to-be-quantized module one by one, and constructing a disturbance model; based on the calibration sample set, generating an undisturbed reference reply and a disturbance reply by using the end-side large model to be quantized and each disturbance model respectively, and calculating the behavior degradation score of each disturbance model; calculating the emotional behavior sensitivity score according to the gradient sensitivity score and the behavior degradation score, and allocating corresponding quantization bit width to each to-be-quantized module in combination with a preset average bit budget; and performing mixed precision quantization according to the quantization bit width to obtain the end-side large model after mixed precision quantization. The application solves the problem that end-side low-bit quantization leads to emotional accompanying behavior degradation.

Description

Technical Field

[0001] This invention relates to the field of large model quantization, specifically to an edge-side large model hybrid precision quantization method and apparatus based on emotion sensitivity. Background Technology

[0002] With the development of edge devices such as smart speakers, home control systems, mobile terminals, and companion robots, large-scale models are gradually being deployed from the cloud to the edge and terminal sides. Their input can include user text, text processed by the front end of voice interaction, or multi-turn dialogue context. The output is a response statement with emotional responsiveness, which can be used for tasks such as emotion understanding, companion-style question answering, and voice interaction response generation. Therefore, the efficient deployment of large-scale models on the edge is a crucial technology in smart home, companion robot, and edge intelligence service systems.

[0003] In edge deployments of large models, existing methods often employ unified low-bit quantization or hybrid precision quantization based on general error metrics to reduce model storage overhead and inference resource consumption. However, in edge-based emotional interaction scenarios, existing quantization methods still suffer from a mismatch between the degradation of emotional companionship behavior and quantization bit allocation. Specifically, low-bit quantization may lead to longer model responses, an increased number of numbering lists, decreased voice broadcast adaptability, insufficient negative emotional responses, and reduced output format stability; while higher-bit quantization, although better preserving model behavior, increases edge storage and inference overhead. Existing general quantization methods typically focus more on weight reconstruction error, activation distribution, perplexity, or general task accuracy, making it difficult to establish a correspondence between low-bit quantization perturbations of the module to be quantized and the degradation of emotional interaction behavior, and failing to directly reflect the behavioral preservation requirements such as voice broadcast adaptability, emotional response matching degree, and format stability. Therefore, there is an urgent need for an edge-based hybrid precision quantization method for large models that allocates quantization bits based on emotional sensitivity and the degradation of output behavior after low-bit quantization perturbations at each layer, in order to improve the ability to preserve emotional interaction behavior under limited edge resources. Summary of the Invention

[0004] The purpose of this application is to propose a hybrid precision quantization method and device for edge-side large models based on emotion sensitivity to address the aforementioned technical problems.

[0005] In a first aspect, the present invention provides a hybrid precision quantization method for edge-side large models based on emotion sensitivity, comprising the following steps:

[0006] The process involves obtaining a large edge model to be quantized and a calibration sample set containing sentiment labels. The large edge model includes multiple modules to be quantized. A sentiment-weighted loss function is constructed based on the calibration sample set and the large edge model to be quantized, and the gradient sensitivity score of each module is calculated according to the sentiment-weighted loss function. Low-bit quantization perturbations are applied to each module to be quantized, constructing corresponding perturbation models. Based on the calibration sample set, unperturbed reference responses and corresponding perturbed responses are generated using the large edge model to be quantized and each perturbed model, respectively. The behavior degradation score of each module is calculated based on the difference in sentiment interaction behavior scores between the unperturbed reference responses and the perturbed responses generated by each perturbed model. The sentiment behavior sensitivity score of each module to be quantized is calculated based on the gradient sensitivity score and the behavior degradation score. A corresponding quantization bit width is allocated to each module to be quantized based on the sentiment behavior sensitivity score and a preset average bit budget. Mixed-precision quantization is performed on the large edge model to be quantized according to the quantization bit width to obtain the mixed-precision quantized large edge model.

[0007] The system acquires the input text or the text sequence obtained through speech processing and inputs it into the edge-side large model after mixed precision quantization to obtain the corresponding response statement.

[0008] As a preferred option, the process of constructing the sentiment-weighted loss function is as follows:

[0009] The calibration sample set containing sentiment labels is represented as follows: ;

[0010] in, Indicates the first Input text, Indicates the first The standard reference response for each input text. Indicates the first The sentiment tag corresponding to each input text. Indicates the total number of input texts;

[0011] The sentiment weight of a sample is calculated based on its sentiment label, using the following formula:

[0012] ;

[0013] in, Indicates the first Sentiment weights for each input text. This represents the negative sentiment weighting coefficient. This indicates a pre-defined set of negative emotions. Indicates an indicator function;

[0014] The sentiment-weighted loss function is constructed based on the sample sentiment weights, as shown in the following equation:

[0015] ;

[0016] in, This represents the sentiment-weighted loss function. This represents the large edge model to be quantized. This represents the parameters of the large edge model to be quantized. This represents the task loss function. Indicates the emotion enhancement coefficient. ;

[0017] The calculation process for the gradient sensitivity score of each module to be quantized is as follows:

[0018] Calculate the sentiment-weighted loss function Regarding the first in the large end-side model to be quantized Weight matrix of each module to be quantized The gradient of is calculated, and the first gradient is calculated based on the weight matrix and the corresponding gradient. The gradient sensitivity scores of each module to be quantized are shown in the following formula:

[0019] ;

[0020] in, Indicates the first Gradient sensitivity scores of each module to be quantized. Represents the weight matrix The number of elements in , This indicates the total number of modules to be quantized. This represents the Hadamard product. express Norm.

[0021] As a preferred method, the calculation process for behavioral degradation scores is as follows:

[0022] Set low bit width , for the Weight matrix of each module to be quantized Perform low-bit quantization perturbation to obtain the perturbation weight matrix. The calculation process is as follows:

[0023] ;

[0024] ;

[0025] in, Indicates the first The quantization scaling factor corresponding to each module to be quantized during the low-bit quantization perturbation stage. This represents the floor function. This represents the truncation function. This indicates taking the maximum value;

[0026] The first in the large end-side model to be quantized The weight matrix of each module to be quantized is replaced with... The remaining modules to be quantized maintain reference precision, resulting in the first... The perturbation model corresponding to each module to be quantized ;

[0027] Based on the The input text is used to generate an unperturbed reference response using an unperturbed edge-side large model, as shown in the following equation:

[0028] ;

[0029] Based on the The input text uses the first The perturbation model for each module to be quantized generates a corresponding perturbation response, as shown in the following equation:

[0030] ;

[0031] The first point is calculated based on the difference between the sentiment interaction score of the unperturbed reference response and the sentiment interaction score of the perturbed response. The behavioral degradation score of each module to be quantified is shown in the following formula:

[0032] ;

[0033] in, Indicates no disturbance (see reference reply) In emotional tags The emotional interaction behavior score is expressed as follows: ; Indicates a disturbance response In emotional tags The emotional interaction behavior score is expressed as follows: ; Indicates the first A behavioral indicator function, Indicates the first The weights corresponding to each behavioral indicator This represents the total number of behavioral indicators. Indicates the first The behavioral degradation score of each module to be quantified; the behavioral index function is a function that maps unperturbed reference responses or perturbed responses and their corresponding sentiment tags to behavioral scores by means of rule judgment, classification model, scoring model or manual annotation rules.

[0034] As a preferred approach, the calculation process for the emotional behavior sensitivity score of each module to be quantified is as follows:

[0035] The gradient sensitivity score and behavior degradation score are normalized separately to obtain the normalized gradient sensitivity score and normalized behavior degradation score, as shown in the following formula:

[0036] ;

[0037] ;

[0038] in, Indicates the first Gradient sensitivity scores of each module to be quantized. Indicates the first The behavioral degradation score of each module to be quantified This represents the normalized gradient sensitivity score. This represents the normalized behavior degradation score. Indicates a stable term;

[0039] The emotional behavior sensitivity score is calculated based on the normalized gradient sensitivity score and the normalized behavior degradation score, as shown in the following formula:

[0040] ;

[0041] in, Indicates the first The emotional behavior sensitivity score of each module to be quantified. Represents the fusion weight coefficient, and .

[0042] Preferably, the corresponding quantization bit width is allocated to each module to be quantized based on the emotional behavior sensitivity score and the preset average bit budget, specifically including:

[0043] Based on emotional behavior sensitivity score Sort all modules to be quantized in descending order to obtain the sorted index sequence, as shown in the following formula:

[0044] ;

[0045] in, This represents the sorted index sequence of all modules to be quantified, arranged from highest to lowest according to their emotional behavior sensitivity scores. `desc` indicates descending order. This indicates that the sorted index sequence is returned.

[0046] Set the high bit width Low bit width and preset average bit budget The number of high-bit protections that satisfy the average bit budget is calculated using the following formula. :

[0047] ;

[0048] in, Indicates sorting the index sequence Rank the emotional behavior sensitivity scores first The set of indices of modules to be quantized. This indicates that the variable that makes the function reach its maximum value is returned;

[0049] The quantization bit width is allocated to each module to be quantized based on the number of high-bit protections, as shown in the following formula:

[0050] ;

[0051] in, Indicates the first The quantization bit width corresponding to each module to be quantized Indicates sorting the index sequence Rank the emotional behavior sensitivity scores first The set of indexes of modules to be quantized.

[0052] Preferably, mixed-precision quantization is performed on the large end-side model to be quantized based on the quantization bit width to obtain the mixed-precision quantized large end-side model, specifically including:

[0053] According to the The quantization bit width corresponding to each module to be quantized Calculate the first The quantization scaling factor corresponding to each module to be quantized in the mixed-precision quantization stage. As shown in the following formula:

[0054] ;

[0055] Based on the quantization scaling factor, the first Weight matrix of each module to be quantized Quantization is performed to obtain the first Mixed-precision quantization weight matrix for each module to be quantized As shown in the following formula:

[0056] ;

[0057] in, This represents the floor function. This represents the truncation function;

[0058] Replace the weight matrices of each module in the large edge model to be quantized with the corresponding mixed-precision quantization weight matrices to obtain the large edge model after mixed-precision quantization. .

[0059] Secondly, the present invention provides an edge-side large-model hybrid precision quantization device based on emotion sensitivity, comprising:

[0060] The quantization module is configured to acquire the large edge model to be quantized and a calibration sample set containing sentiment labels. The large edge model to be quantized includes multiple modules. A sentiment-weighted loss function is constructed based on the calibration sample set and the large edge model to be quantized, and the gradient sensitivity score of each module to be quantized is calculated according to the sentiment-weighted loss function. Low-bit quantization perturbations are applied to each module to be quantized one by one to construct a corresponding perturbation model. Based on the calibration sample set, unperturbed reference responses and corresponding perturbed responses are generated using the large edge model to be quantized and each perturbed model, respectively. The behavior degradation score of each module to be quantized is calculated based on the difference in sentiment interaction behavior scores between the unperturbed reference responses and the perturbed responses generated by each perturbed model. The sentiment behavior sensitivity score of each module to be quantized is calculated based on the gradient sensitivity score and the behavior degradation score. A corresponding quantization bit width is allocated to each module to be quantized based on the sentiment behavior sensitivity score and a preset average bit budget. Mixed-precision quantization is performed on the large edge model to be quantized according to the quantization bit width to obtain the mixed-precision quantized large edge model.

[0061] The response module is configured to acquire the input text or the text sequence obtained through speech processing and input it into the edge-side large model after mixed precision quantization to obtain the corresponding response statement.

[0062] Thirdly, the present invention provides an electronic device including one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any implementation of the first aspect.

[0063] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any of the implementations of the first aspect.

[0064] Fifthly, the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the method as described in any of the implementations in the first aspect.

[0065] Compared with the prior art, the present invention has the following beneficial effects:

[0066] (1) The end-side large model hybrid precision quantization method based on emotion sensitivity mentioned in this invention constructs an emotion-weighted loss function by using a calibration sample set containing emotion labels, so that the gradient sensitivity calculation of the module to be quantized can be tilted towards negative emotion companion samples, thereby avoiding the problem of ignoring the key layer of emotion interaction when only based on the loss of general language modeling or weight error for bit allocation.

[0067] (2) The end-side large model hybrid precision quantization method based on emotional sensitivity mentioned in this invention applies low-bit quantization perturbation one by one and compares the emotional interaction behavior scores of the model output before and after the perturbation. It transforms end-side interaction degradation phenomena such as lengthy replies, increased numbering lists, abnormal format output, and insufficient negative emotional response into calculable behavior degradation scores, so that the hybrid precision quantization allocation can directly perceive the impact of low-bit quantization on output behavior.

[0068] (3) The emotional sensitivity-based end-side large model hybrid precision quantization method mentioned in this invention, under the constraint of average bit budget, performs high-bit protection on high-sensitivity modules to be quantized and low-bit quantization on low-sensitivity modules to be quantized according to the emotional behavior sensitivity score, thereby reducing end-side storage overhead and inference resource occupation, while improving response stability and behavior retention ability in emotional companionship scenarios. Attached Figure Description

[0069] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0070] Figure 1 This is a flowchart illustrating the edge-side large-model hybrid precision quantization method based on emotion sensitivity, as an embodiment of this application.

[0071] Figure 2 This is a flowchart illustrating an embodiment of the emotional sensitivity-based edge-side large model hybrid precision quantization method of this application.

[0072] Figure 3 This is a schematic diagram of an end-side large model hybrid precision quantization device based on emotion sensitivity, as an embodiment of this application.

[0073] Figure 4 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0074] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. The described embodiments are merely some embodiments of this invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.

[0075] Figure 1 This application illustrates an embodiment of an edge-side large model hybrid precision quantization method based on emotion sensitivity, comprising the following steps:

[0076] S1. Obtain the large edge model to be quantized and a calibration sample set containing sentiment labels. The large edge model to be quantized includes multiple modules to be quantized. Construct a sentiment-weighted loss function based on the calibration sample set and the large edge model to be quantized, and calculate the gradient sensitivity score of each module to be quantized according to the sentiment-weighted loss function. Apply low-bit quantization perturbation to each module to be quantized one by one to construct the corresponding perturbation model. Based on the calibration sample set, generate unperturbed reference responses and corresponding perturbed responses using the large edge model to be quantized and each perturbed model respectively. Calculate the behavior degradation score of the module to be quantized based on the difference in sentiment interaction behavior scores between the unperturbed reference responses and the perturbed responses generated by each perturbed model. Calculate the sentiment behavior sensitivity score of each module to be quantized based on the gradient sensitivity score and the behavior degradation score. Allocate a corresponding quantization bit width to each module to be quantized based on the sentiment behavior sensitivity score and the preset average bit budget. Perform mixed-precision quantization on the large edge model to be quantized according to the quantization bit width to obtain the large edge model after mixed-precision quantization.

[0077] Specifically, refer to Figure 2In the embodiments of this invention, the large-scale edge model to be quantized uses a pre-trained large language model as the core semantic understanding and text generation entity, specifically employing a Transformer-based sequence modeling architecture. For text-based tasks, the input to the large-scale edge model to be quantized is a text sequence or an encoded instruction sequence, and the output is the corresponding response statement. For voice interaction tasks, the input end can first convert the user's speech into a text sequence through a speech recognition module, and then the large-scale edge model to be quantized generates the corresponding text response. Thus, the model architecture applicable to this hybrid precision quantization method can be summarized as a unified sequence processing architecture based on Transformer, oriented towards emotion understanding, companion-style question answering, and multi-turn interaction generation. In one example, the large-scale edge model to be quantized can adopt the Qwen2.5 series instruction large-scale model adapted for emotional companionship tasks, such as Qwen2.5-1.5B-Instruct or Qwen2.5-3B-Instruct, and can further use the EQ-DoRA method to efficiently adapt parameters for emotional companionship scenarios before weight merging to obtain the large-scale edge model to be quantized. The modules to be quantized in the large edge model can include attention projection layers, feedforward network layers, gated projection layers, up projection layers, down projection layers, or other linear layers in the Transformer architecture. In one example, an MLP linear layer in the feedforward network can be used as the module to be quantized, and the emotional behavior sensitivity score of each module to be quantized is calculated based on the gradient significance under sentiment-weighted loss and the behavior degradation under low-bit quantization perturbation. The quantization hyperparameters can be set according to edge deployment constraints, such as low bit width. It can be set to 4 bits, with the high bit width as the maximum. It can be set to 8 bits, and the number of high-bit protection layers can be set to 5 layers, 7 layers, or automatically determined according to the average bit budget. Under the constraint of the average bit budget, high-bit protection is performed on the module to be quantized with high emotional sensitivity, and low-bit quantization is performed on the other modules to be quantized, so as to obtain the end-side large model after quantization with 4-bit / 8-bit mixed precision.

[0078] In smart home, companion robot, or mobile terminal scenarios, user text data, speech-to-text data, and multi-turn emotional interaction data are collected, and a calibration sample set containing emotional tags is constructed. Each calibration sample can include input text, a standard reference response, and an emotional tag. Emotional tags can include positive, neutral, and negative emotions. Negative emotions can be further categorized into sadness, anxiety, loneliness, fatigue, anger, or need for comfort. In one example, the calibration sample set can be divided into a probe set for sensitivity calculation and a test set for final evaluation. The probe set is used to calculate the emotional weighted loss function, gradient sensitivity score, behavior degradation score under layer-by-layer low-bit quantization perturbation, and the ranking of emotional behavior sensitivity scores for each module to be quantized. The test set is used to evaluate the behavior preservation effect of the large edge model after mixed-precision quantization on data that did not participate in layer selection. Evaluation metrics may include the voice broadcast adaptation rate of negative sentiment samples, the negative sentiment response matching rate, the overall abnormal format rate, the negative sentiment behavior retention rate, and the average number of bits for linear parameters. Higher voice broadcast adaptation and sentiment response matching rates indicate better edge-side interaction performance; lower abnormal format rates indicate more suitable responses for voice broadcasting and edge-side interaction; and higher behavior retention rates indicate that the quantized model better retains the effective behavior from the unquantized reference model. Therefore, the calibration sample set, sentiment labels, evaluation metrics, and quantization hyperparameters collectively provide the foundation for subsequent construction of the sentiment-weighted loss function, gradient sensitivity score calculation, low-bit quantization perturbation, behavior degradation score calculation, sentiment behavior sensitivity score calculation, quantization bit width allocation, and edge-side mixed-precision inference.

[0079] The following section details the hybrid quantization process for the large end-side model to be quantized.

[0080] In a specific embodiment, the process of constructing the sentiment-weighted loss function is as follows:

[0081] The calibration sample set containing sentiment labels is represented as follows: ;

[0082] in, Indicates the first Input text, Indicates the first The standard reference response for each input text. Indicates the first The sentiment tag corresponding to each input text. Indicates the total number of input texts;

[0083] The sentiment weight of a sample is calculated based on its sentiment label, using the following formula:

[0084] ;

[0085] in, Indicates the first Sentiment weights for each input text. This represents the negative sentiment weighting coefficient. This indicates a pre-defined set of negative emotions. Indicates an indicator function;

[0086] The sentiment-weighted loss function is constructed based on the sample sentiment weights, as shown in the following equation:

[0087] ;

[0088] in, This represents the sentiment-weighted loss function. This represents the large edge model to be quantized. This represents the parameters of the large end-side model to be quantized. This represents the task loss function. Indicates the emotion enhancement coefficient. ;

[0089] The calculation process for the gradient sensitivity score of each module to be quantized is as follows:

[0090] Calculate the sentiment-weighted loss function Regarding the first in the large end-side model to be quantized Weight matrix of each module to be quantized The gradient of is calculated, and the first gradient is calculated based on the weight matrix and the corresponding gradient. The gradient sensitivity score of each module to be quantized is shown in the following formula:

[0091] ;

[0092] in, Indicates the first Gradient sensitivity scores of each module to be quantized. Represents the weight matrix The number of elements in , This indicates the total number of modules to be quantized. This represents the Hadamard product. express Norm.

[0093] Specifically, refer to Figure 2 First, the large edge model to be quantized is determined, and the modules to be quantized are selected. In the embodiments of this application, the large edge model to be quantized includes multiple modules, which can be at least one of a feedforward network module, attention module, word embedding module, output mapping module, linear projection layer, gated projection layer, upper projection layer, or lower projection layer. The parameters of the large edge model to be quantized are... The process involves quantizing the weight matrices of each module to be quantized, which essentially adjusts the parameters of the large edge model. Next, a calibration sample set is constructed. The sentiment labels in the calibration sample set can be obtained through manual annotation, rule-based annotation, or automatic annotation by a sentiment classification model. During calibration, the calibration sample set containing sentiment labels is input into the large edge model to be quantized, yielding the response statements output by the large edge model. Sentiment weights are then determined based on the sentiment labels to calculate the sentiment-weighted loss function. The sentiment weights are used to reflect the priority of behavior preservation for different sentiment categories in edge-side interaction tasks. The sentiment-weighted loss function is used to construct a parameter sensitivity evaluation objective for sentiment-based interaction scenarios. The determination of sentiment weights relies on an indicator function. For negative emotion samples, such as sadness, anxiety, loneliness, fatigue, anger, or a need for comfort, calibration samples are selected based on the following conditions: hour, The value is 1, otherwise The value is 0, and combined with the negative sentiment weighting coefficient, input text belonging to the negative sentiment label is assigned a higher sentiment weight, while input text belonging to the positive sentiment label has a sentiment weight of 1. Furthermore, the sentiment weight can be used to... This enhances its contribution to the emotion-weighted loss function, allowing subsequent sensitivity calculations to focus more on the response retention capability of the large edge model to be quantified in negative emotional companionship scenarios. The task loss function... Cross-entropy loss or other sequence-generated losses can be used, and the specific type of task loss function is not limited in the embodiments of this application.

[0094] Further utilize the sentiment-weighted loss function Weight matrix for each module to be quantized Calculate the gradient and evaluate the influence of the module to be quantized on the sentiment-weighted loss function by element-wise multiplication of the gradient with the weight matrix. Gradient sensitivity score. The larger the value, the more sensitive the corresponding module to quantization is to quantization disturbances in the emotional interaction task, and the higher the protection priority should be given in the subsequent mixed precision quantization allocation.

[0095] In a specific embodiment, the calculation process for the behavioral degradation score is as follows:

[0096] Set low bit width , for the Weight matrix of each module to be quantized Perform low-bit quantization perturbation to obtain the perturbation weight matrix. The calculation process is as follows:

[0097] ;

[0098] ;

[0099] in, Indicates the first The quantization scaling factor corresponding to each module to be quantized during the low-bit quantization perturbation stage. This represents the floor function. This represents the truncation function. This indicates taking the maximum value;

[0100] The first in the large end-side model to be quantized The weight matrix of each module to be quantized is replaced with... The remaining modules to be quantized maintain reference precision, resulting in the first... The perturbation model corresponding to each module to be quantized ;

[0101] Based on the The input text is used to generate an unperturbed reference response using an unperturbed, unquantized edge-side large model, as shown in the following equation:

[0102] ;

[0103] Based on the The input text uses the first The perturbation model for each module to be quantized generates a corresponding perturbation response, as shown in the following equation:

[0104] ;

[0105] The first point is calculated based on the difference between the sentiment interaction score of the unperturbed reference response and the sentiment interaction score of the perturbed response. The behavioral degradation score of each module to be quantified is shown in the following formula:

[0106] ;

[0107] in, Indicates no disturbance (see reference reply) In emotional tags The emotional interaction behavior score is expressed as follows: ; Indicates a disturbance response In emotional tags The emotional interaction behavior score is expressed as follows: ; Indicates the first A behavioral indicator function, Indicates the first The weights corresponding to each behavioral indicator This represents the total number of behavioral indicators. Indicates the first The behavioral degradation score of each module to be quantified; the behavioral index function is a function that maps unperturbed reference responses or perturbed responses and their corresponding sentiment tags to behavioral scores by means of rule judgment, classification model, scoring model or manual annotation rules.

[0108] Specifically, embodiments of the present invention employ a layer-by-layer or module-by-module low-bit quantization perturbation method to evaluate the quantization sensitivity of different modules to be quantized. During each perturbation process, only the weight matrix of the current module to be quantized is... Temporarily replace it with a perturbation weight matrix obtained after quantization perturbation using a low bit width. The remaining modules to be quantized retain their original or reference precision, thus obtaining the corresponding perturbation models. This method isolates the low-bit quantization effects of different modules, preventing the difficulty in distinguishing the sources of behavioral degradation caused by simultaneous quantization of multiple modules. (Preset low bit width) It can take at least one of 2 bits, 3 bits, 4 bits or 5 bits.

[0109] Furthermore, behavioral degradation scores are calculated based on the changes in behavioral metrics of the model's responses before and after perturbation in the emotional interaction task. Specifically, the input text of each calibration sample in the calibration sample set is first input into the unperturbed, unquantized end-side large model and each perturbed model, generating corresponding unperturbed reference responses and perturbed responses. The emotional labels of the unperturbed reference responses and perturbed responses in the corresponding calibration samples are then calculated using behavioral metric functions. The emotional interaction behavior score is calculated based on the difference between the emotional interaction behavior score of the unperturbed reference response and the emotional interaction behavior score of the perturbed response. Behavioral degradation scores for each module to be quantified; behavioral degradation scores The larger the value, the higher the value. The more a module to be quantized is subjected to low bit perturbations, the more likely it is to cause a decrease in emotional interaction behavior.

[0110] The behavioral indicator functions may include at least one of the following: a voice broadcast adaptability function, an emotion response matching function, a negative emotion response matching function, a format stability function, a numbered list suppression function, a repetition suppression function, an abnormal output suppression function, or a response length adaptation function. The voice broadcast adaptability function measures whether the response is suitable for natural broadcasting by the edge-side speech synthesis module; the emotion response matching function measures whether the response matches the emotion label of the input sample; the negative emotion response matching function measures whether the model can provide comforting or supportive responses to negative emotion inputs such as sadness, anxiety, loneliness, and fatigue; the format stability function measures whether complex numbering, abnormal formats, or excessive list structures unsuitable for voice interaction appear in the response; and the response length adaptation function measures whether the response length is suitable for real-time edge interaction. The functions listed above can be used to map unperturbed or perturbed reference responses and their corresponding emotion labels to behavioral scores using rule-based judgment, classification models, scoring models, or manually labeled rules, and the emotional interaction behavioral score is calculated by combining the weights corresponding to each behavioral indicator. The embodiments of this application do not limit the formulas of the functions listed above.

[0111] In a specific embodiment, the calculation process for the emotional behavior sensitivity score of each module to be quantified is as follows:

[0112] The gradient sensitivity score and behavior degradation score are normalized separately to obtain the normalized gradient sensitivity score and normalized behavior degradation score, as shown in the following formula:

[0113] ;

[0114] ;

[0115] in, Indicates the first Gradient sensitivity scores of each module to be quantized. Indicates the first The behavioral degradation score of each module to be quantified This represents the normalized gradient sensitivity score. This represents the normalized behavior degradation score. Indicates a stable term;

[0116] The emotional behavior sensitivity score is calculated based on the normalized gradient sensitivity score and the normalized behavior degradation score, as shown in the following formula:

[0117] ;

[0118] in, Indicates the first The emotional behavior sensitivity score of each module to be quantified. This represents the fusion weight coefficient, and .

[0119] Specifically, after calculating the gradient sensitivity score and the behavioral degradation score, the emotional behavioral sensitivity score is calculated based on these scores. Gradient sensitivity score The behavioral degradation score reflects the parameter sensitivity of the module to be quantified under the sentiment-weighted loss function. This reflects the degree to which the module to be quantized affects the output behavior under low-bit quantization perturbations. Since the dimensions and numerical ranges of the two may differ, they are first normalized separately, and then fused using weighting coefficients. The emotional and behavioral sensitivity score is obtained by weighting. .when When the value is large, the mixed precision quantization allocation focuses more on gradient sensitivity; when... When the value is low, the mixed precision quantization allocation focuses more on the degradation of actual output behavior. Therefore, the emotional behavior sensitivity score can simultaneously reflect the sensitivity at the parameter level and the risk of degradation at the level of emotional interaction behavior.

[0120] In a specific embodiment, the corresponding quantization bit width is allocated to each module to be quantized based on the emotional behavior sensitivity score and the preset average bit budget, specifically including:

[0121] Based on emotional behavior sensitivity score Sort all modules to be quantized in descending order to obtain the sorted index sequence, as shown in the following formula:

[0122] ;

[0123] in, This represents the sorted index sequence of all modules to be quantified, arranged from highest to lowest according to their emotional behavior sensitivity scores. `desc` indicates descending order. This indicates that the sorted index sequence is returned.

[0124] Set the high bit width Low bit width and preset average bit budget The number of high-bit protections that satisfy the average bit budget is calculated using the following formula. :

[0125] ;

[0126] in, Indicates sorting the index sequence Rank the emotional behavior sensitivity scores first The set of indices of modules to be quantized. This indicates that the variable that makes the function reach its maximum value is returned;

[0127] The quantization bit width is allocated to each module to be quantized based on the number of high-bit protections, as shown in the following formula:

[0128] ;

[0129] in, Indicates the first The quantization bit width corresponding to each module to be quantized Indicates sorting the index sequence Rank the emotional behavior sensitivity scores first The set of indexes of modules to be quantized.

[0130] In a specific embodiment, mixed-precision quantization is performed on the large end-side model to be quantized based on the quantization bit width to obtain the mixed-precision quantized large end-side model, specifically including:

[0131] According to the The quantization bit width corresponding to each module to be quantized Calculate the first The quantization scaling factor corresponding to each module to be quantized in the mixed-precision quantization stage. As shown in the following formula:

[0132] ;

[0133] Based on the quantization scaling factor, the first Weight matrix of each module to be quantized Quantization is performed to obtain the first Mixed-precision quantization weight matrix for each module to be quantized As shown in the following formula:

[0134] ;

[0135] in, This represents the floor function. This represents the truncation function;

[0136] Replace the weight matrices of each module in the large edge model to be quantized with the corresponding mixed-precision quantization weight matrices to obtain the large edge model after mixed-precision quantization. .

[0137] Specifically, embodiments of this application first determine the sensitivity score based on emotional behavior. The modules to be quantized are sorted, with modules having higher emotional behavior sensitivity scores using a higher bit width and modules having lower emotional behavior sensitivity scores using a lower bit width. High bit width It can be at least one of 6 bits, 8 bits or higher bit width, with low bit width. It can be at least one of 2 bits, 3 bits, 4 bits, or 5 bits. In one example, it can be... , A 4-bit / 8-bit mixed-precision quantization strategy is employed, and the number of high-bit protections is determined under a preset average bit budget. The constraints allow for mixed-precision quantization within the limits of edge resources. This enables high-sensitivity layers or modules to receive higher bit protection, while low-sensitivity layers or modules employ lower bit quantization, thereby reducing model storage overhead and inference resource consumption while mitigating the risk of degradation in emotional interaction behavior.

[0138] The following specific embodiments illustrate the effectiveness of the edge-side large model hybrid precision quantization method based on emotion sensitivity proposed in this application.

[0139] The experimental configuration of this embodiment is as follows: This embodiment runs on an Ubuntu 22.04.5 LTS system. The server is equipped with dual Intel Xeon Gold 6530 CPUs, totaling 128 logical processors, and 251 GiB of system memory; the GPU is an NVIDIA GeForce RTX 5090 with 32GB of video memory. The base model uses a large language model aligned with Chinese instructions, and further performs efficient parameter fine-tuning on emotional companionship dialogue data. The fine-tuning stage uses a LoRA / DoRA-like parameter-efficient training method. In the LoRA training method, the rank... Set to 8, The settings are: 16, dropout: 0.05, maximum sequence length: 512, training epochs: 2, training steps: 476; optimizer: AdamW, learning rate: cosine annealing, peak learning rate: 16. The warmup ratio was set to 0.03, and the maximum gradient clipping norm was set to 0.3. Gradient checkpointing was enabled during training to reduce GPU memory usage, and validation and model saving were performed at fixed steps.

[0140] In the quantization experiment, a large language model fine-tuned on emotional companionship dialogue data was used as the large edge model to be quantized. The test set contained 192 edge voice companionship scenario samples, covering emotional categories such as neutral, positive, sadness, anxiety, loneliness, fatigue, anger, and soothing needs, of which AWQ calibration samples numbered 64. Comparison methods included the FP16 unquantized reference model, the unified INT4 quantization model, and the AWQ INT4 quantization model. In the method of this invention, the linear layer adopted a mixed precision configuration of INT4 / INT8, and the quantization group size was set to 128. Based on the emotional sensitivity score, layers 9, 10, 12, 16, 17, 18, and 27 were selected as high-bit protection layers, and the remaining layers adopted low-bit quantization; the final average bit width of the linear layer was 4.74 bits, and the theoretical linear layer storage size was approximately 873.87 MB; AWQ INT4 used 4 bits and a group size of 128. The maximum number of newly generated tokens in the response generation stage was set to 128, the repetition penalty coefficient was set to 1.05, and a deterministic decoding method was adopted. The evaluation phase automatically calculates behavioral indicators such as voice broadcast adaptability, emotional response matching degree, negative emotional response matching degree, format stability, numbered list suppression, response length adaptation, and repetition suppression, which are used to evaluate the behavior preservation effect of the quantified large model in the emotional interaction scenario on the device side.

[0141] Table 1. Comparison of different quantization methods for relevant indicators of voice interaction on the device side.

[0142]

[0143] According to the results in Table 1, in terms of terminal-side voice interaction related indicators, the average generation rate of the FP16 unquantized reference model is 67.15, the format stability is 99.79%, the numbered list suppression score is 67.97%, and the response length adaptation score is 98.45%; the average generation rate of the unified INT4 quantized model is 72.40, the format stability is 93.85%, the numbered list suppression score is 58.59%, and the response length adaptation score is 95.40%; the average generation rate of the AWQ-INT4 quantized model is 45.39, the format stability is 99.69%, the numbered list suppression score is 84.90%, and the response length adaptation score is 99.11%; the average generation rate of the method of this invention is 68.78, the format stability is 98.44%, the numbered list suppression score is 93.23%, and the response length adaptation score is 96.98%. The above results show that, while maintaining a generation rate close to that of the FP16 unquantized reference model, the method of the present invention significantly improves format stability and numbered list suppression capability compared to the unified INT4 quantized model, increasing from 93.85% and 58.59% to 98.44% and 93.23%, respectively. At the same time, the response length adaptation score increases from 95.40% to 96.98%, indicating that the method of the present invention can reduce the abnormal response format, increased list-like expression, and unsuitable response length after low-bit quantization, which are detrimental to the terminal-side speech playback, thereby improving the stability and playback adaptability of the terminal-side companion-style interactive response.

[0144] S2: Obtain the input text or the text sequence obtained through speech processing and input it into the edge-side large model after mixed precision quantization to obtain the corresponding response statement.

[0145] Specifically, the hybrid precision quantized edge-side large model obtained from the embodiments of this application is deployed in smart home terminals, companion robots, mobile terminals, edge computing devices, or other edge-side devices. During the edge-side inference phase, different modules to be quantized in the hybrid precision quantized edge-side large model perform forward propagation computation according to their allocated quantization bit widths; for high emotional sensitivity layers or modules, a higher bit width is used to maintain emotional understanding and response generation capabilities; for low emotional sensitivity layers or modules, a lower bit width is used to reduce storage overhead and computational resource consumption. Finally, the hybrid precision quantized edge-side large model generates response statements suitable for emotional interaction scenarios based on the input text.

[0146] The data mentioned in the embodiments of this application were all obtained in a legal and compliant manner, and their application scenarios are also legal and compliant.

[0147] Further reference Figure 3As an implementation of the methods shown in the above figures, this application provides an embodiment of an end-side large model hybrid precision quantization device based on emotion sensitivity. This device embodiment is similar to... Figure 1 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0148] This application provides an edge-side large model hybrid precision quantization device based on emotion sensitivity, including:

[0149] Quantization module 1 is configured to acquire the large edge model to be quantized and a calibration sample set containing sentiment labels. The large edge model to be quantized includes multiple modules. A sentiment-weighted loss function is constructed based on the calibration sample set and the large edge model to be quantized, and the gradient sensitivity score of each module to be quantized is calculated according to the sentiment-weighted loss function. Low-bit quantization perturbations are applied to each module to be quantized one by one to construct corresponding perturbation models. Based on the calibration sample set, unperturbed reference responses and corresponding perturbed responses are generated using the large edge model to be quantized and each perturbed model, respectively. The behavior degradation score of each module to be quantized is calculated based on the difference in sentiment interaction behavior scores between the unperturbed reference responses and the perturbed responses generated by each perturbed model. The sentiment behavior sensitivity score of each module to be quantized is calculated based on the gradient sensitivity score and the behavior degradation score. A corresponding quantization bit width is allocated to each module to be quantized based on the sentiment behavior sensitivity score and a preset average bit budget. Mixed-precision quantization is performed on the large edge model to be quantized according to the quantization bit width to obtain the mixed-precision quantized large edge model.

[0150] Response module 2 is configured to acquire the input text or the text sequence obtained through speech processing and input it into the edge-side large model after mixed precision quantization to obtain the corresponding response statement.

[0151] Figure 4 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. For example... Figure 4 As shown, the electronic device in this embodiment includes a processor 401 and a memory 402; wherein the memory 402 is used to store computer execution instructions; and the processor 401 is used to execute the computer execution instructions stored in the memory to implement the various steps performed by the electronic device in the above embodiment. For details, please refer to the relevant descriptions in the foregoing method embodiments.

[0152] Alternatively, the memory 402 can be either standalone or integrated with the processor 401.

[0153] When the memory 402 is set up independently, the electronic device also includes a bus 403 for connecting the memory 402 and the processor 401.

[0154] This invention also provides a computer storage medium storing computer execution instructions, which, when executed by processor 401, implement the above method.

[0155] This invention also provides a computer program product, including a computer program, which, when executed by a processor 401, implements the above-described method.

[0156] In the embodiments provided by this invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0157] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to implement the solution of this embodiment according to actual needs.

[0158] Furthermore, the functional modules in the various embodiments of this invention can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit formed by the above modules can be implemented in hardware or in the form of hardware plus software functional units.

[0159] The integrated modules implemented as software functional modules described above can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor 401 to execute some steps of the methods of the various embodiments of this application.

[0160] It should be understood that the processor 401 described above can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor, or the processor 401 can be any conventional processor 401. The steps of the method disclosed in this invention can be directly manifested as the hardware processor 401 executing the steps, or as a combination of hardware and software modules within the processor 401 executing the steps.

[0161] The memory 402 may include high-speed RAM memory, and may also include non-volatile memory NVM, such as at least one disk storage device, and may also be a USB flash drive, portable hard drive, read-only memory, disk or optical disc, etc.

[0162] Bus 403 can be an Industry Standard Architecture (ISA), a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Bus 403 can be divided into address bus, data bus, control bus, etc. For ease of illustration, the bus 403 in the accompanying drawings of this application is not limited to only one bus 403 or one type of bus 403.

[0163] The aforementioned storage medium can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium accessible to general-purpose or special-purpose computers.

[0164] An exemplary storage medium is coupled to processor 401, enabling processor 401 to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of processor 401. Processor 401 and storage medium can reside in application-specific integrated circuits (ASICs). Alternatively, processor 401 and storage medium can exist as discrete components in an electronic device or host device.

[0165] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0166] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A hybrid precision quantization method for edge-side large models based on emotion sensitivity, characterized in that, Includes the following steps: Obtain the large edge model to be quantized and a calibration sample set containing sentiment labels. The large edge model to be quantized includes multiple modules to be quantized. Construct a sentiment-weighted loss function based on the calibration sample set and the large edge model to be quantized, and calculate the gradient sensitivity score of each module to be quantized according to the sentiment-weighted loss function. Low-bit quantization perturbation is applied to each module to be quantized one by one to construct the corresponding perturbation model; based on the calibration sample set, unperturbed reference response and corresponding perturbation response are generated using the end-side large model to be quantized and each perturbation model respectively; and the behavior degradation score of the module to be quantized is calculated based on the difference in emotional interaction behavior scores between the unperturbed reference response and the perturbation responses generated by each perturbation model. The emotional and behavioral sensitivity scores of each module to be quantified are calculated based on the gradient sensitivity scores and the behavioral degradation scores. Based on the emotional behavior sensitivity score and the preset average bit budget, a corresponding quantization bit width is allocated to each module to be quantized; based on the quantization bit width, mixed-precision quantization is performed on the large edge model to be quantized to obtain the large edge model after mixed-precision quantization, specifically including: According to the The quantization bit width corresponding to each module to be quantized Calculate the first The quantization scaling factor corresponding to each module to be quantized in the mixed-precision quantization stage. As shown in the following formula: ； in, , This indicates the total number of modules to be quantized. This indicates taking the maximum value; According to the quantization scaling factor, the first Weight matrix of each module to be quantized Quantization is performed to obtain the first Mixed-precision quantization weight matrix of each module to be quantized As shown in the following formula: ； in, This represents the floor function. This represents the truncation function; Replace the weight matrices of each module in the large edge model to be quantized with the corresponding mixed-precision quantization weight matrices to obtain the large edge model after mixed-precision quantization. ; The input text or the text sequence obtained through speech processing is acquired and input into the large end-side model after mixed precision quantization to obtain the corresponding response statement.

2. The edge-side large-model hybrid precision quantization method based on emotion sensitivity according to claim 1, characterized in that, The process of constructing the sentiment-weighted loss function is as follows: The calibration sample set containing sentiment labels is represented as follows: ; in, Indicates the first Input text, Indicates the first The standard reference response for each input text. Indicates the first The sentiment tag corresponding to each input text. Indicates the total number of input texts; The sample sentiment weight is calculated based on the sentiment tags, using the following formula: ； in, Indicates the first Sentiment weights for each input text. This represents the negative sentiment weighting coefficient. This indicates a pre-defined set of negative emotions. Indicates an indicator function; Based on the sample sentiment weights, a sentiment-weighted loss function is constructed, as shown in the following equation: ； in, This represents the sentiment-weighted loss function. This represents the large edge model to be quantized. This represents the parameters of the large edge model to be quantized. This represents the task loss function. Indicates the emotion enhancement coefficient. ; The calculation process for the gradient sensitivity score of each module to be quantized is as follows: Calculate the sentiment-weighted loss function Regarding the first part of the large end-side model to be quantized Weight matrix of each module to be quantized The gradient of the first gradient is calculated based on the weight matrix and the corresponding gradient. The gradient sensitivity scores of each module to be quantized are shown in the following formula: ； in, Indicates the first Gradient sensitivity scores of each module to be quantized. Represents the weight matrix The number of elements in This represents the Hadamard product. express Norm.

3. The edge-side large-model hybrid precision quantization method based on emotion sensitivity according to claim 2, characterized in that, The calculation process for the behavioral degradation score is as follows: Set low bit width , for the Weight matrix of each module to be quantized Perform low-bit quantization perturbation to obtain the perturbation weight matrix. The calculation process is as follows: ； ； in, Indicates the first The quantization scaling factor corresponding to each module to be quantized during the low-bit quantization perturbation stage. This represents the floor function. This represents the truncation function; The first in the large end-side model to be quantized The weight matrix of each module to be quantized is replaced with... The remaining modules to be quantized maintain reference precision, resulting in the first... The perturbation model corresponding to each module to be quantized ; Based on the The input text is used to generate an unperturbed reference response using an unperturbed edge-side large model, as shown in the following equation: ； Based on the The input text uses the first The perturbation model for each module to be quantized generates a corresponding perturbation response, as shown in the following equation: ； The first point is calculated based on the difference between the sentiment interaction score of the unperturbed reference response and the sentiment interaction score of the perturbed response. The behavioral degradation score of each module to be quantified is shown in the following formula: ； in, Indicates no disturbance (see reference reply) In emotional tags The emotional interaction behavior score is expressed as follows: ; Indicates a disturbance response In emotional tags The emotional interaction behavior score is expressed as follows: ; Indicates the first A behavioral indicator function, Indicates the first The weights corresponding to each behavioral indicator This represents the total number of behavioral indicators. Indicates the first The behavioral degradation score of each module to be quantified; the behavioral index function is a function that maps unperturbed reference responses or perturbed responses and their corresponding sentiment tags to behavioral scores by means of rule judgment, classification model, scoring model or manual annotation rules.

4. The edge-side large-model hybrid precision quantization method based on emotion sensitivity according to claim 3, characterized in that, The calculation process for the emotional behavior sensitivity scores of each module to be quantified is as follows: The gradient sensitivity score and the behavior degradation score are normalized respectively to obtain the normalized gradient sensitivity score and the normalized behavior degradation score, as shown in the following formula: ； ； in, Indicates the first Gradient sensitivity scores of each module to be quantized. Indicates the first The behavioral degradation score of each module to be quantified This represents the normalized gradient sensitivity score. This represents the normalized behavior degradation score. Indicates a stable term; The emotional behavior sensitivity score is calculated based on the normalized gradient sensitivity score and the normalized behavior degradation score, as shown in the following formula: ； in, Indicates the first The emotional behavior sensitivity score of each module to be quantified. Represents the fusion weight coefficient, and .

5. The edge-side large-model hybrid precision quantization method based on emotion sensitivity according to claim 2, characterized in that, Based on the emotional behavior sensitivity score and the preset average bit budget, a corresponding quantization bit width is allocated to each module to be quantized, specifically including: Based on the emotional behavior sensitivity score Sort all modules to be quantized in descending order to obtain the sorted index sequence, as shown in the following formula: ； in, This represents the sorted index sequence of all modules to be quantified, arranged from highest to lowest according to their emotional behavior sensitivity scores. `desc` indicates descending order. This indicates that the sorted index sequence is returned. Set the high bit width Low bit width and preset average bit budget The number of high-bit protections that satisfy the average bit budget is calculated using the following formula. : ； in, Indicated in the sorted index sequence Rank the emotional behavior sensitivity scores first The set of indices of modules to be quantized. This indicates that the variable that makes the function reach its maximum value is returned; The quantization bit width is allocated to each module to be quantized based on the number of high-bit protections, as shown in the following formula: ； in, Indicates the first The quantization bit width corresponding to each module to be quantized Indicated in the sorted index sequence Rank the emotional behavior sensitivity scores first The set of indexes of modules to be quantized.

6. A hybrid precision quantization device for edge-side large models based on emotion sensitivity, characterized in that, include: The quantization module is configured to acquire the large edge model to be quantized and a calibration sample set containing sentiment labels. The large edge model to be quantized includes multiple modules to be quantized. A sentiment-weighted loss function is constructed based on the calibration sample set and the large edge model to be quantized, and the gradient sensitivity score of each module to be quantized is calculated according to the sentiment-weighted loss function. Low-bit quantization perturbation is applied to each module to be quantized one by one to construct the corresponding perturbation model; based on the calibration sample set, unperturbed reference response and corresponding perturbation response are generated using the end-side large model to be quantized and each perturbation model respectively; and the behavior degradation score of the module to be quantized is calculated based on the difference in emotional interaction behavior scores between the unperturbed reference response and the perturbation responses generated by each perturbation model. The emotional and behavioral sensitivity scores of each module to be quantified are calculated based on the gradient sensitivity scores and the behavioral degradation scores. Based on the emotional behavior sensitivity score and the preset average bit budget, a corresponding quantization bit width is allocated to each module to be quantized; based on the quantization bit width, mixed-precision quantization is performed on the large edge model to be quantized to obtain the large edge model after mixed-precision quantization, specifically including: According to the The quantization bit width corresponding to each module to be quantized Calculate the first The quantization scaling factor corresponding to each module to be quantized in the mixed-precision quantization stage. As shown in the following formula: ； in, , This indicates the total number of modules to be quantized. This indicates taking the maximum value; According to the quantization scaling factor, the first Weight matrix of each module to be quantized Quantization is performed to obtain the first Mixed-precision quantization weight matrix of each module to be quantized As shown in the following formula: ； in, This represents the floor function. This represents the truncation function; Replace the weight matrices of each module in the large edge model to be quantized with the corresponding mixed-precision quantization weight matrices to obtain the large edge model after mixed-precision quantization. ; The response module is configured to acquire the input text or the text sequence obtained through speech processing and input it into the large edge model after mixed precision quantization to obtain the corresponding response statement.

7. An electronic device, comprising: One or more processors; A storage device for storing one or more programs, characterized in that, when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the method as described in any one of claims 1-5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-5.

Images