Request response method, related device, equipment and storage medium

Abstract

The invention discloses a request response method, a related device, equipment and a storage medium, and the request response method comprises the steps: responding to a data processing request of a target task received by terminal equipment, and loading a data processing model suitable for the target task as a target processing model; wherein the data processing model is subjected to quantization training in advance to obtain model quantization parameters, the model quantization parameters comprise quantization sub-parameters of network blocks in the data processing model under different quantization bit widths, and the quantization sub-parameters comprise rounding values and quantization step lengths; selecting a quantization sub-parameter under one quantization bit width for the network block based on a hardware parameter of the terminal device as a target sub-parameter of the network block; and based on the target sub-parameter of each network block in the target processing model, performing quantification processing on the to-be-processed data requested to be processed by the data processing request to obtain a data processing result. According to the scheme, the success rate of executing the data processing request by the terminal equipment can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a request-response method and related apparatus, devices and storage media. Background Technology

[0002] With the rapid development of deep learning technology, neural networks are becoming increasingly complex, especially data processing models represented by large-scale language models, which can even process various types of data such as text, speech, and images. Consequently, the compatibility of data processing models with data processing requests is also getting better and better.

[0003] Meanwhile, corresponding to the significant increase in model complexity, the demand for hardware resources in data processing models is also increasing. Due to the varying and inconsistent hardware parameters of hardware platforms (such as mobile terminals, IoT devices, and embedded devices), existing data processing models struggle to operate effectively on various terminal devices, making it difficult to guarantee the success rate of data processing requests. Therefore, improving the success rate of data processing requests executed by terminal devices has become an urgent problem to be solved. Summary of the Invention

[0004] The main technical problem addressed by this application is to provide a request-response method and related apparatus, devices, and storage media that can improve the success rate of terminal devices executing data processing requests.

[0005] To address the aforementioned technical problems, the first aspect of this application provides a request-response method, comprising: responding to a terminal device receiving a data processing request for a target task, loading a data processing model suitable for the target task as a target processing model; wherein the data processing model is pre-trained with quantization parameters, the model quantization parameters including quantization sub-parameters of network blocks in the data processing model at different quantization bit widths, the quantization sub-parameters including rounding values ​​and quantization step sizes; based on the hardware parameters of the terminal device, selecting one of the quantization sub-parameters of a quantization bit width for a network block as a target sub-parameter of the network block; and based on the target sub-parameters of each network block in the target processing model, performing quantization processing on the data to be processed requested by the data processing request to obtain a data processing result.

[0006] To address the aforementioned technical problems, a second aspect of this application provides a request-response method, comprising: responding to a terminal device receiving a data processing request for a natural language processing task, loading a data processing model suitable for the natural language processing task as a target processing model; wherein the data processing model is pre-trained with quantization parameters, the model quantization parameters including quantization sub-parameters of network blocks in the data processing model at different quantization bit widths, the quantization sub-parameters including rounding values ​​and quantization step sizes; selecting a quantization sub-parameter of one of the quantization bit widths for a network block based on the hardware parameters of the terminal device, as a target sub-parameter of the network block; and quantizing a first session sequence requested by the data processing request based on the target sub-parameters of each network block in the target processing model to obtain a second session sequence for responding to the first session sequence.

[0007] To address the aforementioned technical problems, a third aspect of this application provides a request-response apparatus, comprising: a first model loading module, a first parameter selection module, and a first quantization processing module. The first model loading module is used to load a data processing model suitable for the target task as the target processing model in response to a data processing request received by a terminal device for a target task. The data processing model is pre-trained with quantization parameters, which include quantization sub-parameters of network blocks in the data processing model at different quantization bit widths. These sub-parameters include rounding values ​​and quantization step sizes. The first parameter selection module is used to select a quantization sub-parameter at one of the quantization bit widths for a network block based on the hardware parameters of the terminal device, as the target sub-parameter of the network block. The first quantization processing module is used to quantize the data to be processed requested by the data processing request based on the target sub-parameters of each network block in the target processing model, thereby obtaining a data processing result.

[0008] To address the aforementioned technical problems, a fourth aspect of this application provides a request-response apparatus, comprising: a second model loading module, a second parameter selection module, and a second quantization processing module. The second model loading module is used to load a data processing model suitable for a natural language processing task as the target processing model in response to a data processing request received by a terminal device for a natural language processing task. The data processing model is pre-trained with quantization parameters, which include quantization sub-parameters of network blocks in the data processing model at different quantization bit widths. These sub-parameters include rounding values ​​and quantization step sizes. The second parameter selection module is used to select a quantization sub-parameter of one of the quantization bit widths for a network block based on the hardware parameters of the terminal device, as the target sub-parameter of the network block. The second quantization processing module is used to quantize the first session sequence requested by the data processing request based on the target sub-parameters of each network block in the target processing model, to obtain a second session sequence for responding to the first session sequence.

[0009] To address the aforementioned technical problems, a fifth aspect of this application provides an electronic device comprising at least a memory and a processor coupled to each other, wherein the memory stores at least program instructions, and the processor executes the program instructions to implement the request-response method in the first or second aspect described above.

[0010] To address the aforementioned technical problems, a sixth aspect of this application provides a computer-readable storage medium storing program instructions executable by a processor, the program instructions being used to implement the request-response method of the first or second aspect described above.

[0011] The above scheme, in response to the terminal device receiving a data processing request for the target task, loads a data processing model suitable for the target task as the target processing model. This model is pre-trained with quantization parameters, which include quantization sub-parameters for network blocks in the data processing model at different quantization bit widths. These sub-parameters include rounding values ​​and quantization step sizes. Based on the terminal device's hardware parameters, a quantization sub-parameter at one of the quantization bit widths is selected for each network block as its target sub-parameter. Then, based on the target sub-parameters of each network block in the target processing model, the data to be processed in the data processing request is quantized to obtain the data processing result. Since the data processing model has been pre-trained with quantization parameters... The model quantization parameters are obtained through training, and these parameters include quantization sub-parameters for network blocks in the data processing model at different quantization bit widths. Therefore, when a terminal device executes a data processing request for the target task, it only needs to load a data processing model suitable for the target task first, and then, combined with its own hardware parameters, select a quantization sub-parameter at one of the quantization bit widths for each network block of the data processing model as the target sub-parameter. This ensures that the quantization sub-parameters of the network blocks in the data processing model loaded on the terminal device are as compatible as possible with the terminal device's own hardware parameters during subsequent quantization processing. Consequently, the hardware resources required by the terminal device when performing quantization processing on the data to be processed are also adapted to its own hardware parameters as much as possible. Therefore, this improves the success rate of the terminal device executing data processing requests. Attached Figure Description

[0012] Figure 1 This is a flowchart illustrating an embodiment of the request-response method of this application; Figure 2 This is a flowchart illustrating another embodiment of the request-response method of this application; Figure 3 This is a schematic diagram of the framework of an embodiment of the request response device of this application; Figure 4 This is a schematic diagram of another embodiment of the request response device of this application; Figure 5This is a schematic diagram of the framework of an embodiment of the electronic device of this application; Figure 6 This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0013] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0014] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0015] In this paper, the terms "system" and "network" are often used interchangeably. The term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the slash " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this paper indicates two or more objects.

[0016] Please see Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the request-response method of this application. It should be noted that the process operations in this embodiment can be executed by a terminal device with computing capabilities, and the specific structure or type of the terminal device is not limited herein. Specifically, this embodiment may include the following steps: Step S11: In response to the terminal device receiving a data processing request for the target task, load a data processing model suitable for the target task as the target processing model.

[0017] In this embodiment, the data processing model can be pre-trained using quantization to obtain model quantization parameters. These parameters can include quantization sub-parameters for network blocks in the data processing model at different quantization bit widths. Each sub-parameter can include a rounding value and a quantization step size. It should be noted that the rounding value after quantization training is a preset lower rounding limit or a preset upper rounding limit. The preset lower rounding limit represents rounding down (e.g., rounding down when quantizing model weights), and the preset upper rounding limit represents rounding up (e.g., rounding up when quantizing model weights). For example, the preset lower rounding limit can be set to 0, representing rounding down, and the preset upper rounding limit can be set to 1, representing rounding up. Furthermore, the quantization step size represents the scaling ratio during quantization. A larger quantization step size results in a larger scaling ratio during quantization, and vice versa. Further, the quantization step size can include the quantization step size for model weights and the quantization step size for activation values; at least one of these can be optimized and adjusted during quantization training. For example, only the quantization step size of the activation values ​​can be optimized. In this case, the quantization step size of the model weights can refer to the technical details of the quantization formula (such as symmetric quantization), which will not be elaborated here. Furthermore, for data processing models that are convolutional neural networks, network blocks can be convolutional blocks, residual blocks, etc.; for data processing models that are large models, network blocks can be Transformer blocks, etc. Of course, the above examples are only a few possible examples of network blocks in practical applications. Other possible situations are not limited here (e.g., a predetermined number of network layers can also be considered a network block), and will not be listed in detail here.

[0018] In one implementation scenario, the terminal device may include, but is not limited to: mobile terminals (such as smartphones, tablets, etc.), IoT (Internet of Things) devices (such as smart meters, smart cameras, etc.), and embedded devices (such as programmable logic controllers, industrial robots, etc.). The specific type of terminal device is not limited here.

[0019] In an implementation scenario, the target task can vary depending on the application context. For example, in applications such as traffic management and product inspection, the target task might be an image processing task (e.g., target detection in images of road traffic, defect detection in images of production line products). Alternatively, in applications such as online meetings and overseas travel, the target task might be an audio processing task (e.g., speech recognition in audio collected during online meetings, speech translation in audio collected during overseas travel). Or, in applications such as game development and social media, the target task might be a video processing task (e.g., video generation to drive game characters to perform specified actions, video generation based on social media text and image materials). Or, in applications such as intelligent customer service, the target task might be a natural language processing task (e.g., answering customer-input questions). Of course, the above examples are merely a few possible examples of target tasks in practical applications; the specific types of target tasks are not limited here, nor will they be listed in detail.

[0020] In an implementation scenario, the data processing request for the target task may include data to be processed. For example, for image processing tasks in applications such as traffic management and product testing, the data to be processed may include, but is not limited to, images of road traffic or images of products on the production line. Similarly, for audio processing tasks in applications such as online meetings or overseas travel, the data to be processed may include, but is not limited to, audio collected from online meetings or overseas travel. Or, for video processing tasks in applications such as game development and social media, the data to be processed may include, but is not limited to, game characters and their action commands, or social media graphic materials. Or, for natural language processing tasks in applications such as intelligent customer service, the data to be processed may include, but is not limited to, customer input questions. Of course, the above examples are merely a few possible examples of data to be processed under different target tasks; the specific content of the data to be processed is not limited here, nor will all examples be provided.

[0021] In one implementation scenario, as a possible example, the data processing model suitable for the target task can be a data processing model specifically designed for a particular target task. For example, for the target task of "image processing," data processing models such as object detection models and image segmentation models can be pre-trained; for the target task of "audio processing," data processing models such as speech recognition models and speech translation models can be pre-trained, and so on for other cases. As another possible example, the data processing model suitable for the target task can also be a data processing model compatible with multiple target tasks. For example, the data processing model can be a multimodal large model, compatible with image processing tasks, audio processing tasks, video processing tasks, and natural language processing tasks. It should be noted that the model architecture of the above data processing models can be found in relevant technical details, which will not be elaborated upon here.

[0022] In one implementation scenario, different quantization bit widths can share a rounding value, while different quantization bit widths can independently have their own quantization step size (e.g., the quantization step size of the activation value). That is, for any given network block, it can share the same rounding value across different quantization bit widths, but it can independently have its own quantization step size across different quantization bit widths. For ease of description, the rounding value shared by any given network block across different quantization bit widths can be denoted as R, and the quantization step size for an activation value independently possessed at a quantization bit width b can be denoted as... The quantization step size of the model weights can be denoted as: Furthermore, different quantization bit widths can include positive integers between a preset lower bit width limit and a preset upper bit width limit. For example, the preset lower bit width limit can be set to 2, and the preset upper bit width limit can be set to 8. Then, different quantization bit widths are positive integers between 2 and 8. For ease of description, the selected set B of different quantization bit widths can be represented as: , In the above formula, b represents the quantization bit width, and Z... + This represents a positive integer. Of course, the above example is merely one possible illustration of different quantization bit widths in practical applications; other cases are not limited here, nor will they be listed individually. To facilitate understanding of the following description of quantization training, the training objective of quantization training will be briefly explained below. The training objective of quantization training can be found in the following formula: , In the above formula, w represents the model weights, and x represents the activation value. This represents the quantization step size of the model weights. The model weights after quantization with the rounded value R. Indicates the quantization step size of the activated value. The activation value after quantization. This represents the rounded value obtained after quantitative training based on the above training objectives. This represents the final optimized quantization step size after quantization training based on the above training objective. Furthermore, function N represents the model output, and function E represents the difference comparison (e.g., mean squared error function). In other words, during quantization training, the training objective can be to minimize the difference between the model output before and after quantization. Of course, during quantization training, network blocks can also be used as the basic unit, in which case the above training objective can be expanded into the following formula: , In the above formula, the function This represents the model output of the l-th network block in the data processing model before quantization. This represents the model output after quantizing the l-th network block in the data processing model with a quantization bit width of b. Furthermore, the parameters in the above formula are the same as those in the previous formula; please refer to the relevant descriptions of the previous formulas, which will not be repeated here. During quantization training, the rounding value R is a learnable continuous variable about the model weights, which can be regularized to converge to {0,1}, i.e., the rounding value after training convergence. As mentioned earlier, this indicates whether rounding is performed up or down during quantization. To better understand the role of rounding values ​​in quantization, the following example illustrates the symmetric quantization of model weights w. Model weights after quantization. It can be represented as: , In the above formula, clip(x,q) min ,q max The function represents constraining x to q. min q max between, This indicates rounding down x. Furthermore, for the other parameters in the above formula that are the same as in the previous formula, please refer to the relevant descriptions of the previous formulas; they will not be repeated here.

[0023] In one implementation scenario, to optimize the quantization sub-parameters for quantization training of a data processing model, initial quantization parameters can be obtained based on a first calibration dataset of the task to which the data processing model is applicable. The quantization bit width is then divided into multiple bit width intervals based on the bit width size. It should be noted that the initial quantization parameters can include the initial sub-parameters of network blocks in the data processing model at different quantization bit widths. Furthermore, the initial sub-parameters can also include rounding values ​​and quantization step sizes. Of course, since training optimization has not yet begun, the specific values ​​of the rounding values ​​and quantization step sizes in the initial sub-parameters are different from those after training convergence. Based on this, various quantization bit widths belonging to different bit width intervals can be selected as candidate bit widths. Thus, based on the initial sub-parameters of network blocks in the data processing model at the candidate bit widths, the sample data of the task to which the data processing model is applicable can be quantized to obtain the output features of the network blocks at the candidate bit widths. Furthermore, the initial sub-parameters can be adjusted based on the training loss measured by the difference between the output features of the same network blocks at different candidate bit widths. It should be noted that the latest initial sub-parameters after the data processing model training converges are used as the quantization sub-parameters. This method, combined with the training loss measured by the differences in output features across candidate bit widths in different bit width intervals, optimizes and adjusts the initial sub-parameters, enhancing the correlation between different bit width intervals and improving the overall performance and robustness of the elastic quantization neural network (i.e., the data processing model).

[0024] In a specific implementation scenario, as one possible example, if the data processing model is suitable for image processing tasks, then an image dataset can be used as the first calibration dataset; or, as another possible example, if the data processing model is suitable for audio processing tasks, then an audio dataset can be used as the first calibration dataset; or, as yet another possible example, if the data processing model is suitable for video processing tasks, then a video dataset can be used as the first calibration dataset; or, as yet another possible example, if the data model is suitable for natural language processing tasks, then a natural language dataset can be used as the first calibration dataset. Of course, if the data processing model is compatible with multiple tasks mentioned above, then multiple datasets can be used as the first calibration dataset.

[0025] In a specific implementation scenario, the specific process of obtaining the initial quantization parameters of the data processing model based on the first calibration dataset can be found in technical details such as symmetric quantization and asymmetric quantization, which will not be elaborated here.

[0026] In a specific implementation scenario, the exact number of multiple bit-width intervals can be set according to the actual application needs. For example, when the accuracy requirements of the quantization sub-parameters are high, the number of multiple bit-width intervals can be set larger; when the training speed requirements of quantization training are high, the number of multiple bit-width intervals can be set smaller. For ease of understanding, taking three bit-width intervals as an example, they can be referred to as the low bit-width interval, the middle bit-width interval, and the high bit-width interval, respectively. Taking the aforementioned different quantization bit widths of 2 to 8 as an example, the low bit-width interval can be 2 to 3, the middle bit-width interval can be 4 to 6, and the high bit-width interval can be 7 to 8. Of course, the above example is only one possible way to divide multiple bit-width intervals in practical applications. Other possible division methods are not limited here, nor will they be listed one by one.

[0027] In a specific implementation scenario, after obtaining multiple bit-width intervals, a quantization bit width can be randomly selected for each interval as a candidate bit width. Taking the aforementioned three bit-width intervals as an example, quantization bit width 2 can be selected as a candidate bit width in the low bit-width interval, quantization bit width 5 in the middle bit-width interval, and quantization bit width 7 in the high bit-width interval. Of course, the above example is merely one possible illustration of candidate bit width selection in practical applications; other possible scenarios are not limited here, nor will they be listed individually.

[0028] In a specific implementation scenario, after obtaining the candidate bit width, the sample data of the task to which the data processing model is applicable (refer to the aforementioned first calibration dataset) can be quantized directly based on the initial sub-parameters of the network block in the data processing model at the candidate bit width to obtain the output features of the network block at the candidate bit width. Specifically, the model weights of the network block can be quantized based on the quantization step size and rounding value of the model weights in the initial sub-parameters at the candidate bit width (e.g., model weights). The quantization formula is used to process the input data of the network block. During network block processing, the activation values ​​are quantized using the quantization step size of the activation values ​​in the initial sub-parameters under the candidate bit width (see details of asymmetric quantization for more information). This yields the output features of the network block under the candidate bit width. It should be noted that for the first network block in the data processing model, its input data can be sample data, while for other network blocks (e.g., the second, third, etc.), its input data can be the output features of its predecessor network block under the same candidate bit width.

[0029] In a specific implementation scenario, as one possible approach, after obtaining the output features of a network block at candidate bit widths, the initial sub-parameters can be adjusted directly based on the training loss measured by the differences between the output features of the same network block at different candidate bit widths. The latest initial sub-parameters after the data processing model training converges are then used as the quantization sub-parameters. Furthermore, based on the differences between the output features of the same network block at different candidate bit widths in bit width combinations (e.g., measured using a mean squared error function), the sub-loss corresponding to the bit width combination can be obtained. These sub-losses are then fused to obtain the training loss, which in turn allows for the adjustment of the initial sub-parameters. For example, adjacent candidate bit widths can be selected as bit width combinations based on sequentially arranged candidate bit widths. Taking the aforementioned selection of quantization bit widths 2, 5, and 7 as candidate bit widths as an example, candidate bit widths 2 and 5 can be used as a bit width combination, and candidate bit widths 5 and 7 as a bit width combination. Based on this, sub-losses corresponding to bit-width combinations 2 and 5 can be obtained based on the difference between the output features of candidate bit-width 2 and candidate bit-width 5, and sub-losses corresponding to bit-width combinations 5 and 7 can be obtained based on the difference between the output features of candidate bit-width 5 and candidate bit-width 7. Of course, the above example is only one possible example in practical applications; other possible scenarios are not limited here, nor will they be listed one by one. The above method, based on sequentially arranged different candidate bit-widths, selects adjacent candidate bit-widths as bit-width combinations, and then obtains the sub-losses corresponding to the bit-width combinations based on the differences in output features of the same network block under different candidate bit-widths within the bit-width combinations. The sub-losses of each bit-width combination are then fused to obtain the training loss. Based on the training loss, the initial sub-parameters are adjusted. This allows the learning process of relatively low bit-width groups to be guided by relatively high bit-width groups, promoting information exchange and mutual learning between different bit-width groups, and stabilizing the quantization optimization process.

[0030] As another possible implementation, different from the aforementioned implementation, after obtaining the output features of the network block at the candidate bit width, element-wise fusion can be performed based on the output features of the same network block at different candidate bit widths according to the target probability to obtain the fused features of the corresponding network block. Simultaneously, sample data can be directly processed based on the network blocks in the data processing model to obtain the output features of the network block at the original precision (e.g., FP32, i.e., 32-bit floating-point number) as reference features. For example, each element position can be selected as the current position. For instance, taking an N-bit output feature as an example, the 1st, 2nd, ..., Nth bits can be selected as the current position. Based on this, it can be determined whether the current position needs to undergo fusion operation based on the target probability (e.g., 50%). That is, in the above example, N bits of the output feature have N... The p-th bit (where p represents the target probability) needs to undergo a fusion operation, while N bits are involved. (1-p) bits do not require fusion. Thus, in response to a fusion operation requiring the current position, the element values ​​at the current position in the output features of the same network block at different candidate bit widths are fused (e.g., averaged, weighted, etc.) to obtain the element value of the fused feature at the current position. Conversely, in response to a fusion operation not requiring the current position, the element value of the fused feature at the current position is obtained based on the element value of the reference feature at the current position. For ease of description, let's take the example of dividing different quantization bit widths into three combinations: low, medium, and high. Then, for any position of the output feature, the element value after the fusion operation can be represented as: , In the above formula, f represents the element value of the reference feature at the corresponding position. This indicates that the network block outputs the element value of the feature at the corresponding position with a low bit width. This indicates the element value of the output feature at the corresponding position of the network block with the median bit width. This indicates the element value of the output feature at the corresponding position in the high-bit width of the network block. This represents the element value at the corresponding position in the output features under different candidate bit widths after performing the fusion operation. Furthermore, λ1, λ2, and λ3 represent the weight values ​​of each candidate bit width when performing element value fusion using a weighted operation, p represents the target probability, and MFM stands for Multi-Bit Width Mixing Operation. Based on this, the training loss can be obtained by considering the differences between the output features of the same network block under different candidate bit widths, and the differences between the fused features and the reference features of the same network block. The initial sub-parameters can then be adjusted based on the training loss. For ease of description, the training loss L... recon It can be represented as: , In the above formula, MSE represents the mean square error function, O mixed This represents the fused features of the network block outputting features at multiple candidate bit widths, O fp32 This represents the output features of the network block (i.e., the aforementioned reference features) with an original precision of 32-bit floating-point. h This represents the output characteristics of a network block at its high bit width, O m O represents the output characteristics of a network block at the median width. lThis represents the output feature of the network block at a low bit width. Furthermore, γ1, γ2, and γ3 represent the weight values ​​of the three differences. The above method fuses the output features of the same network block at different candidate bit widths element-wise according to the target probability to obtain the fused feature of the corresponding network block. Then, based on the network block in the data processing model, the sample data is directly processed to obtain the output feature of the network block at its original precision as a reference feature. Thus, based on the differences between the output features of the same network block at different candidate bit widths, and the differences between the fused feature and the reference feature of the same network block, the training loss is obtained. Subsequently, based on the training loss, the initial sub-parameters are adjusted. Since multi-bit width feature mixing can fuse the output features obtained at different quantization bit widths, it helps to fully utilize the advantages of each bit width, rather than just favoring one bit width. Furthermore, by replacing the element values ​​of the fused bit widths with the element values ​​of the original precision output features at each element position with a certain probability, the flatness of the training loss can be enhanced.

[0031] In a specific implementation scenario, during the adjustment of initial sub-parameters based on training loss, the quantization step size (e.g., the quantization step size for activation values) in the initial sub-parameters can be adjusted based on the loss gradient calculated from the training loss. Specifically, in response to the sample data being scaled by the quantization step size in the initial sub-parameters and not falling below the upper limit of the quantization step size, the loss gradient can be obtained based on the product of the partial derivative of the training loss and the upper limit of the quantization step size. In response to the sample data being scaled by the quantization step size in the initial sub-parameters and falling between the lower and upper limits of the quantization step size, the loss gradient can be obtained based on the product of the partial derivative of the training loss and the scaling loss. The scaling loss can be characterized as the accuracy loss of the sample data after scaling and rounding by the quantization step size in the initial sub-parameters. In response to the sample data being scaled by the quantization step size in the initial sub-parameters and not falling above the lower limit of the quantization step size, the loss gradient can be obtained based on the product of the partial derivative of the training loss and the lower limit of the quantization step size. For ease of description, the quantization step size in the initial sub-parameters (e.g., the quantization step size for activation values)... The loss gradient of ) can be expressed as: , In the above formula, L recon Indicates training loss, q represents the partial derivative of the training loss. + q represents the upper limit of the quantization step size. - Indicates the lower limit of the quantization step size. This represents the sample data after scaling by the quantization step size in the initial sub-parameters (specifically, x can be the activation value of the sample data after processing by the network block). Furthermore, This represents the scaling loss (i.e., the precision loss of the activation value after scaling and rounding by the quantization step size in the initial sub-parameters).

[0032] In a specific implementation scenario, the convergence objective of the rounding values ​​in the initial sub-parameters during training is to minimize the optimization function. This optimization function is negatively correlated with the optimization term that includes the rounding values, and the optimization term is exponentially related to an adjustable factor that gradually decreases with each training epoch. For ease of description, the optimization function can be expressed as: , In the above formula, This represents an optimization term that includes rounding values, where β represents an adjustable factor. During training, the adjustable factor β can be gradually decreased to allow the rounded values ​​to gradually converge to {0,1}. Furthermore, the initial rounded values ​​in the sub-parameters undergo normalization and stretching during training. The new rounded values ​​are then optimized based on whether they fall between the preset lower and upper rounding limits. If the new rounded value is between the preset lower and upper rounding limits, it remains unchanged. If the new rounded value is lower than the preset lower rounding limit, it is reset to the preset lower rounding limit. If the new rounded value is higher than the preset upper rounding limit, it is reset to the preset upper rounding limit. The preset lower rounding limit represents rounding down, and the preset upper rounding limit represents rounding up. For ease of description, this process can be represented as: , In the above formula, σ represents the normalization function (e.g., sigmoid). These represent the stretching parameters, which can be set to 1.1 and -0.1 respectively. Furthermore, the specific meaning of the clip function can be found in the aforementioned descriptions, and will not be repeated here.

[0033] It should be noted that the above processing operations can obtain the quantization sub-parameters of the network blocks in the data processing model under different quantization bit widths, so as to facilitate subsequent quantization processing.

[0034] Step S12: Based on the hardware parameters of the terminal device, select one of the quantization sub-parameters with a quantization bit width for the network block as the target sub-parameter of the network block.

[0035] In one implementation scenario, as a possible approach, in response to hardware parameters including battery level, a quantization bit width matching the battery level can be selected as the target bit width. It should be noted that the quantization bit width matching the battery level is directly related to the quantization sub-parameters under the target bit width, which serve as the target sub-parameters of the network block. For example, when the terminal device has a relatively high battery level, a relatively large quantization bit width can be selected; conversely, when the terminal device has a relatively low battery level, a relatively small quantization bit width can be selected. For instance, a mapping relationship between battery level and quantization bit width can be pre-set, allowing the appropriate quantization bit width to be selected in real-time based on the terminal device's battery level during data processing requests. Of course, in practical applications, hardware parameters can also include other parameters besides battery level, such as memory, video memory, the number of processor cores, and clock speed; the specific type of hardware parameters is not limited here.

[0036] In another implementation scenario, as another possible implementation method, unlike the aforementioned implementation methods, in practical applications, there may be situations where the target sub-parameter cannot be determined based on hardware parameters (e.g., the hardware parameters of the terminal device cannot be mapped to any quantization bit width, or the hardware parameters of the terminal device can be mapped to multiple quantization bit widths). In such cases, the quantization sub-parameter under one quantization bit width can be further selected as the target sub-parameter of the network block based on the configuration parameters of the terminal device executing the target task. It should be noted that the configuration parameters refer to the configuration requirements of the terminal device executing the target task (e.g., response speed, processing accuracy, etc.). As a possible example, in response to configuration parameters including response speed, a quantization bit width that matches the response speed can be selected as the target bit width. Furthermore, the quantization bit width that matches the response speed is negatively correlated with the response speed (i.e., the faster the required response speed, the smaller the corresponding quantization bit width can be, and vice versa). As another possible example, in response to configuration parameters including processing precision, a quantization bit width matching the required processing precision is selected as the target bit width. This target bit width is positively correlated with the required processing precision (i.e., the higher the required processing precision, the larger the corresponding quantization bit width can be, and vice versa). It should be noted that the quantization sub-parameters under the target bit width can also serve as the target sub-parameters of the network block.

[0037] In another implementation scenario, as yet another possible implementation method, distinct from the aforementioned methods, as mentioned earlier, in practical applications, there may be situations where the target sub-parameters cannot be determined based on hardware parameters. Therefore, further analysis can be performed on the network blocks in the target processing model based on the second calibration dataset of the target task to obtain the quantization sensitivity of each network block under different quantization bit widths. The quantization sensitivity under a given quantization bit width can be characterized as the quantization error relative to the original precision at that quantization bit width. Based on this, a Pareto front search can be performed under the constraint of the objective function, with the goal of minimizing the search function, to obtain the target sub-parameters of each network block in the target processing model. The objective function can be characterized as: the sum of the products of the quantization bit width and the spatial size of each network block does not exceed the target threshold. The spatial size can be characterized as the storage space occupied by the network parameters of the network block. The search function can be characterized as: the sum of the quantization sensitivities of each network block. The above approach, when the target sub-parameters cannot be determined based on hardware parameters, first analyzes the quantization sensitivity of network blocks under different quantization bit widths based on the second calibration dataset of the target task. Then, under the constraint of the objective function, Pareto front search is performed with the goal of minimizing the search function to obtain the target sub-parameters of each network block in the target processing model. This approach can find the most appropriate bit width configuration as much as possible, which helps to meet the size constraints of the target processing model as much as possible while ensuring the minimum overall sensitivity.

[0038] In a specific implementation scenario, the specific meaning of the second calibration dataset can be found in the aforementioned description of the first calibration dataset, and will not be repeated here.

[0039] In a specific implementation scenario, to measure the quantization sensitivity of network blocks at different quantization bit widths, the second calibration dataset can be directly processed based on the network blocks in the target processing model to obtain the output features of the network blocks at the original precision (e.g., the aforementioned 32-bit floating-point number) as the first feature. Then, based on the quantization sub-parameters of the network blocks in the target processing model at the quantization bit width, the second calibration dataset is quantized to obtain the output features of the network blocks at the quantization bit width as the second feature. Based on this, the quantization sensitivity of the network block at the quantization bit width can be obtained based on the distribution difference between the first and second features (e.g., KL divergence can be used to measure the distribution difference). For ease of description, the quantization sensitivity of the i-th network block at the quantization bit width b... i Quantization sensitivity Ω i (b i This can be represented as: , In the above formula, KL represents the KL divergence function. The expression before the minus sign in the KL divergence function represents the first feature, and the expression after the minus sign represents the second feature. Furthermore, the specific meanings of the other parameters can be found in the relevant formulas for the training objective mentioned above, and will not be repeated here. It should be noted that if the quantization sensitivity Ω... i (b i If the value is smaller, it indicates that the i-th network block is quantized to a quantization bit width of b. i At that time, the model output is not significantly different from the original precision model output, which means that the i-th network block has a quantization bit width of b. i Relatively insensitive, conversely, if the quantization sensitivity Ω i (b i If the value is large, it indicates that the i-th network block has a large quantization bit width b. i Relatively sensitive. The above method directly processes the second calibration dataset based on the network block in the target processing model to obtain the output feature of the network block at the original accuracy as the first feature. Then, based on the quantization sub-parameter of the network block in the target processing model at the quantization bit width, the second calibration dataset is quantized to obtain the output feature of the network block at the quantization bit width as the second feature. Finally, based on the distribution difference between the first feature and the second feature, the quantization sensitivity of the network block at the quantization bit width is obtained. This method can quickly and accurately measure the sensitivity of the network block to each quantization bit width.

[0040] In a specific implementation scenario, as mentioned earlier, the objective function is characterized by the sum of the products of the quantization bit width and the storage space of each network block not exceeding a target threshold, where the storage space represents the storage space occupied by the network parameters of the network block. It should be noted that the storage space, more precisely, is the size of the storage space occupied by the network parameters of the network block after quantization using the specified bit width. For ease of description, the objective function can be expressed as: , In the above formula, P i This represents the storage space occupied by the network parameters of the i-th network block after quantization using the quantization bit width, b. i Let s represent the quantization bit width of the i-th network block, L represent the total number of network blocks in the target processing model, and s represent the quantization bit width of the i-th network block. t This represents the target threshold. Furthermore, as mentioned earlier, the search function can be characterized as the sum of the quantization sensitivities of each network block. For ease of description, the search function can be expressed as: , In the above formula, This indicates that the i-th network block has a quantization bit width of b. iLet L represent the total number of network blocks in the target processing model, and L be the quantization sensitivity. Based on this, a Pareto front search can be performed under the constraint of the objective function, aiming to minimize the search function, to obtain the target sub-parameters of each network block in the target processing model. It should be noted that the main idea is to allocate higher quantization bit widths to network blocks with high sensitivity and lower quantization bit widths to those with low sensitivity. Specifically, in the initial selection phase, several bit width configurations (e.g., 500) close to the target uniform bit width can be selected, and all their sensitivities can be obtained. Then, under the constraints of sensitivity and average model size, a Pareto front search is performed to obtain the optimal bit width configuration. Furthermore, the specific process of the Pareto front search can be found in the technical details of Pareto front search, which will not be elaborated here.

[0041] It should be noted that, although the above three implementation methods for selecting the quantization bit width for network blocks were described earlier as being able to attempt to determine the quantization bit width based on configuration parameters or quantization sensitivity when hardware parameters cannot determine it, in practical applications, the above three implementation methods can also be executed in parallel. Furthermore, as a possible example, if the quantization bit widths determined by the three methods conflict when executed in parallel, the hardware parameters can be prioritized.

[0042] Step S13: Based on the target sub-parameters of each network block in the target processing model, quantize the data to be processed requested by the data processing request to obtain the data processing result.

[0043] Specifically, after determining a quantization sub-parameter with a quantization bit width for each network block in the target processing model, and using it as the target sub-parameter of the network block, quantization processing can be performed on the data to be processed accordingly. As mentioned earlier, during the quantization process, for each network block, the model weights of the network block can be quantized based on the rounded value in its target sub-parameter and the quantization step size of the model weights. The activation values ​​during the processing of the network block are also quantized based on the quantization step size of the activation values ​​in its target sub-parameter, ultimately obtaining the output features of the network block, which serve as the input features of the next network block. This process is repeated until the last network block, thus obtaining the data processing result. In this way, since the target sub-parameter of each network block during quantization processing can be adapted to the hardware parameters of the terminal device, the hardware resources required by the terminal device during quantization processing can also be adapted to its own hardware parameters as much as possible.

[0044] Furthermore, as mentioned earlier, when the target task is an image processing task, the data to be processed can be image data, and the data processing result can vary depending on the specific type of image processing task (e.g., for an image segmentation task, the data processing result can be the mask region of each instance object in the image data); similarly, when the target task is an audio processing task, the data to be processed can be audio data, and the data processing result can vary depending on the specific type of audio processing task (e.g., for a speech recognition task, the data processing result can be the recognized text of the audio data); similarly, when the target task is a video processing task, the data to be processed can be video data (in some cases...). The data to be processed can also be other types of data. For example, in a video generation task, the data to be processed can be text, images, etc., to generate video data (i.e., the data processing result can be video data). In this case, the data processing result will vary depending on the specific type of video processing task (e.g., for a target tracking task, the data processing result can be a tracking sequence of the target object). Similarly, when the target task is a natural language processing task, the data to be processed can be a conversation sequence (e.g., a question text sequence). In this case, the data processing result can vary depending on the specific type of natural language processing task (e.g., for an intelligent question answering task, the data processing result can be a sequence of answer texts from a question text sequence). Of course, the above examples are just a few possible scenarios in practical applications. We will not limit the possible situations here, nor will we list them all.

[0045] The above scheme, in response to the terminal device receiving a data processing request for the target task, loads a data processing model suitable for the target task as the target processing model. This model is pre-trained with quantization parameters, which include quantization sub-parameters for network blocks in the data processing model at different quantization bit widths. These sub-parameters include rounding values ​​and quantization step sizes. Based on the terminal device's hardware parameters, a quantization sub-parameter at one of the quantization bit widths is selected for each network block as its target sub-parameter. Then, based on the target sub-parameters of each network block in the target processing model, the data to be processed in the data processing request is quantized to obtain the data processing result. Since the data processing model has been pre-trained with quantization parameters... The model quantization parameters are obtained through training, and these parameters include quantization sub-parameters for network blocks in the data processing model at different quantization bit widths. Therefore, when a terminal device executes a data processing request for the target task, it only needs to load a data processing model suitable for the target task first, and then, combined with its own hardware parameters, select a quantization sub-parameter at one of the quantization bit widths for each network block of the data processing model as the target sub-parameter. This ensures that the quantization sub-parameters of the network blocks in the data processing model loaded on the terminal device are as compatible as possible with the terminal device's own hardware parameters during subsequent quantization processing. Consequently, the hardware resources required by the terminal device when performing quantization processing on the data to be processed are also adapted to its own hardware parameters as much as possible. Therefore, this improves the success rate of the terminal device executing data processing requests.

[0046] Please see Figure 2 , Figure 2 This is a flowchart illustrating another embodiment of the request-response method of this application. It should be noted that the process operations in this embodiment can be executed by a terminal device with computing capabilities, and the specific structure or type of the terminal device is not limited here. Furthermore, the main difference between this embodiment and the foregoing embodiments is that this embodiment focuses on the specific application of the request-response method in natural language processing tasks. Other similarities can be directly referred to in the foregoing embodiments, and will not be repeated here. Specifically, this embodiment may include the following steps: Step S21: In response to the terminal device receiving a data processing request for a natural language processing task, load a data processing model suitable for the natural language processing task as the target processing model.

[0047] In this embodiment, the data processing model is pre-trained using quantization to obtain model quantization parameters. These parameters include quantization sub-parameters for network blocks in the data processing model at different quantization bit widths. Each sub-parameter includes a rounding value and a quantization step size. It should be noted that the specific process of quantization training for the data processing model, the specific type of terminal device, and the specific meanings of network blocks, rounding values, and quantization step sizes can be found in the descriptions in the foregoing embodiments and will not be repeated here.

[0048] In one implementation scenario, for natural language processing tasks, the suitable data processing model can include, but is not limited to, language models such as BERT (Bidirectional Encoder Representations from Transformers), large language models such as Llama, and multimodal large models such as Qwen3-VL. Here, no limitation is made on the data processing model suitable for natural language processing tasks. Furthermore, in natural language processing tasks, the network blocks in the data processing model can specifically include, but are not limited to, Transformer blocks.

[0049] In one implementation scenario, within a natural language processing task, the data processing request may contain a first conversation sequence. It should be noted that the first conversation sequence can contain several tokens. For example, taking a question-and-answer dialogue in a natural language processing task as an example, the first conversation sequence could be a sequence of question texts. Other scenarios can be deduced similarly, and will not be listed here.

[0050] Step S22: Based on the hardware parameters of the terminal device, select one of the quantization sub-parameters with a quantization bit width for the network block as the target sub-parameter of the network block.

[0051] For details, please refer to the relevant descriptions in the foregoing disclosed embodiments, which will not be repeated here.

[0052] Step S23: Based on the target sub-parameters of each network block in the target processing model, quantize the first session sequence requested by the data processing request to obtain the second session sequence used to reply to the first session sequence.

[0053] Specifically, the detailed process of quantization can be found in the descriptions of the aforementioned disclosed embodiments, and will not be repeated here. Furthermore, in natural language processing tasks, the second conversation sequence, similar to the first conversation sequence, may also contain several tokens. For example, taking a question-and-answer dialogue in a natural language processing task as an example, the second conversation sequence could be a sequence of answer texts. Other scenarios can be deduced similarly, and will not be listed here.

[0054] The above scheme, in response to a data processing request for a natural language processing task received by a terminal device, loads a data processing model suitable for the natural language processing task as the target processing model. The data processing model is pre-trained with quantization parameters, which include quantization sub-parameters for network blocks in the data processing model at different quantization bit widths. These sub-parameters include rounding values ​​and quantization step sizes. Based on the hardware parameters of the terminal device, a quantization sub-parameter at one quantization bit width is selected for each network block as its target sub-parameter. Then, based on the target sub-parameters of each network block in the target processing model, the first session sequence requested by the data processing request is quantized to obtain a second session sequence used to respond to the first session sequence. Since the data processing model... The model's quantization parameters are obtained through pre-quantization training, and these parameters include quantization sub-parameters for network blocks in the data processing model at different quantization bit widths. Therefore, when a terminal device executes a data processing request for a natural language processing task, it only needs to load a data processing model suitable for the task first, and then, based on its own hardware parameters, select a quantization sub-parameter at one of the quantization bit widths as the target sub-parameter for the network blocks of the data processing model. This ensures that the quantization sub-parameters of the network blocks of the data processing model loaded on the terminal device are as compatible as possible with the terminal device's own hardware parameters during subsequent quantization processing. Consequently, the hardware resources required by the terminal device when performing quantization processing on the first session sequence are also adapted to its own hardware parameters as much as possible. Therefore, the success rate of the terminal device executing data processing requests can be improved.

[0055] Please see Figure 3 , Figure 3 This is a schematic diagram of a framework of an embodiment of the request response device of this application. The request response device 30 includes: a first model loading module 31, a first parameter selection module 32, and a first quantization processing module 33. The first model loading module 31 is used to load a data processing model suitable for the target task as the target processing model in response to the terminal device receiving a data processing request for the target task. The data processing model is pre-trained with quantization parameters, which include quantization sub-parameters of network blocks in the data processing model at different quantization bit widths. The quantization sub-parameters include rounding values ​​and quantization step sizes. The first parameter selection module 32 is used to select a quantization sub-parameter of one of the quantization bit widths for the network block based on the hardware parameters of the terminal device as the target sub-parameter of the network block. The first quantization processing module 33 is used to quantize the data to be processed requested by the data processing request based on the target sub-parameters of each network block in the target processing model to obtain the data processing result.

[0056] In the above scheme, the request response device 30 responds to the terminal device receiving a data processing request for the target task, loads a data processing model suitable for the target task as the target processing model, and the data processing model is pre-trained with quantization parameters. The model quantization parameters include quantization sub-parameters of network blocks in the data processing model at different quantization bit widths. The quantization sub-parameters include rounding values ​​and quantization step sizes. Based on the hardware parameters of the terminal device, a quantization sub-parameter at one of the quantization bit widths is selected for the network block as the target sub-parameter of the network block. Then, based on the target sub-parameters of each network block in the target processing model, the data to be processed requested by the data processing request is quantized to obtain the data processing result. First, quantization training is performed to obtain the model's quantization parameters. These parameters include quantization sub-parameters for network blocks within the data processing model at different quantization bit widths. Therefore, when a terminal device executes a data processing request for the target task, it only needs to load a data processing model suitable for the target task. Then, combined with its own hardware parameters, it selects a quantization sub-parameter at one of the quantization bit widths for each network block of the data processing model as the target sub-parameter. This ensures that the quantization sub-parameters of the network blocks in the data processing model loaded on the terminal device are as compatible as possible with the terminal device's own hardware parameters during subsequent quantization processing. Consequently, the hardware resources required by the terminal device during quantization processing of the data to be processed are also adapted to its own hardware parameters as much as possible. Therefore, the success rate of the terminal device executing data processing requests can be improved.

[0057] In some disclosed embodiments, the first parameter selection module 32 includes a first response submodule, which, in response to hardware parameters including power level, selects a quantization bit width that matches the power level as a target bit width based on the power level; wherein the quantization bit width that matches the power level is positively correlated with the power level, and the quantization sub-parameters under the target bit width are used as target sub-parameters of the network block.

[0058] In some disclosed embodiments, the first parameter selection module 32 is further configured to select one of the quantization sub-parameters with a quantization bit width as the target sub-parameter of the network block based on the configuration parameters of the terminal device executing the target task when the target sub-parameter cannot be determined based on the hardware parameters.

[0059] In some disclosed embodiments, the first parameter selection module 32 includes a second response submodule, used to select a quantization bit width matching the speed of the response based on the speed of the response, in response to configuration parameters including response speed; wherein the quantization bit width matching the speed of the response is negatively correlated with the speed of the response. The first parameter selection module 32 also includes a third response submodule, used to select a quantization bit width matching the processing precision based on the processing precision level, in response to configuration parameters including processing precision, as the target bit width; wherein the quantization bit width matching the processing precision is positively correlated with the processing precision level; wherein the quantization sub-parameters under the target bit width are used as target sub-parameters of the network block.

[0060] In some disclosed embodiments, the request response device 30 includes an initial partitioning module, used to initialize the initial quantization parameters of the data processing model based on a first calibration dataset of the task to which the data processing model is applicable, and to partition different quantization bit widths based on the bit width size to obtain multiple bit width intervals; wherein, the initial quantization parameters include initial sub-parameters of network blocks in the data processing model under different quantization bit widths; the request response device 30 includes a bit width combination module, used to select each quantization bit width belonging to different bit width intervals as candidate bit widths; the request response device 30 includes a sample processing module, used to quantize the sample data of the task to which the data processing model is applicable based on the initial sub-parameters of network blocks in the data processing model under the candidate bit widths to obtain the output features of the network blocks under the candidate bit widths; the request response device 30 includes a parameter adjustment module, used to adjust the initial sub-parameters based on the training loss measured by the difference between the output features of the same network blocks under different candidate bit widths; wherein, the latest initial sub-parameters after the data processing model training converges are used as quantization sub-parameters.

[0061] In some disclosed embodiments, the request response device 30 includes a fusion processing module, which is used to perform element-wise fusion of the output features of the same network block under different candidate bit widths according to the target probability to obtain the fusion features of the corresponding network block, and to directly process the sample data based on the network block in the data processing model to obtain the output features of the network block at the original accuracy as reference features; the parameter adjustment module is used to obtain the training loss based on the difference between the output features of the same network block under different candidate bit widths, and the difference between the fusion features of the same network block and the reference features, and to adjust the initial sub-parameters based on the training loss.

[0062] In some disclosed embodiments, the fusion processing module includes a position selection submodule for selecting each element position as the current position; the fusion processing module includes a determination execution submodule for determining whether a fusion operation needs to be performed at the current position based on the target probability; the fusion processing module includes an element fusion submodule for fusing the element values ​​at the current position in the output features of the same network block at different candidate bit widths in response to the need for a fusion operation at the current position, thereby obtaining the element value of the fused feature at the current position; and the fusion processing module includes a direct acquisition submodule for obtaining the element value of the fused feature at the current position based on the element value of the reference feature at the current position in response to the need for a fusion operation at the current position.

[0063] In some disclosed embodiments, the parameter adjustment module includes a bit width selection submodule for selecting adjacent candidate bit widths as a bit width combination based on sequentially arranged different candidate bit widths; the parameter adjustment module includes a loss measurement submodule for obtaining the sub-loss corresponding to the bit width combination based on the difference between the output features of the same network block under different candidate bit widths in the bit width combination; the parameter adjustment module includes a loss fusion submodule for fusing the sub-losses of each bit width combination to obtain the training loss; and the parameter adjustment module includes a parameter adjustment submodule for adjusting the initial sub-parameters based on the training loss.

[0064] In some disclosed embodiments, the quantization step size in the initial sub-parameters is adjusted based on the loss gradient calculated from the training loss. The parameter adjustment module includes a first calculation sub-module, used to obtain the loss gradient based on the product of the partial derivative of the training loss and the upper limit of the quantization step size, in response to the sample data being scaled by the quantization step size in the initial sub-parameters and not falling below the upper limit of the quantization step size. The parameter adjustment module also includes a second calculation sub-module, used to obtain the loss gradient based on the product of the partial derivative of the training loss and the scaling loss, in response to the sample data being scaled by the quantization step size in the initial sub-parameters and falling between the lower limit and the upper limit of the quantization step size, in response to the sample data being scaled by the quantization step size in the initial sub-parameters and falling between the lower limit and the upper limit of the quantization step size. Herein, the scaling loss represents the accuracy loss of the sample data after scaling and rounding by the quantization step size in the initial sub-parameters. The parameter adjustment module also includes a third calculation sub-module, used to obtain the loss gradient based on the product of the partial derivative of the training loss and the lower limit of the quantization step size, in response to the sample data being scaled by the quantization step size in the initial sub-parameters and not falling above the lower limit of the quantization step size.

[0065] In some disclosed embodiments, the convergence objective of the rounding value in the initial sub-parameters during the training process is to minimize the optimization function. The optimization function is negatively correlated with the optimization term that includes the rounding value, and the optimization term is exponentially sized by an adjustable factor that gradually decreases with each training round.

[0066] In some disclosed embodiments, the rounded values ​​in the initial sub-parameters are normalized and stretched sequentially during training. The new rounded values ​​are then optimized based on whether they are between a preset lower rounding limit and a preset upper rounding limit. If the new rounded value is between the preset lower rounding limit and the preset upper rounding limit, the new rounded value remains unchanged. If the new rounded value is lower than the preset lower rounding limit, the new rounded value is reset to the preset lower rounding limit. If the new rounded value is higher than the preset upper rounding limit, the new rounded value is reset to the preset upper rounding limit. The preset lower rounding limit represents rounding down, and the preset upper rounding limit represents rounding up.

[0067] In some disclosed embodiments, the request response device 30 includes a sensitivity analysis module, used to analyze network blocks in the target processing model based on a second calibration dataset of the target task when the target sub-parameters cannot be determined based on hardware parameters, to obtain the quantization sensitivity of the network blocks at different quantization bit widths; wherein, the quantization sensitivity at the quantization bit width is characterized as: the quantization error at the quantization bit width compared to the original precision; the request response device 30 includes a parameter search module, used to perform a Pareto front search under the constraint of an objective function with the goal of minimizing the search function, to obtain the target sub-parameters of each network block in the target processing model; wherein, the objective function is characterized as: the sum of the products of the quantization bit width and the spatial size of each network block is not higher than the target threshold, the spatial size is characterized as the storage space occupied by the network parameters of the network block, and the search function is characterized as: the sum of the quantization sensitivities of each network block.

[0068] In some disclosed embodiments, the sensitivity analysis module includes a feature acquisition submodule, which is used to directly process the second calibration dataset based on the network block in the target processing model to obtain the output feature of the network block at the original accuracy as the first feature, and to quantize the second calibration dataset based on the quantization sub-parameter of the network block in the target processing model at the quantization bit width to obtain the output feature of the network block at the quantization bit width as the second feature; the sensitivity analysis module includes a sensitivity measurement submodule, which is used to obtain the quantization sensitivity of the network block at the quantization bit width based on the distribution difference between the first feature and the second feature.

[0069] In some disclosed embodiments, the hardware parameters include at least the power level; and / or, different quantization bit widths include positive integers between a preset lower bit width limit and a preset upper bit width limit; and / or, the rounding value after quantization training is a preset lower rounding limit or a preset upper rounding limit, where the preset lower rounding limit represents rounding down and the preset upper rounding limit represents rounding up; and / or, the target task belongs to any one of image processing, audio processing, video processing, or natural language processing tasks.

[0070] Please see Figure 4 , Figure 4This is a schematic diagram of another embodiment of the request response device of this application. The request response device 40 includes: a second model loading module 41, a second parameter selection module 42, and a second quantization processing module 43. The second model loading module 41 is used to load a data processing model suitable for the natural language processing task as the target processing model in response to the terminal device receiving a data processing request for a natural language processing task. The data processing model is pre-trained with quantization parameters, which include quantization sub-parameters of network blocks in the data processing model at different quantization bit widths. The quantization sub-parameters include rounding values ​​and quantization step sizes. The second parameter selection module 42 is used to select a quantization sub-parameter of one of the quantization bit widths for a network block based on the hardware parameters of the terminal device as the target sub-parameter of the network block. The second quantization processing module 43 is used to quantize the first session sequence requested by the data processing request based on the target sub-parameters of each network block in the target processing model to obtain a second session sequence for replying to the first session sequence.

[0071] In the above scheme, the request response device 40 responds to the terminal device receiving a data processing request for a natural language processing task, loads a data processing model suitable for the natural language processing task as the target processing model, and the data processing model is pre-trained with quantization parameters. These quantization parameters include quantization sub-parameters for network blocks in the data processing model at different quantization bit widths. These sub-parameters include rounding values ​​and quantization step sizes. Based on the hardware parameters of the terminal device, a quantization sub-parameter at one of the quantization bit widths is selected for the network block as its target sub-parameter. Then, based on the target sub-parameters of each network block in the target processing model, the first session sequence requested by the data processing request is quantized to obtain a second session sequence used to respond to the first session sequence. Since the data processing request... The data processing model is pre-trained with quantization parameters, which include quantization sub-parameters for network blocks within the model at different quantization bit widths. Therefore, when a terminal device executes a data processing request for a natural language processing task, it only needs to load a data processing model suitable for the task and then, based on its own hardware parameters, select a quantization sub-parameter at one of the quantization bit widths as the target sub-parameter for each network block. This ensures that the quantization sub-parameters of the network blocks in the data processing model loaded on the terminal device are as compatible as possible with the terminal device's hardware parameters during subsequent quantization processing. Consequently, the hardware resources required by the terminal device when performing quantization processing on the first session sequence are also adapted to its own hardware parameters as much as possible. Therefore, this improves the success rate of the terminal device executing data processing requests.

[0072] Please see Figure 5 , Figure 5This is a schematic diagram of a framework of an embodiment of the electronic device of this application. The electronic device 50 includes at least a memory 51 and a processor 52 coupled to each other. The memory 51 stores at least program instructions, and the processor 52 is used to execute the program instructions to implement the steps in any of the above-described request-response method embodiments. For details, please refer to the foregoing disclosed embodiments, which will not be repeated here.

[0073] Specifically, processor 52 controls itself and memory 51 to implement the steps in any of the above-described request-response method embodiments. Processor 52 may also be referred to as a CPU (Central Processing Unit). Processor 52 may be an integrated circuit chip with signal processing capabilities. Processor 52 may also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. A general-purpose processor may be a microprocessor or any conventional processor. Furthermore, processor 52 may be implemented using integrated circuit chips.

[0074] In the above scheme, the electronic device 50, in response to the terminal device receiving a data processing request for the target task, loads a data processing model suitable for the target task as the target processing model. This data processing model is pre-trained with quantization parameters, which include quantization sub-parameters for network blocks in the data processing model at different quantization bit widths. These sub-parameters include rounding values ​​and quantization step sizes. Based on the hardware parameters of the terminal device, the electronic device 50 selects a quantization sub-parameter at one of the quantization bit widths for each network block as its target sub-parameter. Then, based on the target sub-parameters of each network block in the target processing model, the electronic device 50 quantizes the data to be processed in the data processing request to obtain the data processing result. Since the data processing model is pre-trained with quantization parameters, the electronic device 50 performs quantization processing on the data to be processed in the data processing request, thus obtaining the data processing result. The model's quantization parameters are obtained through quantization training. These parameters include quantization sub-parameters for network blocks in the data processing model at different quantization bit widths. Therefore, when a terminal device executes a data processing request for the target task, it only needs to load a data processing model suitable for the target task first, and then, combined with its own hardware parameters, select a quantization sub-parameter at one of the quantization bit widths for the network blocks of the data processing model as the target sub-parameter. This ensures that the quantization sub-parameters of the network blocks of the data processing model loaded on the terminal device are as compatible as possible with the terminal device's own hardware parameters during subsequent quantization processing. Consequently, the hardware resources required by the terminal device when performing quantization processing on the data to be processed are also adapted to its own hardware parameters as much as possible. Therefore, the success rate of the terminal device executing data processing requests can be improved.

[0075] Please see Figure 6 , Figure 6 This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium of this application. The computer-readable storage medium 60 stores program instructions 61 that can be executed by a processor. The program instructions 61 are used to implement the steps in any of the above-described request-response method embodiments.

[0076] In the above scheme, the computer-readable storage medium 60, in response to the terminal device receiving a data processing request for the target task, loads a data processing model suitable for the target task as the target processing model. This data processing model is pre-trained with quantization parameters, which include quantization sub-parameters for network blocks in the data processing model at different quantization bit widths. These sub-parameters include rounding values ​​and quantization step sizes. Based on the hardware parameters of the terminal device, a quantization sub-parameter at one of the quantization bit widths is selected for each network block as its target sub-parameter. Then, based on the target sub-parameters of each network block in the target processing model, the data to be processed requested by the data processing request is quantized to obtain the data processing result. The model is pre-trained with quantization parameters, which include quantization sub-parameters for network blocks in the data processing model at different quantization bit widths. Therefore, when a terminal device executes a data processing request for the target task, it only needs to load a data processing model suitable for the target task first, and then, based on its own hardware parameters, select a quantization sub-parameter at one of the quantization bit widths for each network block of the data processing model as the target sub-parameter. This ensures that the quantization sub-parameters of the network blocks in the data processing model loaded on the terminal device are as compatible as possible with the terminal device's own hardware parameters during subsequent quantization processing. Consequently, the hardware resources required by the terminal device during the quantization processing of the data to be processed are also adapted to its own hardware parameters as much as possible. Therefore, this improves the success rate of the terminal device executing data processing requests.

[0077] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0078] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0079] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0080] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0081] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0082] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0083] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

Claims

1. A request response method characterized by, The method comprises: in response to the terminal device receiving a data processing request of a target task, loading a data processing model suitable for the target task as a target processing model; wherein the data processing model is pre-trained to obtain model quantization parameters, the model quantization parameters comprising quantization sub-parameters of network blocks in the data processing model respectively under different quantization bit widths, the quantization sub-parameters comprising rounding values and quantization steps; based on the hardware parameters of the terminal device, selecting a quantization sub-parameter under one of the quantization bit widths as a target sub-parameter of the network block; based on the target sub-parameters of each network block in the target processing model, performing quantization processing on to-be-processed data requested to be processed by the data processing request to obtain a data processing result.

2. The method of claim 1, wherein, The method comprises: in response to the hardware parameters comprising power, based on the high or low degree of the power, selecting a quantization bit width matching the high or low degree as a target bit width; wherein the quantization bit width matching the high or low degree of the power is positively correlated with the high or low degree of the power, and the quantization sub-parameter under the target bit width is the target sub-parameter of the network block.

3. The method of claim 1, wherein, In the case where the target sub-parameter cannot be determined based on the hardware parameters, the method further comprises: based on the configuration parameters of the terminal device for executing the target task, selecting a quantization sub-parameter under one of the quantization bit widths as a target sub-parameter of the network block.

4. The method of claim 3, wherein, The method comprises: in response to the configuration parameters comprising response speed, based on the fast or slow degree of the response speed, selecting a quantization bit width matching the fast or slow degree as a target bit width; wherein the quantization bit width matching the fast or slow degree of the response speed is negatively correlated with the fast or slow degree of the response speed; in response to the configuration parameters comprising processing precision, based on the high or low degree of the processing precision, selecting a quantization bit width matching the high or low degree as a target bit width; wherein the quantization bit width matching the high or low degree of the processing precision is positively correlated with the high or low degree of the processing precision; wherein the quantization sub-parameter under the target bit width is the target sub-parameter of the network block.

5. The method of claim 1, wherein, The method comprises: based on a first calibration data set of a task to which the data processing model is applicable, initializing to obtain initial quantization parameters of the data processing model, and dividing different quantization bit widths based on bit width sizes to obtain a plurality of bit width intervals; wherein the initial quantization parameters comprise initial sub-parameters of the network blocks in the data processing model respectively under different quantization bit widths; selecting each quantization bit width belonging to a different bit width interval as a candidate bit width; quantize sample data of a task to which the data processing model is applied based on the initial sub-parameters of the network block at the candidate bit-widths in the data processing model, to obtain output features of the network block at the candidate bit-widths; adjust the initial sub-parameters based on a training loss obtained by measuring differences between the output features of the same network block at different candidate bit-widths; wherein the latest initial sub-parameters of the data processing model after training convergence are taken as the quantized sub-parameters.

6. The method of claim 5, wherein, After the quantization of the sample data of the task to which the data processing model is applied based on the initial sub-parameters of the network block at the candidate bit-widths in the data processing model, and before the adjustment of the initial sub-parameters based on the training loss obtained by measuring differences between the output features of the same network block at different candidate bit-widths, the method further comprises: fuse the output features of the same network block at different candidate bit-widths element by element according to a target probability to obtain fused features corresponding to the network block, and directly process the sample data based on the network block in the data processing model to obtain output features of the network block at original precision as reference features; the adjustment of the initial sub-parameters based on the training loss obtained by measuring differences between the output features of the same network block at different candidate bit-widths, comprises: obtaining the training loss based on differences between the output features of the same network block at different candidate bit-widths and differences between the fused features of the same network block and the reference features; adjusting the initial sub-parameters based on the training loss.

7. The method of claim 6, wherein, the fusion of the output features of the same network block at different candidate bit-widths element by element according to a target probability to obtain fused features corresponding to the network block, comprises: selecting each element position as a current position; determining whether the current position needs to perform a fusion operation based on the target probability; in response to the current position needing to perform a fusion operation, fusing element values of the current position in the output features of the same network block at different candidate bit-widths to obtain an element value of the current position in the fused features; in response to the current position not needing to perform a fusion operation, obtaining an element value of the current position in the fused features based on an element value of the current position in the reference features.

8. The method of claim 5, wherein, the adjustment of the initial sub-parameters based on the training loss obtained by measuring differences between the output features of the same network block at different candidate bit-widths, comprises: selecting adjacent candidate bit-widths as a bit-width combination based on different candidate bit-widths arranged in sequence; obtaining a sub-loss corresponding to the bit-width combination based on differences between the output features of the same network block at different candidate bit-widths in the bit-width combination; fusing the sub-losses of each bit-width combination to obtain the training loss; adjusting the initial sub-parameters based on the training loss.

9. The method of claim 5, wherein, The initial sub-parameter quantization step is adjusted based on a loss gradient calculated from the training loss, and the calculation of the loss gradient comprises: In response to the sample data being scaled by the initial sub-parameter quantization step being no lower than the upper limit of the quantization step, the loss gradient is obtained based on the product of the partial derivative of the training loss and the upper limit of the quantization step; In response to the sample data being scaled by the initial sub-parameter quantization step being between the lower limit of the quantization step and the upper limit of the quantization step, the loss gradient is obtained based on the product of the partial derivative of the training loss and the scaling loss, wherein the scaling loss represents the accuracy loss of the sample data being scaled by the initial sub-parameter quantization step and being rounded; In response to the sample data being scaled by the initial sub-parameter quantization step being no higher than the lower limit of the quantization step, the loss gradient is obtained based on the product of the partial derivative of the training loss and the lower limit of the quantization step.

10. The method of claim 5, wherein, The convergence target of the initial sub-parameter rounding value in the training process is to minimize an optimization function, the optimization function is negatively correlated with an optimization term containing the rounding value in the optimization function, and the optimization term is exponentially related to an adjustable factor gradually decreasing with the training round; And / or, the initial sub-parameter rounding value in the training process is sequentially subjected to normalization and stretching to obtain a new rounding value, which is optimized based on whether it is between a preset rounding lower limit and a preset rounding upper limit, in the case that the new rounding value is between the preset rounding lower limit and the preset rounding upper limit, the new rounding value remains unchanged, in the case that the new rounding value is lower than the preset rounding lower limit, the new rounding value is reset to the preset rounding lower limit, in the case that the new rounding value is higher than the preset rounding upper limit, the new rounding value is reset to the preset rounding upper limit, and the preset rounding lower limit represents rounding down and the preset rounding upper limit represents rounding up.

11. The method of claim 1, wherein, In the case that the target sub-parameter cannot be determined based on the hardware parameter, the method further comprises: analyzing the network block in the target processing model based on a second calibration data set of the target task to obtain quantization sensitivities of the network block under different quantization bit widths, wherein the quantization sensitivity under the quantization bit width represents the quantization error under the quantization bit width compared to the original precision; performing a Pareto frontier search to minimize a search function under the constraint of a target function to obtain target sub-parameters of each network block in the target processing model, wherein the target function represents that the cumulative sum of the product of the quantization bit width and the space size of each network block is not higher than a target threshold, the space size represents the storage space occupied by the network parameters of the network block, and the search function represents the sum of the quantization sensitivities of each network block.

12. The method of claim 11, wherein, The analysis of the network block in the target processing model based on the second calibration data set of the target task to obtain the quantization sensitivities of the network block under different quantization bit widths comprises: The network block in the target processing model directly processes the second calibration data set to obtain output features of the network block at original precision as first features, and quantizes the second calibration data set based on the quantized sub-parameters of the network block at the quantization bit width in the target processing model to obtain output features of the network block at the quantization bit width as second features; The quantization sensitivity of the network block at the quantization bit width is obtained based on the distribution difference between the first features and the second features.

13. The method according to any one of claims 1 to 12, characterized in that, The hardware parameters at least include power; And / or, the different quantization bit widths include positive integers between a preset lower bit width limit and a preset upper bit width limit; And / or, the rounding value after the quantization training is a preset rounding lower limit or a preset rounding upper limit, the preset rounding lower limit represents rounding down, and the preset rounding upper limit represents rounding up; And / or, the target task belongs to any one of an image processing task, an audio processing task, a video processing task, and a natural language processing task.

14. A request response method characterized by, Comprising: In response to the terminal device receiving a data processing request of a natural language processing task, a data processing model suitable for the natural language processing task is loaded as a target processing model; wherein the data processing model is pre-quantized and trained to obtain model quantization parameters, the model quantization parameters include quantized sub-parameters of network blocks in the data processing model at different quantization bit widths, and the quantized sub-parameters include rounding values and quantization steps; Based on the hardware parameters of the terminal device, the quantized sub-parameters of the network block at one of the quantization bit widths are selected as target sub-parameters of the network block; Based on the target sub-parameters of each network block in the target processing model, a first conversation sequence requested to be processed by the data processing request is quantized to obtain a second conversation sequence for replying to the first conversation sequence.

15. A request response apparatus characterized by comprising: Comprising: A first model loading module is configured to load a data processing model suitable for a target task as a target processing model in response to a terminal device receiving a data processing request of the target task; wherein the data processing model is pre-quantized and trained to obtain model quantization parameters, the model quantization parameters include quantized sub-parameters of network blocks in the data processing model at different quantization bit widths, and the quantized sub-parameters include rounding values and quantization steps; A first parameter selection module is configured to select, based on hardware parameters of the terminal device, quantized sub-parameters of the network block at one of the quantization bit widths as target sub-parameters of the network block; A first quantization processing module is configured to quantize, based on target sub-parameters of each network block in the target processing model, to-be-processed data requested to be processed by the data processing request to obtain a data processing result.

16. A request response apparatus characterized by comprising: Comprising: A second model loading module loads a data processing model suitable for the natural language processing task as a target processing model in response to the terminal device receiving a data processing request of the natural language processing task; wherein the data processing model is obtained by pre-quantitative training of model quantitative parameters, the model quantitative parameters include quantitative sub-parameters of network blocks in the data processing model at different quantitative bit widths, and the quantitative sub-parameters include rounding values and quantitative steps; A second parameter selection module selects quantitative sub-parameters of one of the quantitative bit widths as target sub-parameters of the network blocks based on hardware parameters of the terminal device; A second quantitative processing module performs quantitative processing on a first conversation sequence requested to be processed by the data processing request based on the target sub-parameters of each network block in the target processing model to obtain a second conversation sequence for replying to the first conversation sequence.

17. An electronic device, comprising: At least comprising a memory and a processor coupled with each other, the memory at least stores program instructions, and the processor is configured to execute the program instructions to implement the request response method of any one of claims 1 to 14.

18. A computer-readable storage medium, characterized in that, The memory stores program instructions capable of being executed by the processor, and the program instructions are used to implement the request response method of any one of claims 1 to 14.

Images