Image processing method, related method and device, electronic equipment and storage medium

Abstract

The invention discloses an image processing method, a related method, a related device, electronic equipment and a storage medium, and the image processing method comprises the steps: extracting the image feature representation of a to-be-processed image, and carrying out the quantification of an image processing model disposed on the current equipment based on a quantification tool on the current equipment, and obtaining a first quantification model; wherein the image processing model is at least obtained through clipping training, clipping coefficients are decreased progressively in the clipping process, and the clipping coefficients are used for determining the numerical range of weight parameters in the clipping image processing model; processing the image feature representation based on a first quantization model to obtain a temporary processing result; and carrying out dequantization on the temporary processing result based on a quantization tool to obtain a final processing result of the to-be-processed image. According to the scheme, the reasoning speed of the neural network model can be ensured as much as possible, and the accuracy of the neural network model in task processing is improved.

Description

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on December 4, 2024, with application number 2024117727591 and application name “Image processing method and related methods, devices, electronic devices and storage media”, all contents of which are incorporated by reference in this application. Technical Field

[0002] The present application relates to the field of artificial intelligence technology, and in particular to an image processing method and related methods, devices, electronic devices and storage media. Background Art

[0003] In recent years, neural network models represented by tasks such as images, speech, and natural language have shown great application value. After training, neural network models are usually deployed on servers or edge (embedded) devices for inference.

[0004] Normally, after floating-point training, neural network models need to be converted into low-bit models through quantization technology to improve the model reasoning speed. However, when deploying neural network models, such as image processing models, using existing technologies, there is still a high loss of precision, which makes it difficult to improve the accuracy of tasks such as image processing. In view of this, how to ensure the reasoning speed of neural network models as much as possible and improve the accuracy of neural network models in their processing tasks has become an urgent problem to be solved. Summary of the invention

[0005] The main technical problem solved by the present application is to provide an image processing method and related methods, devices, electronic devices and storage media, which can ensure the reasoning speed of the neural network model as much as possible and improve the accuracy of the neural network model in its processing tasks.

[0006] In order to solve the above-mentioned technical problems, the first aspect of the present application provides an image processing method, including: extracting an image feature representation of an image to be processed, and quantizing an image processing model deployed on the current device based on a quantization tool on the current device to obtain a first quantization model; wherein the image processing model is obtained at least through cropping training, and the cropping coefficient is reduced during the cropping process, and the cropping coefficient is used to determine the numerical range of a weight parameter in the cropped image processing model; processing the image feature representation based on the first quantization model to obtain a temporary processing result; and dequantizing the temporary processing result based on the quantization tool to obtain the final processing result of the image to be processed.

[0007] In order to solve the above-mentioned technical problems, the second aspect of the present application provides a speech processing method, including: extracting the speech feature representation of the speech to be processed, and quantizing the speech processing model deployed on the current device based on the quantization tool on the current device to obtain a second quantization model; wherein the speech processing model is obtained by at least trimming training, and the trimming coefficient is reduced during the trimming process, and the trimming coefficient is used to determine the numerical range of the weight parameter in the trimmed speech processing model; processing the speech feature representation based on the second quantization model to obtain a temporary processing result; dequantizing the temporary processing result based on the quantization tool to obtain the final processing result of the speech to be processed.

[0008] In order to solve the above-mentioned technical problems, the third aspect of the present application provides a natural language processing method, including: extracting a sentence feature representation of a sentence to be processed, and quantizing a natural language processing model deployed on the current device based on a quantization tool on the current device to obtain a third quantization model; wherein the natural language processing model is obtained at least through trimming training, and the trimming coefficient is reduced during the trimming process, and the trimming coefficient is used to determine the numerical range of the weight parameter in the trimmed natural language processing model; processing the sentence feature representation based on the third quantization model to obtain a temporary processing result; and dequantizing the temporary processing result based on the quantization tool to obtain the final processing result of the sentence to be processed.

[0009] In order to solve the above-mentioned technical problems, the fourth aspect of the present application provides an image processing device, including: an extraction and quantization module, a model processing module and a dequantization module, the extraction and quantization module is used to extract the image feature representation of the image to be processed, and quantize the image processing model deployed on the current device based on the quantization tool on the current device to obtain a first quantization model; wherein the image processing model is obtained at least through cropping training, and the cropping coefficient is reduced during the cropping process, and the cropping coefficient is used to determine the numerical range of the weight parameter in the cropped image processing model; the model processing module is used to process the image feature representation based on the first quantization model to obtain a temporary processing result; the dequantization module is used to dequantize the temporary processing result based on the quantization tool to obtain the final processing result of the image to be processed.

[0010] In order to solve the above-mentioned technical problems, the fifth aspect of the present application provides a speech processing device, including: an extraction and quantization module, a model processing module and a dequantization module, the extraction and quantization module is used to extract the speech feature representation of the speech to be processed, and quantize the speech processing model deployed on the current device based on the quantization tool on the current device to obtain a second quantization model; wherein the speech processing model is obtained at least through trimming training, and the trimming coefficient is reduced during the trimming process, and the trimming coefficient is used to determine the numerical range of the weight parameter in the trimmed speech processing model; the model processing module is used to process the speech feature representation based on the second quantization model to obtain a temporary processing result; the dequantization module is used to dequantize the temporary processing result based on the quantization tool to obtain the final processing result of the speech to be processed.

[0011] In order to solve the above-mentioned technical problems, the sixth aspect of the present application provides a natural language processing device, including: an extraction and quantization module, a model processing module and a dequantization module, the extraction and quantization module is used to extract the sentence feature representation of the sentence to be processed, and quantize the natural language processing model deployed on the current device based on the quantization tool on the current device to obtain a third quantization model; wherein the natural language processing model is obtained at least through trimming training, and the trimming coefficient is reduced during the trimming process, and the trimming coefficient is used to determine the numerical range of the weight parameter in the trimmed natural language processing model; the model processing module is used to process the sentence feature representation based on the third quantization model to obtain a temporary processing result; the dequantization module is used to dequantize the temporary processing result based on the quantization tool to obtain the final processing result of the sentence to be processed.

[0012] In order to solve the above-mentioned technical problems, the seventh aspect of the present application provides an electronic device, comprising at least a memory and a processor coupled to each other, wherein the memory at least stores program instructions, and the processor is used to execute the program instructions to implement the image processing method in the above-mentioned first aspect, or to implement the speech processing method in the above-mentioned second aspect, or to implement the natural language processing method in the above-mentioned third aspect.

[0013] In order to solve the above-mentioned technical problems, the eighth aspect of the present application provides a computer-readable storage medium, storing program instructions that can be executed by a processor, and the program instructions are used to implement the image processing method of the first aspect, or to implement the speech processing method of the second aspect, or to implement the natural language processing method of the third aspect.

[0014] The above scheme extracts the image feature representation of the image to be processed, and quantizes the image processing model deployed on the current device based on the quantization tool on the current device to obtain a first quantization model, and the image processing model is at least obtained through cropping training, and the cropping coefficient is reduced during the cropping training process. The cropping coefficient is used to determine the numerical range of the weight parameter in the cropped image processing model, and then the image feature representation is processed based on the first quantization model to obtain a temporary processing result, and then the temporary processing result is dequantized based on the quantization tool to obtain the final processing result of the image to be processed. Then, on the one hand, by quantizing the neural network model deployed on the current device and then using it, the reasoning speed of the neural network model can be ensured as much as possible. On the other hand, the neural network model is at least cropped and trained, and because the cropping coefficient is dynamically reduced during the cropping training process, the weight parameter is gradually stabilized within the range determined by the final cropping coefficient after being cropped within the numerical range determined by the cropping system during the training process, so that the weight parameter is as close to a uniform distribution as possible, which helps to alleviate the accuracy loss caused by model quantization as much as possible, and can improve the accuracy of the neural network model in its processing tasks as much as possible. Therefore, the reasoning speed of the neural network model can be ensured as much as possible, and the accuracy of the neural network model in its processing tasks can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a flowchart of an embodiment of the image processing method of the present application; Figure 2 It is a flowchart of an embodiment of the speech processing method of the present application; Figure 3 This is a flow chart of an embodiment of a natural language processing method of the present application; Figure 4 It is a schematic diagram of the framework of an embodiment of the image processing device of the present application; Figure 5 It is a schematic diagram of the framework of an embodiment of a speech processing device of the present application; Figure 6 It is a schematic diagram of the framework of an embodiment of a natural language processing device of the present application; Figure 7 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application; Figure 8 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0016] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0017] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0018] The terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the fragment " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.

[0019] See also Figure 1 , Figure 1 This is a flow chart of an embodiment of the image processing method of the present application. Specifically, it may include the following steps: Step S11: extracting the image feature representation of the image to be processed, and quantizing the image processing model deployed on the current device based on the quantization tool on the current device to obtain a first quantization model.

[0020] In the disclosed embodiment, the image processing model is obtained by at least cropping training, and the cropping coefficient is decreased during the cropping process, and the cropping coefficient is used to determine the numerical range of the weight parameter in the cropped image processing model. Exemplarily, the numerical range is expressed as [min, max] as an example, wherein min represents the lower limit of the numerical range, and max represents the upper limit of the numerical range. Then a possible example of cropping the weight parameter using the numerical range can be: if the weight parameter is greater than the upper limit value max, the weight parameter can be reset to the upper limit value max; if the weight parameter is less than the lower limit value min, the weight parameter can be reset to the lower limit value min; if the weight parameter is within the above numerical range, the weight parameter can remain unchanged. Of course, the above example is only a possible example of cropping the weight parameter using the numerical range, and it does not limit other possible ways of cropping the weight parameter, and no examples are given here one by one. In addition, it should be noted that during the cropping training process, the image processing model performs data processing with the weight parameter after cropping, and adjusts the weight parameter before cropping (i.e., the original weight parameter) based on the running results.

[0021] In an implementation scenario, the image processing model can be used to perform image processing tasks, and the image processing tasks may include but are not limited to: target detection, target segmentation, target classification, etc. That is to say, the image processing model can be a target detection model (such as YOLO, etc.), a target segmentation model (such as U-Net, etc.), or a target classification model. The image processing tasks and network model structure specifically performed by the image processing model are not limited here. It should be noted that when the image processing model is a target detection model, the running result of the image processing model is the target detection result (such as the rectangular box where the target object is located); when the image processing model is a target segmentation model, the running result of the image processing model is the target segmentation result (such as the connected domain composed of pixels belonging to the target object); when the image processing model is a target classification, the running result of the image processing model is the target classification result (such as the specific category to which the target object belongs).

[0022] In an implementation scenario, as a possible example, during the cropping training process, the cropping coefficient can be initialized first, and then in each round of iteration of the image processing model: based on the current values ​​of the weight parameters and the cropping coefficient in the image processing model, a first range is obtained, and based on the first range, the weight parameters in the image processing model are cropped, and based on the running results of the image processing model after cropping, the weight parameters in the image processing model are adjusted, so that one round of training can be completed, and then the cropping coefficient can be decremented, and a new round of iteration can be returned to the image processing model. In the above method, in each round of iteration of cropping training, the current values ​​of the weight parameters and the cropping coefficients are first used to determine the first range, and then the weight parameters are cropped using the first norm. Then, based on the running results of the image processing model after cropping, the weight parameters before cropping in the image processing model are adjusted, and then the cropping coefficient is decreased to return to execute a new round of iterations. The method can dynamically adjust the numerical range used for cropping the weight parameters following the changes in the cropping coefficient and the weight parameters. The weight parameters can gradually stabilize within the range determined by the final cropping coefficient, so that the weight parameters are as close to a uniform distribution as possible, which helps to alleviate the accuracy loss caused by model quantization as much as possible.

[0023] In a specific implementation scenario, before the cropping coefficient is initialized, the target value of the cropping coefficient (i.e., the value to which the cropping coefficient decreases) can be determined to initialize the cropping coefficient to a value higher than the target value. Exemplarily, the embodiment of the present disclosure takes into account that when a certain set of data is quantized, if it satisfies the uniform distribution, its quantization error is often smaller, so the embodiment of the present disclosure expects that the distribution of the weight parameter is closer to the uniform distribution. If a certain set of data X satisfies the uniform symmetric distribution in the interval [-T, T], its maximum value and average value must satisfy: T=max(X)=2*mean(|X|). Among them, max() means taking the maximum value, mean() means taking the average value, and || means taking the absolute value. In other words, the target value can be set to 2. Of course, as another possible example, in order to avoid the possibility that the model accuracy may be further damaged due to the extreme setting of the target value to 2, the target value can be set to be slightly larger than 2, such as 3, etc., and the specific value of the target value is not limited here. Still taking the target value being set to 2 as an example, the clipping factor can be initialized to a value higher than the target value, such as 4, 5, etc., and the initial value of the clipping factor is not limited here.

[0024] In a specific implementation scenario, in each round of iteration, the average value can be first obtained based on the absolute value of the weight parameter in the image processing model, and then the upper limit value clip_data of the first range can be obtained based on the product of the average value and the current value: …… (1) In the above formula (1), represents the weight parameter, Indicates taking the absolute value, It means taking the average value, Indicates the current value of the crop factor. In addition, the lower limit value and the upper limit value of the first range are opposite to each other. That is, the weight parameter is adjusted using the first range. Weight parameters after clipping It can be expressed as: …… (2) In the above formula (2), Indicates the lower limit value, Indicates the upper limit value, represents the weight parameter, Represents the clipping function. The specific implementation can refer to the aforementioned clipping process, which will not be described in detail here. It should be noted that when clipping the weight parameters in the image processing model, the weight parameters of each network layer in the image processing model can be clipped layer by layer. Exemplarily, the weight parameters in the first network layer can be clipped first. At this time, the average value can be obtained based on the absolute value of the weight parameters in the first network layer, and then the upper limit value of the first range can be obtained based on the product of the average value and the current value of the clipping coefficient. The lower limit value of the first range is the inverse of the upper limit value, and then the weight parameters in the first network layer are clipped based on the first range, and then the weight parameters in the second network layer are clipped. The specific process can refer to the clipping process of the first network layer mentioned above, which will not be described in detail here. Taking the linear layer linear as an example, if it is not clipped during forward reasoning, its forward function can be expressed as: …… (3) In the above formula (3), Represents the linear layer linear and its forward function, Represents the original parameters, including the original weight parameters And the original bias parameters , represents the input tensor, Indicates the return content (i.e. output result) of the linear layer, that is, according to the original weight parameters And the original bias parameters Processing input tensors The results obtained: Different from the unpruned forward function mentioned above, if the linear layer is pruned during forward reasoning, its forward function can be expressed as:

[0025]

[0026] …… (4) In the above formula (4), represents the original weight parameter, Represents the weight parameter after cropping, and the specific meanings of other parameters can be found in formulas (1), (2), and (3), which will not be repeated here. On this basis, similar to the image to be processed, in the cropping training process, the image feature representation of the sample image can also be expressed in tensor form, and the image feature representation of the sample image is processed in sequence by each network layer after cropping, so that the running result of the image processing model after cropping can be obtained, and the weight parameter before cropping can be adjusted based on this. Exemplarily, taking the target detection task as an example, the sample image can be annotated with the real area of ​​the sample object, and the running result of the image processing model after cropping can include the predicted area of ​​the sample object in the sample image, so based on the difference between the real area and the predicted area, at least the weight parameter before cropping in the image processing model can be adjusted, and the bias parameter can also be adjusted simultaneously; or, taking the target classification task as an example, the sample image can be annotated with the real category of the sample object, and the running result of the image processing model after cropping can include the predicted category of the sample object in the sample image, so based on the difference between the real category and the predicted category, at least the weight parameter before cropping in the image processing model can be adjusted, and the bias parameter can also be adjusted simultaneously. The above examples only take the target detection task and the target classification task as examples to illustrate the specific process of adjusting the weight parameters before cropping. Other image processing tasks can be deduced by analogy, and no further examples are given here.

[0027] In a specific implementation scenario, the cropping coefficient may be decreased after each round of iteration. As a possible example, the amplitude value of the decreasing cropping coefficient may be determined based on the current iteration round, and the current iteration round is negatively correlated with the amplitude value. That is, the closer the current iteration round is, the larger the amplitude value of the decreasing cropping coefficient may be, and conversely, the later the current iteration round is, the smaller the amplitude value of the decreasing cropping coefficient may be. In other words, in the early stage of training, the cropping coefficient may be greatly decreased to accelerate the weight distribution to tend toward uniform distribution, that is, in the early stage of training, fast and efficient are pursued, and in the later stage of training, the cropping coefficient may be slightly decreased to force the weight distribution to tend toward uniform distribution more accurately, that is, fine and accurate are pursued in the later stage of training. Exemplarily, the amplitude value may be within a numerical range of 0.2 to 0.02. For example, during the first cropping training, the amplitude value of the decreasing cropping coefficient may be 0.2. During the second cropping training, the amplitude value of the decreasing cropping coefficient may be 0.19. During the third cropping training, the amplitude value of the decreasing cropping coefficient may be 0.18. And so on. Examples are not given one by one here. Alternatively, as another possible example, the cropping coefficient may be decreased by a preset value for each cropping training, that is, the preset value remains constant during the training process of the image processing model. Exemplarily, the preset value may be set to 0.05, 0.04, etc., and the specific value of the preset value is not limited here.

[0028] In one implementation scenario, the image processing model can be pre-trained before the cropping training. It should be noted that during the pre-training process, the weight parameters in the image processing model are not cropped. Exemplarily, taking target detection as an example, the image processing model can be used to process the sample image to obtain the predicted area of ​​the sample object in the sample image, and the network parameters (such as weight parameters, bias parameters, etc.) of the image processing model can be adjusted based on the difference between the predicted area and the real area of ​​the sample object in the sample image; or, taking target classification as an example, the image processing model can be used to process the sample image to obtain the predicted category of the sample object in the sample image, and the network parameters (such as weight parameters, bias parameters, etc.) of the image processing model can be adjusted based on the difference between the predicted category and the real category of the sample object in the sample image. Of course, the above examples are only a few possible examples of pre-training the image processing model before cropping training, and other possible situations will not be given one by one here.

[0029] In one implementation scenario, after the cropping training, the image processing model can also be quantized. It should be noted that the weight parameters in the image processing model at the beginning of the quantization training are the weight parameters after the cropping training. Specifically, the second range can be obtained based on the final values ​​of the weight parameters and the cropping coefficients in the image processing model after the cropping training, and the weight parameters in the image processing model can be cropped based on the second range. On this basis, the image processing model after cropping in the second range can be quantized to obtain an image processing model for deployment on the current device. It should be noted that, as mentioned above, when cropping the weight parameters in the image processing model, the weight parameters in each network layer in the image processing model can be cropped layer by layer. For details, please refer to the above-mentioned related description, which will not be repeated here. In the above manner, the weight parameters in the image processing model are cropped after the cropping training of the image processing model is completed. At this time, the weight parameters after cropping are as close to uniform distribution as possible, so that the subsequent precision loss caused by parameter quantization can be reduced as much as possible, and quantization training based on this will help to further reduce the precision loss caused by parameter quantization.

[0030] In a specific implementation scenario, the specific process of determining the second range can refer to the related description of determining the first range mentioned above, which will not be repeated here.

[0031] In a specific implementation scenario, quantization training can be carried out through technical details of quantization training such as TQT (Trained Quantization Thresholds) training based on W4A8 (i.e., weight quantization is 4 bits and activation quantization is 8 bits). Exemplarily, taking TQT training as an example, for the sake of ease of description, the quantization factor of the weight parameter can be recorded as s_w, the quantization factor of the input (i.e., tensor) can be recorded as s_i, and the quantization factor of the output (i.e., activation) can be recorded as s_o. It should be noted that the quantization factor s_w of the weight parameter can be directly initialized according to the weight parameter weight of each network layer, and the quantization factor s_i of the input (i.e., tensor) and the quantization factor s_o of the output (i.e., activation) can be obtained by the image processing model through a small-scale calibration set statistics. The specific process can be referred to TQT training and other technical details are not limited here. The quantization process can be described as:

[0032] …… (5) In the above formula (5), n is the quantization bit. Taking W4A8 as an example, for the weight parameter, its quantization bit n can be set to 4. In addition, q max =2 n-1 -1,q min =-2 n-1 In order to achieve automatic derivation of pseudo-quantization process during quantization training and for simplicity, you can set ,in, Indicates that forward derivatives are not sought. In addition, the specific meanings of other related parameters can be found in the technical details of pertensor quantization, which will not be repeated here. For example, the quantization factor s_w of the weight parameter is expressed as an exponential power of 2, which can facilitate the multiplication or division calculation of each tensor (that is, shift operations can be performed), which helps to greatly improve the inference speed.

[0033] In a specific implementation scenario, different from the initialization method of the scaling factor (i.e., the aforementioned quantization factor) in conventional TQT training, as another possible example, as mentioned above, before the quantization training, each network layer in the image processing model is initialized with a first scaling factor for quantizing the input parameter / output parameter (e.g., it can be expressed as the aforementioned input parameter quantization factor s_i, the output parameter quantization factor s_o), then for any network layer, its first scaling factor can be initialized by the following steps: First, the statistical variable of the target parameter can be initialized for the network layer. It should be noted that when the first scaling factor of the network layer input parameter is initialized, the target parameter is the input parameter, and when the first scaling factor of the network layer output parameter is initialized, the target parameter is the output parameter. In addition, when initializing the first scaling factor of the input parameter or output parameter of each network layer, the statistical variable can be initialized to 0. On this basis, the maximum absolute value of the target parameter of the network layer when the image processing model processes the sample image in the calibration set can be obtained. Then, the current value of the statistical variable and the above-mentioned maximum absolute value can be weighted to obtain the new current value of the statistical variable. It should be noted that when weighting, the weight factors of the two can be set as needed. For example, when the statistical variable is given more weight, the weight factor of the statistical variable can be set larger, and when the maximum absolute value is given more weight, the weight factor of the maximum absolute value can be set larger. For example, taking the first scaling factor of the initialization input parameter as an example, the updating process of the statistical variable can be expressed as: …… (6) In the above formula (6), the right side of the equal sign Indicates the current value of the statistical variable, on the left side of the equal sign Indicates the new current value after the statistical variable is updated. represents the maximum absolute value of the input parameter, 0.9 represents the weight factor of the statistical variable, and 0.1 represents the weight factor of the maximum absolute value. Of course, formula (6) is only a possible example of the updating process of the statistical variable when the first scaling factor of the input parameter is initialized as an example, and other possible updating methods are not given one by one here. Similarly, taking the first scaling factor of the output parameter as an example, the updating process of the statistical variable can be expressed as: …… (7) In the above formula (7), the right side of the equal sign Indicates the current value of the statistical variable, on the left side of the equal sign Indicates the new current value after the statistical variable is updated. Represents the maximum absolute value of the output parameter, 0.9 represents the weight factor of the statistical variable, and 0.1 represents the weight factor of the maximum absolute value. Of course, formula (7) is only a possible example of the update process of the statistical variable when the first scaling factor of the output parameter is initialized. Other possible update methods are not given examples here. Then, for the next sample image in the calibration set, the maximum absolute value of the target parameter of the network layer when the image processing model processes the sample image in the calibration set can be returned to execute the aforementioned acquisition. The maximum absolute value of the target parameter of the network layer when the image processing model processes the sample image in the calibration set can be repeated until all the sample images in the calibration set are processed. Finally, based on the latest current value of the statistical variable, the first scaling factor expressed as an exponential power of 2 can be obtained. As a possible example, after the latest current value of the input parameter or output parameter of any network layer with respect to the statistical variable, the following formula can be substituted to obtain the first scaling factor expressed as an exponential power of 2: …… (8) In the above formula (8), Indicates the latest current value of the statistical variable, n is the quantization bit, and round indicates rounding. In addition, when the target parameter is an input parameter, s represents the first scaling factor of the input parameter (such as the aforementioned s_i), and when the target parameter is an output parameter, s represents the first scaling factor of the output parameter (such as the aforementioned s_o). The above method initializes the statistical variable of the target parameter for the network layer, obtains the maximum absolute value of the target parameter of the network layer when the image processing model processes the sample image in the calibration set, and then weights it based on the current value and the maximum absolute value of the statistical variable to obtain the new current value of the statistical variable, and for the next sample image in the calibration set, returns to execute the maximum absolute value of the target parameter of the network layer when the image processing model processes the sample image in the calibration set, until all sample images in the calibration set are processed, so as to obtain the first scaling factor expressed as an exponential power of 2 based on the latest current value of the statistical variable, and then can weight and smooth the statistical variable. Compared with directly taking the maximum absolute value to obtain the scaling factor, it can avoid the scaling factor affected by the outliers that may appear in the target parameter as much as possible.

[0034] In a specific implementation scenario, different from the initialization method of the scaling factor (i.e., the aforementioned quantization factor) in conventional TQT training, as another possible example, as mentioned above, each network layer in the image processing model is initialized with a second scaling factor for quantizing the weight parameter (such as the aforementioned quantization factor s_w of the weight parameter) before quantization training, and for any network layer, the second scaling factor of its weight parameter can be initialized by the following steps: First, the value obtained by traversing within the preset numerical range can be used as the exponential power of 2 to obtain the candidate scaling factor of the weight parameter. It should be noted that the upper limit and lower limit of the preset numerical range can be set according to the maximum possible value and the minimum possible value of the weight parameter. Exemplarily, in general, the weight parameter is in the numerical range of -100~100, so the preset numerical range can be set to -8~8, then integers (such as -8, -7, -6, ..., 6, 7, 8, etc.) can be traversed within this preset numerical range, and the traversed values ​​are used as the exponential power of 2 to obtain the candidate scaling factor of the weight parameter. For ease of description, the traversed value can be recorded as factor, and the candidate scaling factor can be recorded as Next, the quantization deviation of the candidate scaling factor can be obtained based on the weight parameter and the weight parameter obtained by quantizing the weight parameter using the candidate scaling factor and then dequantizing it. , where W represents the weight parameter, Represents the use of candidate scaling factors Quantize the weight parameter W, Represents dequantization, and the quantization deviation can be obtained by L2 norm measurement. Finally, based on the quantization deviation of each candidate scaling factor, a candidate scaling factor can be selected as the second scaling factor. Exemplarily, the candidate scaling factor corresponding to the minimum quantization deviation can be selected as the second scaling factor. In the above method, the candidate scaling factor of the weight parameter is obtained by traversing the obtained value within the preset numerical range as the exponential power of 2, and the quantization deviation of the candidate scaling factor is obtained based on the weight parameter and the weight parameter quantized by the candidate scaling factor and then dequantized, and then the candidate scaling factor is selected as the second scaling factor based on the quantization deviation of each candidate scaling factor. Compared with directly initializing the scaling factor according to the maximum value, the quantization loss caused by outliers that may appear in the weight parameter can be minimized.

[0035] In a specific implementation scenario, after quantization, the image feature representation expressed in the form of a tensor can be processed by the image processing model after quantization to obtain the running result, and the weight parameters in the image processing model are fine-tuned accordingly, so that the weight parameters after clipping are adapted to the quantization process, and then after the quantization training converges, the weight parameters can be clipped again with reference to the above-mentioned related description, and further quantized with reference to the above-mentioned related description after clipping to obtain the first quantization model. It should be noted that the quantization tool may include at least one of the following: W4A8 quantization, symmetric quantization, linear quantization, pertensor quantization, and a scaling factor of 2. In actual application scenarios, according to the calculation scheme of mapping from 322-bit data to low bits during quantization, it can be divided into linear quantization and nonlinear quantization, symmetric quantization and asymmetric quantization; according to whether the object of the quantization scheme is tensor or channel (i.e., channel), it can be divided into pertensor quantization and perchannel quantization; according to whether the number of bits after model quantization is exactly the same, hybrid quantization can be introduced. In addition, when the quantization tool includes at least one of W4A8 quantization, symmetric quantization, linear quantization, pertensor quantization, and a scaling factor of 2, the specific process of model quantization can refer to the aforementioned formula (5) and related descriptions, which will not be repeated here.

[0036] In an implementation scenario, the image to be processed may be different images according to the application scenario of image processing. For example, in a traffic control scenario, the image to be processed may be a road traffic image (e.g., a captured image at a road junction, etc.); or, in an industrial scenario, the image to be processed may be a product image (e.g., a captured image of a product to be inspected, etc.); or, in an educational scenario, the image to be processed may be a test paper image, etc. Of course, the above examples are only several possible examples of the images to be processed, and the specific content of the images to be processed is not limited here.

[0037] In one implementation scenario, feature extraction of the image to be processed can be implemented through a network layer such as convolution, so that the image to be processed can be converted into an image feature representation in the form of a tensor through feature extraction. The specific process of feature extraction can refer to the technical details of network layers such as convolution, which will not be repeated here.

[0038] Step S12: Process the image feature representation based on the first quantization model to obtain a temporary processing result.

[0039] It should be noted that the image feature representation is processed layer by layer by the first quantization model, and a temporary processing result can be obtained at the output layer of the first quantization model. Since the temporary processing result is superimposed with the quantization factor, it cannot be directly output as the final processing result, but must be dequantized before output.

[0040] Step S13: Dequantizing the temporary processing result based on the quantization tool to obtain the final processing result of the image to be processed.

[0041] Specifically, please refer to formula (5), the dequantization process can be combined with the quantization factor As a possible example, dequantization can be expressed as Of course, the above formula shows the dequantization of the model weight parameters, and the dequantization of the model output results can be deduced in the same way. For details, please refer to the relevant description of dequantization in quantization training such as TQT, which will not be repeated here. For example, in the target detection task, the final processing result may include but is not limited to the target area of ​​the target object in the image to be processed; or, in the target segmentation task, the final processing result may include but is not limited to the connected domain composed of pixel points belonging to the target object in the image to be processed; or, in the target classification task, the final processing result may include but is not limited to the target category of the target object in the image to be processed. Of course, the above examples are only a few possible examples in actual application, and other possible situations will not be given one by one here.

[0042] The above scheme extracts the image feature representation of the image to be processed, and quantizes the image processing model deployed on the current device based on the quantization tool on the current device to obtain a first quantization model, and the image processing model is at least obtained through cropping training, and the cropping coefficient is reduced during the cropping training process. The cropping coefficient is used to determine the numerical range of the weight parameter in the cropped image processing model, and then the image feature representation is processed based on the first quantization model to obtain a temporary processing result, and then the temporary processing result is dequantized based on the quantization tool to obtain the final processing result of the image to be processed. Then, on the one hand, by quantizing the neural network model deployed on the current device and then using it, the reasoning speed of the neural network model can be ensured as much as possible. On the other hand, the neural network model is at least cropped and trained, and because the cropping coefficient is dynamically reduced during the cropping training process, the weight parameter is gradually stabilized within the range determined by the final cropping coefficient after being cropped within the numerical range determined by the cropping system during the training process, so that the weight parameter is as close to a uniform distribution as possible, which helps to alleviate the accuracy loss caused by model quantization as much as possible, and can improve the accuracy of the neural network model in its processing tasks as much as possible. Therefore, the reasoning speed of the neural network model can be ensured as much as possible, and the accuracy of the neural network model in its processing tasks can be improved.

[0043] See also Figure 2 , Figure 2It is a flowchart of an embodiment of the speech processing method of the present application. It should be noted that the embodiment of the present disclosure mainly describes the differences between the application scenario of speech processing and the application scenario of the aforementioned image processing. The same or similarities can be referred to the disclosed embodiment of the aforementioned image processing method (for example, the image processing model in the disclosed embodiment of the aforementioned image processing method can be replaced with a language processing model), which will not be repeated here. Specifically, in the application scenario of speech processing, the embodiment of the present disclosure may include the following steps: Step S21: extracting the speech feature representation of the speech to be processed, and quantizing the speech processing model deployed on the current device based on the quantization tool on the current device to obtain a second quantization model.

[0044] In the disclosed embodiment, the speech processing model is obtained by at least cropping training, and the cropping coefficient is gradually reduced during the cropping process, and the cropping coefficient is used to determine the numerical range of the weight parameter in the cropped speech processing model. It should be noted that, for the cropping training of the speech processing model, reference can be made to the relevant description of the cropping training of the image processing model in the aforementioned disclosed embodiment, which will not be repeated here. In addition, the specific meaning of the cropping coefficient and the method for determining the numerical range can be referred to the aforementioned disclosed embodiment, which will not be repeated here. In addition, during the cropping training and quantization training process, the speech processing model can use sample speech in the disclosed embodiment, which is different from the sample image used in the aforementioned disclosed embodiment.

[0045] In one implementation scenario, similar to the image processing model in the aforementioned disclosed embodiment, the speech processing model may be pre-trained before the speech processing model performs the cropping training; or, after the speech processing model performs the cropping training, the speech processing model may be quantized. It should be noted that the specific process of the pre-training and quantization training of the speech processing model can refer to the pre-training and quantization training of the image processing model in the aforementioned disclosed embodiment, which will not be described in detail here.

[0046] In an implementation scenario, the speech to be processed may also be different depending on the application scenario. For example, in a speech recognition scenario, the speech to be processed may be a speech to be recognized, such as a speech to be recognized may be collected by a voice assistant (such as a car computer, etc.); or, in a speech translation scenario, the speech to be processed may be a speech to be translated, such as a speech to be translated may be collected by a translation machine, etc. Of course, the above examples are only a few possible examples of speech processing tasks, and other possible situations are not given one by one here. For example, speech processing tasks may also include but are not limited to speech simultaneous interpretation tasks, etc. Accordingly, the speech processing model may specifically include but is not limited to: speech recognition model, speech translation model, speech simultaneous interpretation model, etc., and the network structure of the speech processing model is not limited here.

[0047] In one implementation scenario, feature extraction is performed on the processed speech to obtain a speech feature representation expressed in tensor form, which can be specifically referred to in the description of the image feature representation in the aforementioned disclosed embodiment, which will not be repeated here. In addition, the specific process of quantizing the speech processing model based on the quantization tool can be referred to in the description of quantizing the image processing model based on the quantization tool in the aforementioned disclosed embodiment, which will not be repeated here.

[0048] Step S22: Process the speech feature representation based on the second quantization model to obtain a temporary processing result.

[0049] For details, please refer to the relevant description of "processing the image feature representation based on the first quantization model to obtain a temporary processing result" in the aforementioned public embodiment, which will not be repeated here.

[0050] Step S23: Dequantizing the temporary processing result based on the quantization tool to obtain the final processing result of the speech to be processed.

[0051] For details, please refer to the description of "de-quantizing the temporary processing result based on the quantization tool to obtain the final processing result of the image to be processed" in the aforementioned public embodiment, which will not be repeated here. In addition, after the cropping training, the specific process of quantizing the speech processing model can refer to the quantization training of the image processing model in the aforementioned image processing method embodiment, which will not be repeated here.

[0052] The above scheme extracts the speech feature representation of the speech to be processed, and quantizes the speech processing model deployed on the current device based on the quantization tool on the current device to obtain a second quantized model, and the speech processing model is at least obtained through trimming training, and the trimming coefficient is reduced during the trimming process. The trimming coefficient is used to determine the numerical range of the weight parameter in the trimmed speech processing model, and then the speech feature representation is processed based on the second quantized model to obtain a temporary processing result, and then the temporary processing result is dequantized based on the quantization tool to obtain the final processing result of the speech to be processed. On the one hand, by quantizing the neural network model deployed on the current device and then using it, the reasoning speed of the neural network model can be ensured as much as possible. On the other hand, the neural network model is at least trimmed and trained, and because the trimming coefficient is dynamically reduced during the trimming training process, the weight parameter is trimmed in the numerical range determined by the trimming system during the training process, and gradually stabilizes within the range determined by the final trimming coefficient, so that the weight parameter is as close to a uniform distribution as possible, which helps to alleviate the accuracy loss caused by model quantization as much as possible, and can improve the accuracy of the neural network model in its processing tasks as much as possible. Therefore, the reasoning speed of the neural network model can be ensured as much as possible, and the accuracy of the neural network model in its processing tasks can be improved.

[0053] See also Figure 3 , Figure 3 It is a flowchart of an embodiment of the natural language processing method of the present application. It should be noted that the embodiment of the present disclosure mainly describes the differences between the application scenario of natural language processing and the application scenario of the aforementioned image processing. The same or similarities can be referred to the disclosed embodiment of the aforementioned image processing method (for example, the image processing model in the disclosed embodiment of the aforementioned image processing method can be replaced with a natural language processing model), which will not be repeated here. Specifically, in the application scenario of natural language processing, the embodiment of the present disclosure may include the following steps: Step S31: extract the sentence feature representation of the sentence to be processed, and quantize the natural language processing model deployed on the current device based on the quantization tool on the current device to obtain a third quantization model.

[0054] In the disclosed embodiment, the natural language processing model is obtained by at least cropping training, and the cropping coefficient is decreased during the cropping process, and the cropping coefficient is used to determine the numerical range of the weight parameter in the cropped natural language processing model. It should be noted that the cropping training of the natural language processing model can refer to the relevant description of the cropping training of the image processing model in the aforementioned disclosed embodiment, which will not be repeated here. In addition, the specific meaning of the cropping coefficient and the method for determining the numerical range can refer to the aforementioned disclosed embodiment, which will not be repeated here. In addition, during the cropping training and quantization training process, the natural language processing model can use sample text in the disclosed embodiment, which is different from the sample image used in the aforementioned disclosed embodiment.

[0055] In one implementation scenario, similar to the image processing model in the aforementioned disclosed embodiment, the natural language processing model can be pre-trained before the natural language processing model performs the cropping training; or, after the natural language processing model performs the cropping training, the natural language processing model can be quantized. It should be noted that the specific process of pre-training and quantization training of the natural language processing model can refer to the pre-training and quantization training of the image processing model in the aforementioned disclosed embodiment, which will not be repeated here.

[0056] In an implementation scenario, the sentences to be processed may also be different depending on the application scenario. For example, in an intelligent dialogue scenario, the sentence to be processed may be an input sentence to which the current device is expected to respond (e.g., please help me plan a travel plan from City A to City B); or, in an assisted writing scenario, the sentence to be processed may be the subject scope and reference materials for the expected assisted writing (e.g., please refer to the following materials to help me write a speech). Of course, the above examples are only a few possible examples of natural language processing tasks, and other possible situations will not be given one by one here. For example, natural language processing tasks may also include but are not limited to smart office, etc. Accordingly, the natural language processing model may specifically include but is not limited to: intelligent dialogue model (e.g., large language model, etc.), auxiliary collaboration model, etc., and the network structure of the speech processing model is not limited here.

[0057] In one implementation scenario, feature extraction is performed on the sentence to be processed to obtain a sentence feature representation expressed in tensor form, which can be specifically referred to in the description of the image feature representation in the aforementioned disclosed embodiment, which will not be repeated here. In addition, the specific process of quantizing the natural language processing model based on the quantization tool can be referred to in the description of quantizing the image processing model based on the quantization tool in the aforementioned disclosed embodiment, which will not be repeated here.

[0058] Step S32: Process the sentence feature representation based on the third quantitative model to obtain a temporary processing result.

[0059] For details, please refer to the relevant description of "processing the image feature representation based on the first quantization model to obtain a temporary processing result" in the aforementioned public embodiment, which will not be repeated here.

[0060] Step S33: Dequantize the temporary processing result based on the quantization tool to obtain the final processing result of the statement to be processed.

[0061] For details, please refer to the description of "de-quantizing the temporary processing result based on the quantization tool to obtain the final processing result of the image to be processed" in the aforementioned public embodiment, which will not be repeated here. In addition, after the cropping training, the specific process of quantization training of the natural language processing model can be referred to the quantization training of the image processing model in the aforementioned image processing method embodiment, which will not be repeated here.

[0062] The above scheme extracts the sentence feature representation of the sentence to be processed, and quantizes the natural language processing model deployed on the current device based on the quantization tool on the current device to obtain a third quantized model, and the natural language processing model is at least obtained through trimming training, and the trimming coefficient is reduced during the trimming process. The trimming coefficient is used to determine the numerical range of the weight parameter in the trimmed natural language processing model, and then the sentence feature representation is processed based on the third quantized model to obtain a temporary processing result, and then the temporary processing result is dequantized based on the quantization tool to obtain the final processing result of the sentence to be processed. On the one hand, by quantizing the neural network model deployed on the current device and then using it, the reasoning speed of the neural network model can be ensured as much as possible. On the other hand, the neural network model is at least trimmed and trained, and because the trimming coefficient is dynamically reduced during the trimming training process, the weight parameter is trimmed in the numerical range determined by the trimming system during the training process, and gradually stabilizes within the range determined by the final trimming coefficient, so that the weight parameter is as close to a uniform distribution as possible, which helps to alleviate the accuracy loss caused by model quantization as much as possible, and can improve the accuracy of the neural network model in its processing tasks as much as possible. Therefore, the reasoning speed of the neural network model can be ensured as much as possible, and the accuracy of the neural network model in its processing tasks can be improved.

[0063] See also Figure 4 , Figure 4 It is a schematic diagram of the framework of an embodiment of the image processing device of the present application. The image processing device 40 includes: an extraction quantization module 41, a model processing module 42 and a dequantization module 43, wherein the extraction quantization module 41 is used to extract the image feature representation of the image to be processed, and quantize the image processing model deployed on the current device based on the quantization tool on the current device to obtain a first quantization model; wherein the image processing model is obtained at least through cropping training, and the cropping coefficient is decreased during the cropping process, and the cropping coefficient is used to determine the numerical range of the weight parameter in the cropped image processing model; the model processing module 42 is used to process the image feature representation based on the first quantization model to obtain a temporary processing result; the dequantization module 43 is used to dequantize the temporary processing result based on the quantization tool to obtain the final processing result of the image to be processed.

[0064] In the above scheme, the image processing device 40 extracts the image feature representation of the image to be processed, and quantizes the image processing model deployed on the current device based on the quantization tool on the current device to obtain a first quantization model, and the image processing model is at least obtained through cropping training. During the cropping training process, the cropping coefficient is decreased, and the cropping coefficient is used to determine the numerical range of the weight parameter in the cropped image processing model. The image feature representation is then processed based on the first quantization model to obtain a temporary processing result, and then the temporary processing result is dequantized based on the quantization tool to obtain the final processing result of the image to be processed. On the one hand, by quantizing the neural network model deployed on the current device and then using it, the reasoning speed of the neural network model can be ensured as much as possible. On the other hand, the neural network model is at least cropped and trained, and because the cropping coefficient is dynamically decreased during the cropping training process, the weight parameter is gradually stabilized within the range determined by the final cropping coefficient after being cropped within the numerical range determined by the cropping system during the training process, so that the weight parameter is as close to a uniform distribution as possible, which helps to alleviate the accuracy loss caused by model quantization as much as possible, and can improve the accuracy of the neural network model in its processing tasks as much as possible. Therefore, the reasoning speed of the neural network model can be ensured as much as possible, and the accuracy of the neural network model in its processing tasks can be improved.

[0065] In some disclosed embodiments, the image processing device 40 includes a coefficient initialization module for initializing the cropping coefficient; the image processing device 40 includes an iterative training module for obtaining a first range based on the current values ​​of the weight parameters and the cropping coefficient in the image processing model, and cropping the weight parameters in the image processing model based on the first range, and adjusting the weight parameters before cropping based on the running results of the image processing model after cropping in each round of iteration of the image processing model; the image processing device 40 includes a decrement loop module for decrementing the cropping coefficient and returning the image processing model to perform a new round of iteration.

[0066] In some disclosed embodiments, the decrement loop module includes an amplitude determination submodule, which is used to determine the amplitude value of the decrementing clipping coefficient based on the current iteration round; wherein the current iteration round is negatively correlated with the amplitude value; the decrement loop module includes a first decrementing submodule, which is used to decrement the clipping coefficient based on the amplitude value of the decrementing clipping coefficient.

[0067] In some disclosed embodiments, the decrement loop module includes a second decrement submodule for decrementing the cropping factor by a preset value; wherein the preset value remains constant during the training process of the image processing model.

[0068] In some disclosed embodiments, the iterative training module includes an average value calculation submodule, which is used to obtain the average value based on the absolute value of the weight parameter in the image processing model; the iterative training module includes an endpoint value determination submodule, which is used to obtain the upper limit value of the first range based on the product of the average value and the current value; wherein the lower limit value and the upper limit value of the first range are reciprocal numbers.

[0069] In some disclosed embodiments, the image processing device 40 includes a range determination module for obtaining a second range based on the final values ​​of the weight parameters and cropping coefficients in the image processing model after the cropping training is completed; the image processing device 40 includes a parameter cropping module for cropping the weight parameters in the image processing model based on the second range; the image processing device 40 includes a quantization training module for performing quantization training based on the image processing model after cropping according to the second range to obtain an image processing model for deployment on the current device.

[0070] In some disclosed embodiments, before quantization training, each network layer in the image processing model is respectively initialized with a first scaling factor for quantizing input parameters / output parameters, and the image processing device 40 includes an initialization module for initializing statistical variables of target parameters for the network layer; the image processing device 40 includes an absolute value statistics module for obtaining the maximum absolute value of the target parameter of the network layer when the image processing model processes sample images in the calibration set; wherein, when initializing the first scaling factor of the input parameter, the target parameter is the input parameter, and when initializing the first scaling factor of the output parameter, the target parameter is the output parameter; the image processing device 40 includes a numerical weighting module for weighting based on the current numerical value and the maximum absolute value of the statistical variable to obtain a new current numerical value of the statistical variable; the image processing device 40 includes a loop iteration module for returning to execute, for the next sample image in the calibration set, obtaining the maximum absolute value of the target parameter of the network layer when the image processing model processes the sample image in the calibration set until all sample images in the calibration set are processed; the image processing device 40 includes a factor determination module for obtaining a first scaling factor expressed as an exponential power of 2 based on the latest current numerical value of the statistical variable.

[0071] In some disclosed embodiments, before quantization training, each network layer in the image processing model is respectively initialized with a second scaling factor for quantizing the weight parameter, and the image processing device 40 includes a candidate factor module for obtaining a candidate scaling factor for the weight parameter by taking a numerical value obtained by traversing within a preset numerical range as an exponential power of 2; the image processing device 40 includes a deviation measurement module for obtaining a quantization deviation of the candidate scaling factor based on the weight parameter and the weight parameter quantized by the candidate scaling factor and then dequantized; the image processing device 40 includes a factor selection module for selecting a candidate scaling factor as the second scaling factor based on the quantization deviation of each candidate scaling factor.

[0072] In some disclosed embodiments, weight parameters in each network layer in an image processing model are pruned layer by layer.

[0073] In some disclosed embodiments, the quantization tools include at least one of: W4A8 quantization, symmetric quantization, linear quantization, pertensor quantization, and scaling factors of powers of 2.

[0074] See also Figure 5 , Figure 5 It is a schematic diagram of the framework of an embodiment of the speech processing device of the present application. The speech processing device 50 includes: an extraction quantization module 51, a model processing module 52 and a dequantization module 53, wherein the extraction quantization module 51 is used to extract the speech feature representation of the speech to be processed, and quantize the speech processing model deployed on the current device based on the quantization tool on the current device to obtain a second quantization model; wherein the speech processing model is obtained by at least trimming training, and the trimming coefficient is reduced during the trimming process, and the trimming coefficient is used to determine the numerical range of the weight parameter in the trimmed speech processing model; the model processing module 52 is used to process the speech feature representation based on the second quantization model to obtain a temporary processing result; the dequantization module 53 is used to dequantize the temporary processing result based on the quantization tool to obtain the final processing result of the speech to be processed.

[0075] In the above scheme, the speech processing device 50 extracts the speech feature representation of the speech to be processed, and quantizes the speech processing model deployed on the current device based on the quantization tool on the current device to obtain a second quantization model, and the speech processing model is at least obtained through trimming training, and the trimming coefficient is reduced during the trimming process. The trimming coefficient is used to determine the numerical range of the weight parameter in the trimmed speech processing model, and then the speech feature representation is processed based on the second quantization model to obtain a temporary processing result, and then the temporary processing result is dequantized based on the quantization tool to obtain the final processing result of the speech to be processed. Then, on the one hand, by quantizing the neural network model deployed on the current device and then using it, the reasoning speed of the neural network model can be ensured as much as possible. On the other hand, the neural network model is at least trimmed and trained, and because the trimming coefficient is dynamically reduced during the trimming training process, the weight parameter is trimmed in the numerical range determined by the trimming system during the training process, and gradually stabilizes within the range determined by the final trimming coefficient, so that the weight parameter is as close to a uniform distribution as possible, which helps to alleviate the accuracy loss caused by model quantization as much as possible, and can improve the accuracy of the neural network model in its processing tasks as much as possible. Therefore, the reasoning speed of the neural network model can be ensured as much as possible, and the accuracy of the neural network model in its processing tasks can be improved.

[0076] See also Figure 6 , Figure 6It is a schematic diagram of the framework of an embodiment of the natural speech processing device of the present application. The natural speech processing device 60 includes: an extraction quantization module 61, a model processing module 62 and a dequantization module 63, wherein the extraction quantization module 61 is used to extract the sentence feature representation of the sentence to be processed, and quantize the natural language processing model deployed on the current device based on the quantization tool on the current device to obtain a third quantization model; wherein the natural language processing model is obtained by at least trimming training, and the trimming coefficient is reduced during the trimming process, and the trimming coefficient is used to determine the numerical range of the weight parameter in the trimmed natural language processing model; the model processing module 62 is used to process the sentence feature representation based on the third quantization model to obtain a temporary processing result; the dequantization module 63 is used to dequantize the temporary processing result based on the quantization tool to obtain the final processing result of the sentence to be processed.

[0077] In the above scheme, the natural speech processing device 60 extracts the sentence feature representation of the sentence to be processed, and quantizes the natural language processing model deployed on the current device based on the quantization tool on the current device to obtain a third quantization model, and the natural language processing model is at least obtained through trimming training, and the trimming coefficient is reduced during the trimming process. The trimming coefficient is used to determine the numerical range of the weight parameter in the trimmed natural language processing model, and then the sentence feature representation is processed based on the third quantization model to obtain a temporary processing result, and then the temporary processing result is dequantized based on the quantization tool to obtain the final processing result of the sentence to be processed. Then, on the one hand, by quantizing the neural network model deployed on the current device and then using it, the reasoning speed of the neural network model can be ensured as much as possible. On the other hand, the neural network model is at least trimmed and trained, and because the trimming coefficient is dynamically reduced during the trimming training process, the weight parameter is trimmed in the numerical range determined by the trimming system during the training process, and gradually stabilizes within the range determined by the final trimming coefficient, so that the weight parameter is as close to a uniform distribution as possible, which helps to alleviate the accuracy loss caused by model quantization as much as possible, and can improve the accuracy of the neural network model in its processing tasks as much as possible. Therefore, the reasoning speed of the neural network model can be ensured as much as possible, and the accuracy of the neural network model in its processing tasks can be improved.

[0078] See also Figure 7 , Figure 7: is a schematic diagram of the framework of an embodiment of an electronic device 70 of the present application. The electronic device 70 includes a memory 71 and a processor 72 coupled to each other, the memory 71 at least stores program instructions, and the processor 72 is used to execute the program instructions to implement the steps in any of the above-mentioned image processing method embodiments, or the steps in any of the above-mentioned speech processing method embodiments, or the steps in any of the above-mentioned natural language processing method embodiments. For details, please refer to the aforementioned disclosed embodiments, which will not be repeated here. The electronic device 70 may include but is not limited to: a smart phone, a tablet computer, a server, etc., and the specific type of the electronic device 70 is not limited here.

[0079] Specifically, the processor 72 is used to control itself and the memory 71 to implement the steps in any of the above-mentioned image processing method embodiments. The processor 72 can also be called a CPU (Central Processing Unit). The processor 72 may be an integrated circuit chip with signal processing capabilities. The processor 72 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 72 can be implemented by an integrated circuit chip.

[0080] In the above scheme, the electronic device 70 implements the steps in any of the above-mentioned image processing method embodiments, or the steps in any of the above-mentioned speech processing method embodiments, or the steps in any of the above-mentioned natural language processing method embodiments. On the one hand, by quantizing the neural network model deployed on the current device before using it, the reasoning speed of the neural network model can be ensured as much as possible. On the other hand, the neural network model is at least trained by clipping, and because the clipping coefficient is dynamically reduced during the clipping training process, the weight parameter is gradually stabilized within the range determined by the final clipping coefficient after being clipped by the numerical range determined by the clipping system during the training process, so that the weight parameter is as close to uniform distribution as possible, which helps to alleviate the accuracy loss caused by model quantization as much as possible, and can improve the accuracy of the neural network model in its processing tasks as much as possible. Therefore, the reasoning speed of the neural network model can be ensured as much as possible, and the accuracy of the neural network model in its processing tasks can be improved.

[0081] See also Figure 8 , Figure 81 is a schematic diagram of a framework of an embodiment of a computer-readable storage medium 80 of the present application. The computer-readable storage medium 80 stores program instructions 81 that can be executed by a processor, and the program instructions 81 are used to implement the steps in any of the above-mentioned image processing method embodiments, or the steps in any of the above-mentioned speech processing method embodiments, or the steps in any of the above-mentioned natural language processing method embodiments.

[0082] In the above scheme, the computer-readable storage medium 80 implements the steps in any of the above-mentioned image processing method embodiments, or the steps in any of the above-mentioned speech processing method embodiments, or the steps in any of the above-mentioned natural language processing method embodiments. Therefore, on the one hand, by quantizing the neural network model deployed on the current device and then using it, the reasoning speed of the neural network model can be ensured as much as possible. On the other hand, the neural network model is at least trained by clipping, and because the clipping coefficient is dynamically reduced during the clipping training process, the weight parameter is clipped by the numerical range determined by the clipping system during the training process, and gradually stabilizes within the range determined by the final clipping coefficient, so that the weight parameter is as close to uniform distribution as possible, which helps to alleviate the accuracy loss caused by model quantization as much as possible, and can improve the accuracy of the neural network model in its processing tasks as much as possible. Therefore, the reasoning speed of the neural network model can be ensured as much as possible, and the accuracy of the neural network model in its processing tasks can be improved.

[0083] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0084] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0085] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0086] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0087] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0088] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0089] If the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, the personal information processing rules are notified by obvious signs / information, and the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

Claims

1. An image processing method, characterized in that: include: Extracting an image feature representation of the image to be processed, and quantizing an image processing model deployed on the current device based on a quantization tool on the current device to obtain a first quantized model; wherein the image processing model is obtained at least through cropping training, and the cropping coefficient is decreased during the cropping process, and the cropping coefficient is used to determine the numerical range of the weight parameter in the cropped image processing model; Processing the image feature representation based on the first quantization model to obtain a temporary processing result; The temporary processing result is dequantized based on the quantization tool to obtain a final processing result of the image to be processed.

2. The method according to claim 1, characterized in that The training steps of the image processing model include: Initializing the cropping factor; In each round of iteration of the image processing model: based on the current values ​​of the weight parameter and the cropping coefficient in the image processing model, a first range is obtained, and based on the first range, the weight parameter in the image processing model is cropped, and based on the running result of the image processing model after the cropping, the weight parameter before the cropping is adjusted; The crop factor is decremented and a new iteration is performed on the image processing model again.

3. The method according to claim 2, characterized in that The step of decreasing the crop factor comprises: Based on the current iteration round, determining a magnitude value of decreasing the clipping coefficient; wherein the current iteration round is negatively correlated with the magnitude value; The crop factor is decremented based on the magnitude value of the decremented crop factor.

4. The method according to claim 2, characterized in that: The step of decreasing the crop factor comprises: The cropping factor is decreased by a preset value; wherein the preset value remains constant during the training process of the image processing model.

5. The method according to claim 2, characterized in that: The obtaining of the first range based on the current value of the weight parameter and the cropping coefficient in the image processing model comprises: Calculating an average value based on the absolute value of the weight parameter in the image processing model; Based on the product of the average value and the current value, an upper limit value of the first range is obtained; wherein the lower limit value of the first range and the upper limit value are reciprocal numbers of each other.

6. The method according to claim 1, characterized in that After the clipping training is completed, the method further includes: Obtaining a second range based on final values ​​of the weight parameter and the cropping coefficient in the image processing model after the cropping training is completed; Based on the second range, cutting the weight parameter in the image processing model; Quantization training is performed based on the image processing model after being cropped according to the second range to obtain an image processing model for deployment on the current device.

7. The method according to claim 6, characterized in that Before the quantization training, each network layer in the image processing model is initialized with a first scaling factor for quantizing input parameters / output parameters, and for any of the network layers, the initialization step of the first scaling factor includes: Initialize the statistical variables of the target parameters for the network layer, and obtain the maximum absolute value of the target parameters of the network layer when the image processing model processes the sample images in the calibration set; wherein, when initializing the first scaling factor of the input parameter, the target parameter is the input parameter, and when initializing the first scaling factor of the output parameter, the target parameter is the output parameter; Performing weighting based on the current value and the maximum absolute value of the statistical variable to obtain a new current value of the statistical variable, and returning, for the next sample image in the calibration set, the maximum absolute value of the target parameter of the network layer when the image processing model is used to process the sample image in the calibration set, until all the sample images in the calibration set are processed; Based on the latest current value of the statistical variable, a first scaling factor expressed as an exponential power of 2 is obtained.

8. The method according to claim 6, characterized in that Before the quantization training, each network layer in the image processing model is initialized with a second scaling factor for quantizing weight parameters, and for any of the network layers, the initialization step of the second scaling factor includes: Taking the value obtained by traversing within the preset value range as the exponential power of 2, a candidate scaling factor of the weight parameter is obtained; Obtaining a quantization deviation of the candidate scaling factor based on the weight parameter and the weight parameter obtained by quantizing the weight parameter using the candidate scaling factor and then dequantizing the weight parameter; Based on the quantization deviations of the candidate scaling factors, the candidate scaling factors are selected as the second scaling factors.

9. The method according to claim 2 or 6, characterized in that: The weight parameters in each network layer in the image processing model are trimmed layer by layer.

10. The method according to claim 1, characterized in that The quantization tools include at least one of the following: W4A8 quantization, symmetric quantization, linear quantization, pertensor quantization, and a scaling factor of 2.

11. A speech processing method, characterized in that: include: Extracting speech feature representation of the speech to be processed, and quantizing the speech processing model deployed on the current device based on the quantization tool on the current device to obtain a second quantized model; wherein the speech processing model is obtained by at least trimming training, and the trimming coefficient is reduced during the trimming process, and the trimming coefficient is used to determine the numerical range of the weight parameter in the trimming of the speech processing model; Processing the speech feature representation based on the second quantization model to obtain a temporary processing result; The temporary processing result is dequantized based on the quantization tool to obtain a final processing result of the speech to be processed.

12. A natural language processing method, characterized in that: include: Extracting a sentence feature representation of the sentence to be processed, and quantizing the natural language processing model deployed on the current device based on a quantization tool on the current device to obtain a third quantized model; wherein the natural language processing model is obtained by at least trimming training, and the trimming coefficient is reduced during the trimming process, and the trimming coefficient is used to determine the numerical range of the weight parameter in the trimming of the natural language processing model; Processing the sentence feature representation based on the third quantization model to obtain a temporary processing result; The temporary processing result is dequantized based on the quantization tool to obtain a final processing result of the statement to be processed.

13. An image processing device, characterized in that: include: An extraction and quantization module, used to extract the image feature representation of the image to be processed, and quantize the image processing model deployed on the current device based on the quantization tool on the current device to obtain a first quantization model; wherein the image processing model is obtained by at least cropping training, and the cropping coefficient is decreased during the cropping process, and the cropping coefficient is used to determine the numerical range of the weight parameter in the cropped image processing model; A model processing module, used for processing the image feature representation based on the first quantization model to obtain a temporary processing result; The dequantization module is used to dequantize the temporary processing result based on the quantization tool to obtain the final processing result of the image to be processed.

14. A speech processing device, characterized in that: include: An extraction and quantization module is used to extract the speech feature representation of the speech to be processed, and quantize the speech processing model deployed on the current device based on the quantization tool on the current device to obtain a second quantization model; wherein the speech processing model is obtained by at least trimming training, and the trimming coefficient is reduced during the trimming process, and the trimming coefficient is used to determine the numerical range of the weight parameter in the trimming speech processing model; A model processing module, used for processing the speech feature representation based on the second quantization model to obtain a temporary processing result; The dequantization module is used to dequantize the temporary processing result based on the quantization tool to obtain the final processing result of the speech to be processed.

15. A natural language processing device, characterized in that: include: An extraction and quantization module is used to extract the sentence feature representation of the sentence to be processed, and quantize the natural language processing model deployed on the current device based on the quantization tool on the current device to obtain a third quantization model; wherein the natural language processing model is obtained by at least trimming training, and the trimming coefficient is reduced during the trimming process, and the trimming coefficient is used to determine the numerical range of the weight parameter in the trimming of the natural language processing model; A model processing module, used for processing the sentence feature representation based on the third quantitative model to obtain a temporary processing result; A dequantization module is used to dequantize the temporary processing result based on the quantization tool to obtain the final processing result of the statement to be processed.

16. An electronic device, characterized in that: The method comprises at least a memory and a processor coupled to each other, wherein the memory at least stores program instructions, and the processor is used to execute the program instructions to implement the image processing method according to any one of claims 1 to 10, or the speech processing method according to claim 11, or the natural language processing method according to claim 12.

17. A computer-readable storage medium, characterized in that: Program instructions that can be executed by a processor are stored, and the program instructions are used to implement the image processing method described in any one of claims 1 to 10, or the speech processing method described in claim 11, or the natural language processing method described in claim 12.

Images