Image processing model compression method, apparatus, and device for terminal deployment

Abstract

This application discloses an image processing model compression method, apparatus, and device for terminal deployment, belonging to the field of computer technology. Based on a first quantization step size, the floating-point model parameters of the image processing model to be compressed are quantized and dequantized to obtain dequantized floating-point model parameters; the first quantization step size is mapped according to a target mapping relationship to obtain first trainable parameters; the image processing model configured with the dequantized floating-point model parameters is trained, and during training, the floating-point model parameters and the first trainable parameters are adjusted; the adjusted first trainable parameters are demapped according to the target mapping relationship to obtain a second quantization step size; the adjusted floating-point model parameters are quantized based on the second quantization step size to obtain second integer model parameters; the second integer model parameters are configured for the image processing model, and the second quantization step size is configured into the image processing model, thereby improving the model compression effect.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an image processing model compression method, apparatus, and device for terminal deployment. Background Technology

[0002] Because image processing models involve large amounts of data and terminals have limited computing power, running them directly on the terminal can cause lag and affect normal operation. Therefore, before deploying the image processing model on the terminal, it needs to be quantized to compress the large dataset into a smaller one. The compressed model is then deployed to the terminal to ensure smooth operation. However, quantization introduces quantization errors. Therefore, determining the optimal quantization step size (scale) to minimize these errors is a critical problem that needs to be solved.

[0003] The method for determining the optimal quantization step size is as follows: a pseudo-quantization node is inserted into the image processing model. This pseudo-quantization node quantizes the model parameters in the image processing model based on the quantization step size (scale) and then performs inverse quantization. The model parameters of the image processing model are updated with the inverse quantized model parameters. Based on the loss function of the image processing model, the quantization step size is updated, thereby reducing the error introduced by quantization and determining the optimal quantization step size.

[0004] However, in practical applications, it has been found that the quantization step size is sometimes updated to a negative value during training. The physical meaning of the quantization step size is the step size of the numerical mapping, and it should remain non-negative. If the quantization step size is trained to a negative value, it will prevent the image processing model from being quantized, thus preventing image processing model compression. The image processing model can only be deployed directly to the terminal. Due to the limited computing power of the terminal, the image processing model often experiences lag when running on the terminal, and in severe cases, it can even cause the terminal to crash. Summary of the Invention

[0005] This application provides an image processing model compression method, apparatus, and device for terminal deployment, which improves the accuracy of quantization step size and thus enhances model compression performance. The technical solution is as follows:

[0006] On the one hand, an image processing model compression method for terminal deployment is provided, the method comprising:

[0007] Based on the first quantization step size, the floating-point model parameters of the image processing model to be compressed are quantized to obtain the first integer model parameters. Based on the first quantization step size, the first integer model parameters are dequantized to obtain the dequantized floating-point model parameters. The dequantized floating-point model parameters are then configured into the image processing model.

[0008] According to the target mapping relationship, the first quantization step size is mapped to obtain the first trainable parameter. The first trainable parameter is used to replace the first quantization step size in model training. The target mapping relationship is used to represent the mapping relationship between the quantization step size and the trainable parameter.

[0009] Based on the sample images, the image processing model configured with the dequantized floating-point model parameters is trained. During the training process, the floating-point model parameters and the first trainable parameters of the image processing model are adjusted to reduce the quantization error introduced into the image processing model.

[0010] Based on the target mapping relationship, the adjusted first trainable parameters are inversely mapped to obtain the second quantization step size, which is a positive number.

[0011] Based on the second quantization step size, the adjusted floating-point model parameters are quantized to obtain the second integer model parameters;

[0012] Configure the second integer model parameters for the image processing model and configure the second quantization step size in each layer of the image processing model to obtain a compressed image processing model. Implant the compressed image processing model into the terminal. The second quantization step size in the first layer of the compressed image processing model is used to quantize the input floating-point image data into integer image data. The second quantization step size in the Nth layer of the compressed image processing model is used to quantize the output data of the previous layer to reduce the amount of output data. N is a positive integer greater than 1.

[0013] On one hand, an image processing model compression device for terminal deployment is provided, the device comprising:

[0014] The pseudo-quantization module is used to quantize the floating-point model parameters of the image processing model to be compressed based on a first quantization step size to obtain a first integer model parameter, dequantize the first integer model parameter based on the first quantization step size to obtain the dequantized floating-point model parameter, and configure the dequantized floating-point model parameter into the image processing model.

[0015] The mapping module is used to map the first quantization step size according to the target mapping relationship to obtain the first trainable parameter. The first trainable parameter is used to replace the first quantization step size in the model training. The target mapping relationship is used to represent the mapping relationship between the quantization step size and the trainable parameter.

[0016] The training module is used to train an image processing model with the dequantized floating-point model parameters based on sample images. During the training process, the floating-point model parameters and the first trainable parameters of the image processing model are adjusted to reduce the quantization error introduced into the image processing model.

[0017] The mapping module is further configured to perform inverse mapping on the adjusted first trainable parameters according to the target mapping relationship to obtain a second quantization step size, wherein the second quantization step size is a positive number;

[0018] The pseudo-quantization module is also used to quantize the adjusted floating-point model parameters based on the second quantization step size to obtain the second integer model parameters;

[0019] A configuration module is used to configure the second integer model parameters for the image processing model and configure the second quantization step size in each layer of the image processing model to obtain a compressed image processing model. The compressed image processing model is then implanted into a terminal. The second quantization step size in the first layer of the compressed image processing model is used to quantize the input floating-point image data into integer image data. The second quantization step size in the Nth layer of the compressed image processing model is used to quantize the output data of the previous layer to reduce the amount of output data. N is a positive integer greater than 1.

[0020] In one possible implementation, the target mapping relationship represents that the trainable parameter is the square root of the quantization step size; the mapping module is used to take the square root of the first quantization step size to obtain the first trainable parameter.

[0021] The mapping module is used to determine the square of the adjusted first trainable parameter as the second quantization step size.

[0022] In one possible implementation, the device further includes:

[0023] The first determining module is used to determine the parameter range of the image processing model based on the floating-point model parameters;

[0024] The second determining module is used to determine the quantization value range based on the bit width of the integer parameter, wherein the quantization value range represents the value range of the quantized floating-point model parameter;

[0025] The third determining module is used to determine the first quantization step size based on the parameter range and the quantization value range.

[0026] In one possible implementation, the third determining module is used to determine the ratio of the first difference to the second difference as the first quantization step size, wherein the first difference is the difference between the maximum and minimum values ​​in the parameter range, and the second difference is the difference between the maximum and minimum values ​​in the quantization value range.

[0027] In one possible implementation, there are multiple floating-point model parameters; the first determining module is used to determine an average floating-point tensor based on the floating-point tensors of the multiple floating-point model parameters; the maximum negative value represented by the average floating-point tensor is determined as the minimum value of the parameter range, and the maximum positive value represented by the average floating-point tensor is determined as the maximum value of the parameter range, thereby obtaining the parameter range.

[0028] In one possible implementation, the second determining module is configured to determine, based on the bit width, the maximum positive value and the maximum negative value that the integer parameter can represent; determine the maximum positive value as the maximum value of the quantization value range, and determine the maximum negative value as the minimum value of the quantization value range, thereby obtaining the quantization value range.

[0029] On one hand, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one piece of program code, the at least one piece of program code being loaded and executed by the processor to perform the operations performed in the image processing model compression method for terminal deployment as described in any of the above possible implementations.

[0030] On the one hand, a computer-readable storage medium is provided, which stores at least one piece of program code that is loaded and executed by a processor to perform the operations performed by the image processing model compression method for terminal deployment as described in any of the above possible implementations.

[0031] On one hand, a computer program or computer program product is provided, the computer program or computer program product comprising: computer program code, which, when executed by a computer, causes the computer to perform the operations performed by the image processing model compression method for terminal deployment as described in any of the above possible implementations.

[0032] The image processing model compression method for terminal deployment provided in this application does not directly train the quantization step size during model quantization. Instead, it sets a target mapping relationship to map the quantization step size into trainable parameters, which are then used for training instead of the quantization step size. Since the trained parameters, after adjustment, are mapped to a positive number based on the target mapping relationship, the quantization step size is guaranteed to be positive, thus enabling successful compression of the image processing model. Deploying the compressed image processing model to the terminal reduces the likelihood of lag or crashes. Because the quantization step size determined in this application is more accurate, the quantization error of the compressed image processing model is smaller, reducing the quantization error introduced by model quantization. This allows the terminal to obtain accurate inference results, improving the accuracy of the compressed image processing model. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is a schematic diagram of parameter quantization provided in an embodiment of this application;

[0035] Figure 2 This is a schematic diagram of an implementation environment provided in an embodiment of this application;

[0036] Figure 3 This is a flowchart of an image processing model compression method for terminal deployment provided in an embodiment of this application;

[0037] Figure 4 This is a flowchart of an image processing model compression method for terminal deployment provided in an embodiment of this application;

[0038] Figure 5 This is a schematic diagram illustrating the addition of pseudo-quantization nodes to an image processing model, as provided in an embodiment of this application.

[0039] Figure 6 This is a flowchart of an image processing model provided in an embodiment of this application;

[0040] Figure 7 This is a schematic diagram of the gradient of the quantization step size and the gradient of the trainable parameters provided in an embodiment of this application;

[0041] Figure 8This is a schematic diagram of the structure of an image processing model compression device for terminal deployment provided in an embodiment of this application;

[0042] Figure 9 This is a schematic diagram of the structure of an image processing model compression device for terminal deployment provided in an embodiment of this application;

[0043] Figure 10 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application;

[0044] Figure 11 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0046] It is understood that the terms "first," "second," etc., used in this application may be used to describe various concepts herein, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of this application, the first quantization step size may be referred to as the second quantization step size, and the second quantization step size may be referred to as the first quantization step size.

[0047] As used in this application, the terms "at least one", "multiple", "each", and "any" have the following meanings: at least one includes one, two, or more; multiple includes two or more; each refers to each of the corresponding multiple; and any refers to any one of the multiple. For example, multiple model parameters include three model parameters, where each refers to each of the three model parameters, and any refers to any one of the three model parameters, which could be the first, the second, or the third.

[0048] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the sample images involved in this application were obtained with full authorization.

[0049] With the widespread application of neural networks, their powerful reasoning capabilities have yielded good results in the two major fields of CV (Computer Vision) and NLP (Natural Language Processing). Meanwhile, due to the widespread use of mobile devices, people spend a significant amount of time processing information on them in their daily lives. However, the large data size of neural networks and the substantial resources required for inference make them difficult to use on mobile devices. In response to this situation, a technology that enables neural networks to run smoothly on mobile devices has emerged: neural network quantization technology.

[0050] Quantization is a method of mapping continuous real numbers to a set of discrete values ​​through linear mapping and rounding operations. In other words, quantization refers to dividing the numerical range of floating-point data into multiple intervals, selecting a fixed point from each interval (either directly or through rounding) as integer data, and the quantization step size represents the length of the divided intervals. Figure 1 As shown, V f This represents floating-point model parameters, which are continuous real numbers. [V] amin V amax ] represents the floating-point model parameter V. f The numerical range of V amin V represents f The minimum value of V amax V represents f The maximum value of V. `scale` represents the quantization step size. After determining the quantization step size `scale`, V is quantized based on the quantization step size `scale`. f Mapped to a set of discrete integer parameters, which are: Figure 1 Q in n To Q p Q n Q represents the minimum value of this set of discrete integer parameters. p This represents the maximum value of the set of discrete integer parameters.

[0051] Model quantization involves converting model parameters from floating-point data to integer data. For example, before quantization, the model parameters of a neural network model are represented using `float` (32-bit) values, while after quantization, they are represented using `int` (8-bit) values. Converting model parameters from floating-point to integer data not only compresses the size of the neural network model but also transforms floating-point operations into integer operations, thereby improving the computational speed of the neural network model.

[0052] The image processing model compression method for terminal deployment provided in this application embodiment can be applied to model quantization scenarios. Through the target mapping relationship, the quantization step size is mapped to trainable parameters to replace the quantization step size in model training. After training is completed, the trainable parameters are mapped back to the quantization step size. This target mapping relationship can map both positive and negative trainable parameters to positive quantization step sizes. Therefore, it avoids the problem of directly training the quantization step size and obtaining a negative quantization step size, which leads to training errors.

[0053] It should be noted that the embodiments of this application are merely illustrative examples of the model quantization process of neural network models, using the image processing model compression method for terminal deployment as an example. Therefore, the image processing model compression method for terminal deployment can also be applied to speech processing models.

[0054] The image processing model compression method for terminal deployment provided in this application is executed by a computer device. In some embodiments, the computer device is a server, which can be a single server, a server cluster consisting of several servers, or a cloud computing service center. In other embodiments, the computer device is a terminal, which can be any type of terminal such as an autonomous vehicle, mobile phone, computer, or tablet computer. Optionally, the autonomous vehicle includes vehicles traveling on the ground (e.g., cars, trucks, buses, etc.), vehicles traveling in the air (e.g., drones, airplanes, helicopters, etc.), and vehicles traveling on water (e.g., ships, submarines, etc.). The autonomous vehicle may or may not accommodate one or more passengers. Furthermore, the autonomous vehicle can be applied to unmanned delivery fields, such as express delivery and food delivery. In other embodiments, the computer device includes both a terminal and a server.

[0055] Figure 2 This is a schematic diagram of an implementation environment provided in an embodiment of this application, such as... Figure 2 As shown, the implementation environment includes a terminal 201 and a server 202, which are connected via a wireless or wired network.

[0056] In some embodiments, the server 202 is used to quantize the neural network model and deploy the quantized neural network model to the terminal 201. The neural network model is a model for reasoning about images or speech. Optionally, the neural network model is an image processing model for reasoning about images. For example, the neural network model can be used for image classification, that is, identifying the category to which an object in an image belongs; or for object detection, that is, identifying target objects from an image; or for image recognition, that is, identifying objects of a certain category in an image, or identifying object information (such as identifying the model of a vehicle) in an image. Optionally, the neural network model is a speech processing model for reasoning about speech. For example, the neural network model can be used for speech recognition, that is, recognizing speech as corresponding text information; or for intelligent question answering, that is, outputting speech for answering based on received speech. This application does not limit the neural network model.

[0057] It should be noted that the embodiments in this application are merely illustrative examples of the quantization process of neural network models, using an image processing model as an example. Therefore, Figure 3 and Figure 4 The image processing model in the illustrated embodiment can also be replaced with any other type of neural network model, such as a speech processing model.

[0058] Figure 3 This is a flowchart illustrating an image processing model compression method for terminal deployment provided in this application embodiment. This application embodiment uses a computer device as the execution subject for illustrative purposes; the embodiment includes:

[0059] 301. The computer device quantizes the floating-point model parameters of the image processing model to be compressed based on the first quantization step size to obtain the first integer model parameters. Based on the first quantization step size, the first integer model parameters are dequantized to obtain the dequantized floating-point model parameters. The dequantized floating-point model parameters are then configured into the image processing model.

[0060] Quantization can be viewed as dividing the range of values ​​represented by floating-point data into multiple intervals, and taking a fixed point from each interval (either directly or by rounding) as integer data. The quantization step size represents the length of the divided intervals. The first quantization step size can be the initial quantization step size, or it can be obtained by adjusting the initial quantization step size during model training of an image processing model with dequantized floating-point model parameters based on sample images. In other words, the quantization process of the image processing model is an iterative process, and step 301 above can be the step of the first iteration or the step of the Nth iteration. N is any integer greater than 1.

[0061] In this embodiment, the model parameters of the image processing model are floating-point model parameters. Floating-point model parameters refer to model parameters represented in float32 format. Floating-point model parameters occupy a large amount of storage space, and floating-point operations are time-consuming. Therefore, this embodiment converts the floating-point model parameters of the image processing model into integer model parameters through quantization, thereby obtaining a quantized image processing model. This reduces the size of the image processing model. Furthermore, since the model parameters in the quantized image processing model are integer model parameters, the quantized image processing model uses integer operations, resulting in faster operation speed.

[0062] Specifically, based on the first quantization step size, the floating-point model parameters are quantized to obtain the first integer model parameters. This involves dividing the numerical range represented by the floating-point model parameters into multiple value intervals according to the first quantization step size, selecting a fixed value, such as the median value, from each value interval, and directly using it as the first integer model parameter, or performing a rounding operation to obtain the first integer model parameter.

[0063] In some embodiments, the first quantization step size, the floating-point model parameter, and the first integer model parameter satisfy the following relationship:

[0064] V q =round(V f / scale);

[0065] Among them, V q This represents the first integer parameter, and Round represents the rounding function, which can perform rounding. V f This represents the floating-point model parameters, where scale represents the first quantization step size, and V... f / scale indicates that, according to the first quantization step, a value range is determined from the numerical range represented by the floating-point model parameters, and a fixed point is selected from that value range.

[0066] Furthermore, based on the first quantization step size, the first integer model parameters are dequantized to obtain the dequantized floating-point model parameters. This involves using the first integer model parameters as intermediate values ​​and mapping them to a numerical range according to the first quantization step size, resulting in the dequantized floating-point model parameters. The length of this numerical range is the first quantization step size. This dequantization process is the reverse of the quantization process.

[0067] In some embodiments, the first quantization step size, the dequantized floating-point model parameters, and the first integer model parameters satisfy the following relationship:

[0068]

[0069] in, Represents the dequantized floating-point model parameters, Round(V) f ` / scale` represents the first integer parameter, and `Round` represents the rounding function used for rounding. V f This represents the floating-point model parameters, and scale represents the first quantization step size.

[0070] It should be noted that, because the selected fixed values ​​are rounded during the quantization process to obtain the first integer model parameters, the dequantized floating-point model parameters obtained by dequantizing the first integer model parameters based on the first quantization step size are not exactly the same as the floating-point model parameters of the image processing model. Therefore, quantizing the floating-point model parameters of the image processing model introduces quantization error. Subsequent model training is required to adjust the floating-point model parameters and the first quantization step size of the image processing model to reduce quantization error.

[0071] 302. The computer device maps the first quantization step size according to the target mapping relationship to obtain the first trainable parameter. The first trainable parameter is used to replace the first quantization step size in the model training. The target mapping relationship is used to represent the mapping relationship between the quantization step size and the trainable parameter.

[0072] It should be noted that, in order to reduce quantization error, when training an image processing model and adjusting the quantization step size and floating-point model parameters, the quantization step size may be trained to a negative value, resulting in training errors and the model being unable to quantize. To solve this problem, embodiments of this application propose a method of using trainable parameters instead of the first quantization step size in model training.

[0073] This application embodiment sets a target mapping relationship, which is used to represent the mapping relationship between quantization step size and trainable parameters. The computer device can map the quantization step size to trainable parameters or the trainable parameters to quantization step size based on the target mapping relationship. Moreover, regardless of whether the trainable parameter is a positive or negative value, the trainable parameter can be mapped to a positive quantization step size based on the target mapping relationship.

[0074] Alternatively, to avoid the trainable parameters replacing the first quantization step size in training and affecting the training effect, the target mapping relationship is represented as a target function. In this way, the trainable parameters and the quantization step size have a functional relationship, so even if the trainable parameters replace the first quantization step size in model training, it will not have a significant negative impact.

[0075] Optionally, the objective function represents the trainable parameter as the square root of the quantization step size; alternatively, the objective function represents the quantization step size as the absolute value of the trainable parameter; alternatively, the objective function represents the trainable parameter as half of the square root of the quantization step size, etc. The embodiments of this application do not limit the objective function.

[0076] 303. The computer equipment trains an image processing model with dequantized floating-point model parameters based on sample images. During the training process, the floating-point model parameters and the first trainable parameter of the image processing model are adjusted to reduce the quantization error introduced into the image processing model.

[0077] After obtaining the dequantized floating-point model parameters, the computer equipment configures these parameters into the image processing model, introducing quantization error. Therefore, the image processing model needs to be trained again. Since this quantization error is caused by quantizing the floating-point model parameters in the image processing model, this training does not adjust the current model parameters of the image processing model—the dequantized floating-point model parameters—but rather adjusts the first trainable parameter used to replace the first quantization step size in model training and the original model parameters of the image processing model—the floating-point model parameters of the image processing model in step 301 above.

[0078] 304. The computer device performs inverse mapping on the adjusted first trainable parameters according to the target mapping relationship to obtain the second quantization step size, which is a positive number.

[0079] This target mapping relationship ensures that both positive and negative trainable parameters are mapped to positive values. Therefore, mapping the adjusted first trainable parameter according to this target mapping relationship results in a positive second quantization step size. Thus, this target mapping relationship guarantees that the adjusted quantization step size is positive, preventing the quantization step size from being trained to a negative value, which could lead to training errors and the model's inability to quantize.

[0080] 305. The computer equipment quantizes the adjusted floating-point model parameters based on the second quantization step size to obtain the second integer parameters.

[0081] The steps 305 above are the same as those in step 301 above, which are "quantizing the floating-point model parameters of the image processing model based on the first quantization step size to obtain the first integer parameters", and will not be described in detail here.

[0082] 306. The computer device configures the second integer model parameters for the image processing model and configures the second quantization step size in each layer of the image processing model to obtain the compressed image processing model. The compressed image processing model is then implanted into the terminal. The second quantization step size in the first layer of the compressed image processing model is used to quantize the input floating-point image data into integer image data. The second quantization step size in the Nth layer of the compressed image processing model is used to quantize the output data of the previous layer to reduce the amount of output data. N is a positive integer greater than 1.

[0083] The computer equipment configures a second integer model parameter for the image processing model, thereby obtaining the quantized image processing model, which is also the compressed image processing model. Since the model parameters in the compressed image processing model are integer model parameters, the image processing model performs integer operations. Therefore, it is also necessary to configure a second quantization step size into the image processing model to convert the input floating-point image into an integer image, so that the image processing model can perform integer operations on the integer image.

[0084] Since the image processing model converts int8 data into int32 data during processing, in order to reduce the computational load of the model and reduce lag or crashes caused by the terminal running the image processing model, this embodiment configures the second quantization step size in each layer of the image processing model. The second quantization step size in the first layer is used to quantize the input floating-point image data into integer image data. The second quantization step size in other layers after the first layer is used to quantize the output data of the previous layer to reduce the amount of output data, that is, to quantize the int32 data output by the previous layer into int8 data.

[0085] The image processing model compression method for terminal deployment provided in this application does not directly train the quantization step size during model quantization. Instead, it sets a target mapping relationship to map the quantization step size into trainable parameters, which are then used for training instead of the quantization step size. Since the trained parameters, after adjustment, are mapped to a positive number based on the target mapping relationship, the quantization step size is guaranteed to be positive, thus enabling successful compression of the image processing model. Deploying the compressed image processing model to the terminal reduces the likelihood of lag or crashes. Because the quantization step size determined in this application is more accurate, the quantization error of the compressed image processing model is smaller, reducing the quantization error introduced by model quantization. This allows the terminal to obtain accurate inference results, improving the accuracy of the compressed image processing model.

[0086] Figure 4This is a flowchart illustrating an image processing model compression method for terminal deployment, provided in an embodiment of this application. This embodiment uses a computer device as an example for illustrative purposes; the embodiment includes:

[0087] 401. Computer equipment trains image processing models based on sample images.

[0088] The image processing model can be any model used for reasoning about images, such as an image classification model, an object detection model, or an image recognition model.

[0089] Different image processing models use different sample images. When the image processing model is an image classification model, the sample image can be an image of objects of multiple categories. The computer device trains the image processing model based on the sample image, including: calling the image classification model to classify the sample image to obtain the predicted category; and training the image processing model based on the predicted category and the sample category corresponding to the sample image.

[0090] When the image processing model is an object detection model, the sample image can be an image that includes the target object, or an image that does not include the target object. The computer device trains the image processing model based on the sample image, including: calling the object detection model to perform object detection on the sample image to obtain a predicted detection result; and training the object detection model based on the predicted detection result and the corresponding sample detection result for the sample image.

[0091] When the image processing model is an image recognition model, the sample image includes the target object, and the image recognition model is used to identify the object information of the target object in the image. The computer device trains the image processing model based on the sample image, including: calling the image recognition model to recognize the sample image and obtain a predicted recognition result; and training the image recognition model based on the predicted recognition result and the corresponding sample recognition result.

[0092] In some embodiments, the computer device trains an image processing model based on sample images until the accuracy of the trained image processing model reaches the target requirement, and performs the following steps.

[0093] 402. The computer device acquires the floating-point model parameters of the trained image processing model, and determines the first quantization step size based on the floating-point model parameters.

[0094] In this embodiment, the first quantization step size is the initial quantization step size. It should be noted that if the initial quantization step size used by the computer device is unreasonable, it will cause the subsequent training of the image processing model to reduce quantization error to take too long, or even result in an inaccurate quantization step size after training.

[0095] To address the aforementioned issues, this application proposes a method for initializing the quantization step size using the floating-point model parameter distribution of a trained image processing model. In some embodiments, a computer device determines a first quantization step size based on the floating-point model parameters, including: determining the parameter range of the image processing model based on the floating-point model parameters; determining a quantization value range based on the bit width of the integer parameters, where the quantization value range represents the value range of the quantized floating-point model parameters; and determining the first quantization step size based on the parameter range and the quantization value range. The bit width of the integer parameters is a specified bit width, for example, 8 bits.

[0096] Optionally, the computer device determines the first quantization step size based on the parameter range and the quantization value range, including: determining the ratio of the first difference to the second difference as the first quantization step size, wherein the first difference is the difference between the maximum and minimum values ​​in the parameter range, and the second difference is the difference between the maximum and minimum values ​​in the quantization value range.

[0097] For example, the first quantization step size, the maximum value in the parameter range, the minimum value in the parameter range, the maximum value in the quantization range, and the minimum value in the quantization range satisfy the following relationship:

[0098] scale = (v amax -v amin ) / (Q p -Q n );

[0099] Where scale represents the first quantization step size, v amax v represents the maximum value within the parameter range. amin Q represents the minimum value in the parameter range. p Q represents the maximum value within the quantized range. n This represents the minimum value within the range of quantized values.

[0100] In some embodiments, the parameter range can be determined based on an average floating-point tensor. There are multiple floating-point model parameters. The computer device determines the parameter range of the image processing model based on these floating-point model parameters, including: determining an average floating-point tensor based on the floating-point tensors of the multiple floating-point model parameters; determining the maximum negative value represented by the average floating-point tensor as the minimum value of the parameter range, and determining the maximum positive value represented by the average floating-point tensor as the maximum value of the parameter range, thus obtaining the parameter range.

[0101] In other embodiments, the parameter range can be determined based on the distribution of floating-point model parameters. There are multiple floating-point model parameters. The computer device determines the parameter range of the image processing model based on these floating-point model parameters, including: the computer device acquiring floating-point tensors of the multiple floating-point model parameters; using a histogram or entropy method to extract the maximum positive and maximum negative values ​​represented by the floating-point tensors of the multiple floating-point model parameters; determining the maximum positive value as the maximum value of the parameter range; and determining the maximum negative value as the minimum value of the parameter range, thus obtaining the parameter range.

[0102] Since the bit width of the integer parameter determines the range of values ​​for the quantized model parameters, the quantization range is calculated based on the bit width of the integer parameter. In some embodiments, the computer device determines the quantization range based on the bit width of the integer parameter, including: the computer device determining the maximum positive value and the maximum negative value that the integer parameter can represent based on the bit width; determining the maximum positive value as the maximum value of the quantization range, and determining the maximum negative value as the minimum value of the quantization range, thereby obtaining the quantization range.

[0103] Optionally, the bit width, the maximum positive value that the integer parameter can represent, and the maximum negative value that the integer parameter can represent satisfy the following relationship:

[0104] Q p =2 n-1 -1;

[0105] Q n =-2 n-1 ;

[0106] Among them, Q p Q is the largest positive value that an integer parameter can represent. n n is the maximum negative value that the integer parameter can represent, and n is the bit width. For example, if the integer parameter is 8 bits, then n is 8.

[0107] 403. The computer device quantizes the floating-point model parameters based on the first quantization step size to obtain the first integer model parameters, and dequantizes the first integer model parameters based on the first quantization step size to obtain the dequantized floating-point model parameters.

[0108] In some embodiments, such as Figure 5As shown, the computer device adds a pseudo-quantization node to the image processing model. This pseudo-quantization node includes a quantization sub-node and an inverse quantization sub-node. After obtaining the first quantization step size, the computer device configures the first quantization step size to the quantization and inverse quantization sub-nodes. Based on the first quantization step size, the computer device quantizes the floating-point model parameters to obtain the first integer model parameters. Based on the first quantization step size, the computer device inverse quantizes the first integer model parameters to obtain the inverse floating-point model parameters. This includes: the computer device inputs the floating-point model parameters of the image processing model into the quantization sub-node; through the quantization sub-node, based on the first quantization step size, quantizes the floating-point model parameters to obtain the first integer model parameters; inputs the first integer model parameters into the inverse quantization sub-node; through the inverse quantization sub-node, based on the first quantization step size, inverse quantizes the first integer model parameters to obtain the inverse floating-point model parameters; and then, the inverse floating-point model parameters are configured into the image processing model.

[0109] Optionally, the image processing model includes multiple sets of processing layers, each set including at least one convolutional layer and at least one activation layer. The computer device adds a pseudo-quantization node before each set of processing layers. This pseudo-quantization node is used to process the floating-point model parameters of the connected processing layers. These floating-point model parameters include floating-point weights in the convolutional layers and floating-point activation values ​​in the activation layers. The first quantization step size configured in the pseudo-quantization node is determined based on the floating-point model parameters of the processing layers to which the pseudo-quantization node is connected; therefore, different pseudo-quantization nodes can use different first quantization step sizes. Optionally, the floating-point model parameters are weight values ​​and activation values ​​in the image processing model.

[0110] It should be noted that the embodiments in this application only illustrate the quantization process of the model by taking the quantization of model parameters as an example. In another embodiment, it is also necessary to quantize the input data and output data of the processing layer. Optionally, the image processing model includes multiple processing layers. The computer device adds one or more pseudo-quantization nodes to each processing layer. For example, the computer device adds three pseudo-quantization nodes to the processing layer: one pseudo-quantization node is used to process the input data of the processing layer, one pseudo-quantization node is used to process the output data of the processing layer, and one pseudo-quantization node is used to process the model parameters of the processing layer. The quantization step size used by these three pseudo-quantization nodes is the quantization step size corresponding to the processing layer, and the quantization step size used by different pseudo-quantization nodes has different functions. As another example, the computer device adds two pseudo-quantization nodes to the processing layer: one pseudo-quantization node is used to process the input data of the processing layer, and one pseudo-quantization node is used to process the model parameters of the processing layer. The quantization step size used by these two pseudo-quantization nodes is the quantization step size corresponding to the processing layer, and the quantization step size used by different pseudo-quantization nodes has different functions.

[0111] 404. The computer device maps the first quantization step size based on the target mapping relationship to obtain the first trainable parameter. The first trainable parameter is used to replace the first quantization step size in the model training. The target mapping relationship is used to represent the mapping relationship between the quantization step size and the trainable parameter.

[0112] In some embodiments, the target mapping relationship indicates that the trainable parameter is the square root of the quantization step size. Therefore, the quantization step size is the square of the trainable parameter. Thus, regardless of whether the trainable parameter is positive or negative, the quantization step size determined based on the trainable parameter must be positive. Optionally, the computer device maps the first quantization step size according to the target mapping relationship to obtain the first trainable parameter, including: the computer device taking the square root of the first quantization step size to obtain the first trainable parameter.

[0113] In some embodiments, different pseudo-quantization nodes use different first quantization step sizes; therefore, there are multiple first quantization step sizes. The computer device maps the first quantization step sizes based on a target mapping relationship to obtain first trainable parameters, including: the computer device mapping the first quantization step size used by each pseudo-quantization node based on the target mapping relationship to obtain first trainable parameters corresponding to each first quantization step size.

[0114] 405. The computer equipment configures the dequantized floating-point model parameters into the image processing model. Based on the sample image, the image processing model with the dequantized floating-point model parameters is trained. During the training process, the floating-point model parameters and the first trainable parameter of the image processing model are adjusted.

[0115] In some embodiments, the sample image is a floating-point sample image. The computer device adds a pseudo-quantization node to the image processing model. The image processing model first quantizes the floating-point sample image into an integer sample image using the pseudo-quantization node, and then dequantizes the integer sample image to obtain a dequantized floating-point sample image. The image processing model performs inference on the dequantized floating-point sample image to obtain an inference result. Based on the difference between the inference result and the sample result corresponding to the sample image, the floating-point model parameters and the first trainable parameter of the image processing model are adjusted.

[0116] It should be noted that the image processing model in step 405 and step 403 above are both image processing models configured with pseudo-quantization nodes, and the configuration method of the pseudo-quantization nodes is the same. In some embodiments, multiple processing layers are included, and each processing layer includes at least one convolutional layer and at least one activation layer; a pseudo-quantization node is added before each processing layer in the image processing model. The data to be processed in the image processing model is first quantized and dequantized by the pseudo-quantization node, and then input into the corresponding processing layer for processing.

[0117] In some embodiments, the computer device trains an image processing model with dequantized floating-point model parameters based on sample images. This can be achieved by training the image processing model with dequantized floating-point model parameters based on multiple sample images until the accuracy of the image processing model reaches the target accuracy. Figure 6 As shown. When training an image processing model with dequantized floating-point model parameters based on the first sample image, the model is inferred from the first sample image using the image processing model with dequantized floating-point model parameters. The inference result is obtained, and based on this inference result and the corresponding sample result for the first sample image, a loss value is determined. Based on this loss value, the floating-point model parameters and the first trainable parameters of the image processing model are adjusted. Based on the adjusted first trainable parameters, the first quantization step size is updated. Based on the updated first quantization step size and the adjusted floating-point model parameters, the dequantized parameters in the image processing model are updated. The floating-point model parameters are determined; the updated image processing model is called to infer the second sample image, and the inference result is obtained. Based on the inference result and the sample result corresponding to the second sample image, the loss value is determined. Based on the loss value, the adjusted first trainable parameters and the adjusted floating-point model parameters are further adjusted; based on the current adjusted first trainable parameters, the first quantization step size is updated; based on the updated first quantization step size and the adjusted floating-point model parameters, the dequantized floating-point model parameters in the image processing model are updated, and the image processing model is called again to infer the third sample image, and so on.

[0118] In some embodiments, different pseudo-quantization nodes use different first quantization step sizes, and each first quantization step size corresponds to a first trainable parameter. Therefore, during the training process, the computer device adjusts the first trainable parameter corresponding to each first quantization step size.

[0119] In some embodiments, when adjusting the floating-point model parameters and the first trainable parameters of the image processing model during model training, the computer device can select floating-point model parameters and the first trainable parameters that cause the loss value to converge by reducing the gradients, based on the gradients of the determined floating-point model parameters and the first trainable parameters.

[0120] The gradient of the first trainable parameter is expressed as: Where δ represents the partial derivative, L represents the loss value of the image processing model, S represents the first quantization step size, and S represents the first trainable parameter.

[0121] In most cases, the first quantization step size is a non-negative decimal much smaller than 1. Therefore, the first trainable parameter is larger than the first quantization step size. Also, in most cases... Less than 1 indicates The gradient of is less than the gradient of S, that is,

[0122] like Figure 7 As shown, since S can only be updated within the non-negative interval, and Updates can be performed within both positive and negative number ranges; therefore... It has a larger range of values ​​compared to S. Because therefore, Compared to S, it can use a smaller step size. The adjustments effectively improved the stability of model training, enabling the identification of a more accurate quantization step size and enhancing its accuracy.

[0123] 406. The computer device performs inverse mapping on the adjusted first trainable parameters according to the target mapping relationship to obtain the second quantization step size, which is a positive number.

[0124] In some embodiments, the target mapping relationship represents that the trainable parameter is the square root of the quantization step size, and the quantization step size is the square of the trainable parameter. Therefore, the computer device performs inverse mapping on the adjusted first trainable parameter according to the target mapping relationship to obtain the second quantization step size, including: determining the square of the adjusted first trainable parameter as the second quantization step size.

[0125] It should be noted that in the embodiments of this application, the adjusted first trainable parameter is the value obtained after the model training is completed, and the second quantization step size can be regarded as the value obtained after the first quantization step size training is completed.

[0126] In some embodiments, different pseudo-quantization nodes employ different first quantization step sizes, each first quantization step size corresponding to a first trainable parameter. During training, the computer device adjusts the first trainable parameter corresponding to each first quantization step size. The computer device performs a reverse mapping on the adjusted first trainable parameter according to the target mapping relationship to obtain a second quantization step size, including: the computer device performs a reverse mapping on each adjusted first trainable parameter according to the target mapping relationship to obtain a second quantization step size corresponding to each pseudo-quantization node. The second quantization step size corresponding to a pseudo-quantization node can also be considered as the quantization step size corresponding to the processing layer connected to that pseudo-quantization node.

[0127] 407. The computer equipment quantizes the adjusted floating-point model parameters based on the second quantization step size to obtain the second integer model parameters.

[0128] Step 407 is similar to the steps in steps 301 and 403 above, which involve "quantizing the floating-point model parameters based on the first quantization step size to obtain the first integer model parameters", and will not be described in detail here.

[0129] In some embodiments, as described in step 406 above, different processing layers of the image processing model correspond to different second quantization step sizes. The computer device quantizes the adjusted floating-point model parameters based on these second quantization step sizes to obtain second integer model parameters. This includes: the computer device quantizing the floating-point model parameters of the multiple processing layers based on their respective second quantization step sizes to obtain integer model parameters for the multiple processing layers.

[0130] 408. The computer device configures the second integer model parameters for the image processing model and configures the second quantization step size into each layer of the image processing model to obtain the compressed image processing model. The compressed image processing model is then implanted into the terminal. The second quantization step size in the first layer of the compressed image processing model is used to quantize the input floating-point image data into integer image data. The second quantization step size in the Nth layer of the compressed image processing model is used to quantize the output data of the previous layer to reduce the amount of output data. N is a positive integer greater than 1.

[0131] In this context, configuring the second integer model parameter for the image processing model means using the second integer model parameter as the model parameter for the image processing model.

[0132] In some embodiments, the image processing model generates data with a large bit width during processing, leading to increased computational complexity. Optionally, the second quantization step size is used not only to quantize the input floating-point image data into integer image data, but also to quantize the second-bit-width data into the first-bit-width data, wherein the second bit width is greater than the first bit width. For example, the second bit width is 32, and the first bit width is 8. Optionally, the second quantization step size is used not only to quantize the floating-point data into the first-bit-width integer data, but also to quantize the second-bit-width integer data into the first-bit-width integer data.

[0133] For example, the second quantization step is used to quantize data of type float32 into data of type int8, and also to quantize data of type int32 into data of type int8.

[0134] In some embodiments, the image processing model includes an activation layer. The second quantization step size is used not only to quantize the input data of the image processing model but also to quantize the activation value of the activation layer. Since the activation value of the activation layer is determined based on the output of the previous processing layer, and in integer operations, the determined activation value is an integer parameter of type int32, the computer device also configures a second quantization step size in the activation layer. This allows the computer device to map the int32 activation value to int8 based on the second quantization step size after determining the int32 activation value. As described in step 407 above, different processing layers of the image processing model correspond to different second quantization step sizes, and the second quantization step size configured by the computer device in the activation layer is the second quantization step size corresponding to that activation layer.

[0135] It should be noted that the image processing model in step 408 is different from the image processing models in steps 403 and 405. The image processing models in steps 403 and 405 are models with pseudo-quantization nodes added. These pseudo-quantization nodes are used to assist the model in training. Therefore, after the model training is completed, the pseudo-quantization nodes are no longer needed. Thus, the image processing model in step 408 is an image processing model without pseudo-quantization nodes.

[0136] The computer device configures the second integer model parameters for the image processing model and then configures the second quantization step size into the image processing model, resulting in a compressed image processing model. This compressed image processing model can be implanted into a terminal. It should be noted that this application does not limit the specific method of "implanting the compressed image processing model into the terminal." For example, a model file can be generated based on the compressed image processing model, and the model file can be stored in the terminal so that the terminal can run the image processing model by reading the model file.

[0137] The image processing model compression method for terminal deployment provided in this application does not directly train the quantization step size during model quantization. Instead, it sets a target mapping relationship to map the quantization step size into trainable parameters, which are then used for training instead of the quantization step size. Since the trained parameters, after adjustment, are mapped to a positive number based on the target mapping relationship, the quantization step size is guaranteed to be positive, thus enabling successful compression of the image processing model. Deploying the compressed image processing model to the terminal reduces the likelihood of lag or crashes. Because the quantization step size determined in this application is more accurate, the quantization error of the compressed image processing model is smaller, reducing the quantization error introduced by model quantization. This allows the terminal to obtain accurate inference results, improving the accuracy of the compressed image processing model.

[0138] Furthermore, in this embodiment, trainable parameters are used instead of the quantization step size for model training. Since the trainable parameter is less than 1 and greater than the quantization step size, and in most cases the gradient of the trainable parameter is less than the gradient of the quantization step size, the first trainable parameter can be adjusted with a smaller step size relative to the quantization step size to find more accurate trainable parameters, thereby determining a more accurate quantization step size. Moreover, the quantization step size can only be updated within the non-negative range, while the trainable parameter can be updated within both positive and negative ranges. Therefore, the trainable parameter has a wider value range, enabling the acquisition of more accurate trainable parameters and thus a more accurate quantization step size, improving the accuracy of the quantization step size.

[0139] If the image processing model compression method for terminal deployment provided in this application embodiment is applied to the quantization deployment of the YOLOv6-S visual detection model, the results are shown in Table 1. The mAP (mean Average Precision) of the YOLOv6-S visual detection model is 42.4%. If PTQ (Post-training quantization) technology is used for model quantization, the model accuracy loss is relatively serious, and QAT (quantization aware training) technology is needed to improve the accuracy of the quantized model. Using the traditional LSQ (Learned Step Quantization) technology and the image processing model compression method for terminal deployment (Stable LSQ) provided in this application embodiment for model quantization, it can be seen that the image processing model compression method for terminal deployment provided in this application embodiment can achieve higher accuracy with fewer training iterations (Epochs). We also conducted a 4-bit quantization comparison experiment on the ResNet series models (as shown in Table 2), and the conclusions are the same as above, proving the universality and efficiency of the method provided in this application embodiment.

[0140] Table 1

[0141]

[0142]

[0143] Table 2

[0144]

[0145] Figure 8 This is a schematic diagram of the structure of an image processing model compression device for terminal deployment provided in an embodiment of this application, as shown below. Figure 8 As shown, the device includes:

[0146] The pseudo-quantization module 801 is used to quantize the floating-point model parameters of the image processing model to be compressed based on a first quantization step size to obtain a first integer model parameter, dequantize the first integer model parameter based on the first quantization step size to obtain the dequantized floating-point model parameter, and configure the dequantized floating-point model parameter into the image processing model.

[0147] The mapping module 802 is used to map the first quantization step size according to the target mapping relationship to obtain the first trainable parameter. The first trainable parameter is used to replace the first quantization step size in model training. The target mapping relationship is used to represent the mapping relationship between the quantization step size and the trainable parameter.

[0148] The training module 803 is used to train an image processing model with the dequantized floating-point model parameters based on sample images. During the training process, the floating-point model parameters and the first trainable parameters of the image processing model are adjusted to reduce the quantization error introduced into the image processing model.

[0149] The mapping module 802 is further configured to perform inverse mapping on the adjusted first trainable parameters according to the target mapping relationship to obtain a second quantization step size, wherein the second quantization step size is a positive number;

[0150] The pseudo-quantization module 801 is also used to quantize the adjusted floating-point model parameters based on the second quantization step size to obtain the second integer model parameters;

[0151] The configuration module 804 is used to configure the second integer model parameters for the image processing model and configure the second quantization step size to each layer of the image processing model to obtain a compressed image processing model. The compressed image processing model is then implanted into the terminal. The second quantization step size in the first layer of the compressed image processing model is used to quantize the input floating-point image data into integer image data. The second quantization step size in the Nth layer of the compressed image processing model is used to quantize the output data of the previous layer to reduce the amount of output data. N is a positive integer greater than 1.

[0152] like Figure 9 As shown, in one possible implementation, the target mapping relationship represents that the trainable parameter is the square root of the quantization step size; the mapping module 802 is used to take the square root of the first quantization step size to obtain the first trainable parameter.

[0153] The mapping module 802 is used to determine the square of the adjusted first trainable parameter as the second quantization step size.

[0154] In one possible implementation, the device further includes:

[0155] The first determining module 805 is used to determine the parameter range of the image processing model based on the floating-point model parameters;

[0156] The second determining module 806 is used to determine the quantization value range based on the bit width of the integer parameter, wherein the quantization value range represents the value range of the quantized floating-point model parameter.

[0157] The third determining module 807 is used to determine the first quantization step size based on the parameter range and the quantization value range.

[0158] In one possible implementation, the third determining module 907 is used to determine the ratio of the first difference to the second difference as the first quantization step size, wherein the first difference is the difference between the maximum and minimum values ​​in the parameter range, and the second difference is the difference between the maximum and minimum values ​​in the quantization value range.

[0159] In one possible implementation, there are multiple floating-point model parameters; the first determining module 905 is used to determine an average floating-point tensor based on the floating-point tensors of the multiple floating-point model parameters; the maximum negative value represented by the average floating-point tensor is determined as the minimum value of the parameter range, and the maximum positive value represented by the average floating-point tensor is determined as the maximum value of the parameter range, thereby obtaining the parameter range.

[0160] In one possible implementation, the second determining module 906 is used to determine, based on the bit width, the maximum positive value and the maximum negative value that the integer parameter can represent; determine the maximum positive value as the maximum value of the quantization value range, and determine the maximum negative value as the minimum value of the quantization value range, thereby obtaining the quantization value range.

[0161] It should be noted that the image processing model compression device for terminal deployment provided in the above embodiments is only illustrated by the division of the above functional modules when compressing the image processing model. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device can be divided into different functional modules to complete all or part of the functions described above. In addition, the image processing model compression device for terminal deployment provided in the above embodiments and the image processing model compression method for terminal deployment belong to the same concept, and the specific implementation process can be found in the method embodiment, which will not be repeated here.

[0162] Figure 10 This is a structural block diagram of a terminal 1000 provided in an embodiment of this application. The terminal 1000 includes a processor 1001 and a memory 1002.

[0163] Processor 1001 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1001 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1001 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1001 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 1001 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0164] The memory 1002 may include one or more computer-readable storage media, which may be non-transitory. The memory 1002 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1002 are used to store at least one program code, which is executed by the processor 1001 to implement the image processing model compression method for terminal deployment provided in the method embodiments of this application.

[0165] In some embodiments, the terminal 1000 may also optionally include a peripheral device interface 1003 and at least one peripheral device. The processor 1001, memory 1002, and peripheral device interface 1003 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 1003 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 1004, a display screen 1005, a camera 1006, an audio circuit 1007, a positioning component 1008, and a power supply 1009.

[0166] Peripheral device interface 1003 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 1001 and memory 1002. In some embodiments, processor 1001, memory 1002 and peripheral device interface 1003 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 1001, memory 1002 and peripheral device interface 1003 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0167] The radio frequency (RF) circuit 1004 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 1004 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 1004 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 1004 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 1004 can communicate with other terminals via at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: metropolitan area networks (MANs), various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks (WLANs), and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 1004 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0168] Display screen 1005 is used to display a UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When display screen 1005 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 1001 for processing. In this case, display screen 1005 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 1005, serving as the front panel of terminal 1000; in other embodiments, there may be at least two display screens, respectively disposed on different surfaces of terminal 1000 or in a folded design; in still other embodiments, display screen 1005 may be a flexible display screen, disposed on a curved or folded surface of terminal 1000. Furthermore, display screen 1005 may also be configured as a non-rectangular, irregular shape, i.e., a non-rectangular screen. The display screen 1005 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0169] The camera assembly 1006 is used to acquire images or videos. Optionally, the camera assembly 1006 includes a front-facing camera and a rear-facing camera. The front-facing camera is disposed on the front panel of the terminal, and the rear-facing camera is disposed on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusing the main camera and the depth-sensing camera, panoramic shooting by fusing the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 1006 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cool light flash, which can be used for light compensation at different color temperatures.

[0170] The audio circuit 1007 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to the processor 1001 for processing, or input to the radio frequency circuit 1004 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each positioned at a different location on the terminal 1000. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert electrical signals from the processor 1001 or the radio frequency circuit 1004 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 1007 may also include a headphone jack.

[0171] The positioning component 1008 is used to determine the current geographical location of the terminal 1000 in order to enable navigation or LBS (Location Based Service). The positioning component 1008 can be a positioning component based on the US GPS (Global Positioning System), China's BeiDou system, Russia's Granas system, or the EU's Galileo system.

[0172] The power supply 1009 is used to power the various components in the terminal 1000. The power supply 1009 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When the power supply 1009 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0173] In some embodiments, the terminal 1000 further includes one or more sensors 1010. The one or more sensors 1010 include, but are not limited to: an accelerometer 1011, a gyroscope 1012, a pressure sensor 1013, a fingerprint sensor 1014, an optical sensor 1015, and a proximity sensor 1016.

[0174] Accelerometer 1011 can detect the magnitude of acceleration along the three coordinate axes of a coordinate system established by terminal 1000. For example, accelerometer 1011 can be used to detect the components of gravitational acceleration along the three coordinate axes. Processor 1001 can control display screen 1005 to display the user interface in either a landscape or portrait view based on the gravitational acceleration signal acquired by accelerometer 1011. Accelerometer 1011 can also be used for games or for acquiring user motion data.

[0175] The gyroscope sensor 1012 can detect the orientation and rotation angle of the terminal 1000. The gyroscope sensor 1012, in conjunction with the accelerometer sensor 1011, can collect 3D motion data from the user on the terminal 1000. Based on the data collected by the gyroscope sensor 1012, the processor 1001 can perform the following functions: motion sensing (e.g., changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.

[0176] The pressure sensor 1013 can be disposed on the side bezel of the terminal 1000 and / or on the lower layer of the display screen 1005. When the pressure sensor 1013 is disposed on the side bezel of the terminal 1000, it can detect the user's grip signal on the terminal 1000, and the processor 1001 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 1013. When the pressure sensor 1013 is disposed on the lower layer of the display screen 1005, the processor 1001 can control the operable controls on the UI interface based on the user's pressure operation on the display screen 1005. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0177] The fingerprint sensor 1014 is used to collect a user's fingerprint. The processor 1001 identifies the user based on the fingerprint collected by the fingerprint sensor 1014, or vice versa. When the user's identity is identified as trusted, the processor 1001 authorizes the user to perform relevant sensitive operations, including unlocking the screen, viewing encrypted information, downloading software, making payments, and changing settings. The fingerprint sensor 1014 can be located on the front, back, or side of the terminal 1000. When the terminal 1000 has physical buttons or a manufacturer's logo, the fingerprint sensor 1014 can be integrated with the physical buttons or manufacturer's logo.

[0178] An optical sensor 1015 is used to collect ambient light intensity. In one embodiment, the processor 1001 can control the display brightness of the display screen 1005 based on the ambient light intensity collected by the optical sensor 1015. Specifically, when the ambient light intensity is high, the display brightness of the display screen 1005 is increased; when the ambient light intensity is low, the display brightness of the display screen 1005 is decreased. In another embodiment, the processor 1001 can also dynamically adjust the shooting parameters of the camera assembly 1006 based on the ambient light intensity collected by the optical sensor 1015.

[0179] The proximity sensor 1016, also known as a distance sensor, is installed on the front panel of the terminal 1000. The proximity sensor 1016 is used to detect the distance between the user and the front of the terminal 1000. In one embodiment, when the proximity sensor 1016 detects that the distance between the user and the front of the terminal 1000 is gradually decreasing, the processor 1001 controls the display screen 1005 to switch from a screen-on state to a screen-off state; when the proximity sensor 1016 detects that the distance between the user and the front of the terminal 1000 is gradually increasing, the processor 1001 controls the display screen 1005 to switch from a screen-off state to a screen-on state.

[0180] Those skilled in the art will understand that Figure 10 The structure shown does not constitute a limitation on terminal 1000 and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0181] Figure 11 This is a schematic diagram of a server structure provided in an embodiment of this application. The server 1100 can vary considerably due to different configurations or performance. It may include one or more Central Processing Units (CPUs) 1101 and one or more memories 1102. The memory 1102 stores at least one line of program code, which is loaded and executed by the processor 1101 to implement the methods provided in the above-described method embodiments. Of course, the server may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server may also include other components for implementing device functions, which will not be elaborated here.

[0182] The server 1100 is used to execute the steps performed by the server in the above method embodiments.

[0183] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including program code that can be executed by a processor in a computer device to perform the image processing model compression method for terminal deployment in the above embodiments. For example, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0184] In an exemplary embodiment, a computer program or computer program product is also provided, which includes computer program code that, when executed by a computer, causes the computer to implement the image processing model compression method for terminal deployment described in the above embodiments.

[0185] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0186] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for compressing image processing models for terminal deployment, characterized in that, The method includes: Based on the first quantization step size, the floating-point model parameters of the image processing model to be compressed are quantized to obtain the first integer model parameters. Based on the first quantization step size, the first integer model parameters are dequantized to obtain the dequantized floating-point model parameters. The dequantized floating-point model parameters are then configured into the image processing model. According to the target mapping relationship, the first quantization step size is mapped to obtain the first trainable parameter. The first trainable parameter is used to replace the first quantization step size in model training. The target mapping relationship is used to represent the mapping relationship between the quantization step size and the trainable parameter. Based on the sample images, the image processing model configured with the dequantized floating-point model parameters is trained. During the training process, the floating-point model parameters and the first trainable parameters of the image processing model are adjusted to reduce the quantization error introduced into the image processing model. Based on the target mapping relationship, the adjusted first trainable parameters are inversely mapped to obtain the second quantization step size, which is a positive number. Based on the second quantization step size, the adjusted floating-point model parameters are quantized to obtain the second integer model parameters; Configure the second integer model parameters for the image processing model, and configure the second quantization step size into each layer of the image processing model to obtain a compressed image processing model. Implant the compressed image processing model into a terminal. The second quantization step size in the first layer of the compressed image processing model is used to quantize the input floating-point image data into integer image data. The second quantization step size in the Nth layer of the compressed image processing model is used to quantize the output data of the previous layer to reduce the amount of output data, where N is a positive integer greater than 1. The target mapping relationship indicates that the trainable parameters are the square root of the quantization step size. The mapping of the first quantization step size according to the target mapping relationship to obtain the first trainable parameters includes: The first trainable parameter is obtained by taking the square root of the first quantization step size. The step of inversely mapping the adjusted first trainable parameters according to the target mapping relationship to obtain the second quantization step size includes: The square of the adjusted first trainable parameter is determined as the second quantization step size.

2. The method according to claim 1, characterized in that, The method further includes: Based on the floating-point model parameters, the parameter range of the image processing model is determined; Based on the bit width of the integer parameter, the quantization value range is determined, which represents the value range of the quantized floating-point model parameter. The first quantization step size is determined based on the parameter range and the quantization value range.

3. The method according to claim 2, characterized in that, Determining the first quantization step size based on the parameter range and the quantization value range includes: The ratio of the first difference to the second difference is determined as the first quantization step size. The first difference is the difference between the maximum and minimum values ​​in the parameter range, and the second difference is the difference between the maximum and minimum values ​​in the quantization value range.

4. The method according to claim 2, characterized in that, The floating-point model parameters are multiple; determining the parameter range of the image processing model based on the floating-point model parameters includes: Determine the average floating-point tensor based on floating-point tensors with multiple floating-point model parameters; The maximum negative value represented by the average floating-point tensor is determined as the minimum value of the parameter range, and the maximum positive value represented by the average floating-point tensor is determined as the maximum value of the parameter range, thus obtaining the parameter range.

5. The method according to claim 2, characterized in that, The determination of the quantization value range based on the bit width of the integer parameter includes: Based on the bit width, determine the maximum positive and maximum negative values ​​that the integer parameter can represent; The maximum positive value is determined as the maximum value of the quantization value range, and the maximum negative value is determined as the minimum value of the quantization value range, thus obtaining the quantization value range.

6. An image processing model compression device for terminal deployment, characterized in that, The device includes: The pseudo-quantization module is used to quantize the floating-point model parameters of the image processing model to be compressed based on a first quantization step size to obtain a first integer model parameter, dequantize the first integer model parameter based on the first quantization step size to obtain the dequantized floating-point model parameter, and configure the dequantized floating-point model parameter into the image processing model. The mapping module is used to map the first quantization step size according to the target mapping relationship to obtain the first trainable parameter. The first trainable parameter is used to replace the first quantization step size in the model training. The target mapping relationship is used to represent the mapping relationship between the quantization step size and the trainable parameter. The training module is used to train an image processing model with the dequantized floating-point model parameters based on sample images. During the training process, the floating-point model parameters and the first trainable parameters of the image processing model are adjusted to reduce the quantization error introduced into the image processing model. The mapping module is further configured to perform inverse mapping on the adjusted first trainable parameters according to the target mapping relationship to obtain a second quantization step size, wherein the second quantization step size is a positive number; The pseudo-quantization module is also used to quantize the adjusted floating-point model parameters based on the second quantization step size to obtain the second integer model parameters; A configuration module is used to configure the second integer model parameters for the image processing model and configure the second quantization step size into each layer of the image processing model to obtain a compressed image processing model. The compressed image processing model is then implanted into a terminal. The second quantization step size in the first layer of the compressed image processing model is used to quantize the input floating-point image data into integer image data. The second quantization step size in the Nth layer of the compressed image processing model is used to quantize the output data of the previous layer to reduce the amount of output data. N is a positive integer greater than 1. The target mapping relationship indicates that the trainable parameter is the square root of the quantization step size; the mapping module is used to take the square root of the first quantization step size to obtain the first trainable parameter. The mapping module is used to determine the square of the adjusted first trainable parameter as the second quantization step size.

7. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one line of program code, which is loaded and executed by the processor to perform the operations performed in the image processing model compression method for terminal deployment as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The storage medium stores at least one piece of program code, which is loaded and executed by a processor to perform the operations performed in the image processing model compression method for terminal deployment as described in any one of claims 1 to 5.

Images