Image processing method and related device

Abstract

The embodiment of the invention provides an image processing method and a related device, and relates to the technical field of terminals. The method comprises the steps that a mode of pre-storing a model and multiple quantization parameters is adopted, when an image is processed, the model can be called in each round, the quantization parameters corresponding to the round are called based on the preset multiple quantization parameters, quantization of input and output of the model is dynamically achieved, in this way, a set of model can be supported to be pre-stored, memory occupation can be reduced, and the quantization efficiency is improved. Moreover, the input and output of the model can be quantized based on the pre-stored quantization parameters, the quantization parameters do not need to be calculated in real time, the operation amount of model processing is reduced, the timeliness of model quantization is improved, and since the preset adaptive quantization parameters can be adopted in each round of calling, the accuracy of the model in each round of processing is improved, and the accuracy of model quantization is improved. Therefore, the accuracy, the memory occupation and the processing time delay can be well considered.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to image processing methods and related devices. Background Technology

[0002] To improve image quality, electronic devices can employ image processing models to process images and optimize them. These models can be pre-trained neural network models designed to optimize image quality. Image processing models can include floating-point models or quantized models, where quantization refers to mapping the model parameters from floating-point numbers to fixed-point numbers.

[0003] Floating-point image processing models typically offer high accuracy but require significant computing power, making them difficult to deploy on edge devices. Even when deployed on edge, they often suffer from high memory consumption and long processing times. Quantized image processing models, while requiring less memory than floating-point models and being easier to deploy on edge devices, suffer from lower accuracy. This makes it difficult for edge electronic devices to balance accuracy, memory usage, and processing latency when using image processing models for image processing. Summary of the Invention

[0004] The image processing method and related apparatus provided in this application embodiment can better balance accuracy, memory usage and processing latency.

[0005] In a first aspect, the image processing method provided in the embodiments of this application includes:

[0006] Obtain the image to be processed; perform N rounds of processing on the image to be processed according to the preset model and the preset M quantization parameters; where M and N are both integers greater than 1, and M is less than or equal to N; the M quantization parameters include quantization parameters with different values; the i-th round of processing includes: calling the model, and using the quantization parameter corresponding to i in the M quantization parameters in the model to process the input of the i-th round of processing; the i-th round of processing is any round of processing in the N-th round of processing.

[0007] In this way, by pre-storing the model and M quantization parameters, the model can be called in each of the N rounds of image processing. Based on the preset multiple quantization parameters, the corresponding quantization parameters for that round are called, and the quantization of the model's input and output is dynamically realized. This supports pre-storing a set of models, which helps reduce memory usage. Moreover, the input and output of the model can be quantized based on the pre-stored quantization parameters, without the need for real-time calculation of quantization parameters, which reduces the computational load of model processing and improves the timeliness of model quantization. Furthermore, since the pre-set adapted quantization parameters can be used in each round of calling, it helps improve the accuracy of the model in each round of processing. Thus, it can better balance accuracy, memory usage, and processing latency.

[0008] In one possible implementation, the M quantization parameters include at least a first quantization parameter and a second quantization parameter. The first quantization parameter is used in round 'a' of the N rounds of processing, and the second quantization parameter is used in round 'b' of the N rounds of processing. 'a' and 'b' are different, and both 'a' and 'b' are integers greater than 1. This allows for multiple iterations corresponding to a single quantization parameter, reducing the memory footprint of the quantization parameters.

[0009] In one possible implementation, the M quantization parameters are set based on the importance of each stage in the N rounds of processing. The N rounds of processing include multiple processing stages, each of which comprises consecutive rounds. The importance of each stage in the N rounds varies, and the target value for a more important stage is greater than that for a less important stage. The target value is the ratio between the number of quantization parameters for any given stage and the number of rounds in that stage. This allows for setting appropriate quantization parameters for each round in important stages to improve accuracy, while setting fewer quantization parameters for less important stages to save memory. Thus, it achieves both accuracy and memory savings in terms of quantization parameters.

[0010] In one possible implementation, any one of the M quantization parameters includes a scaling factor and an offset, where the scaling factor is related to the data range in each processing stage. Because the scaling factor and offset have a significant impact on quantization accuracy, this approach can help improve quantization accuracy, thereby enhancing image processing accuracy.

[0011] In one possible implementation, the preset M quantization parameters are stored using an index-value pair. The quantization parameter corresponding to i in the M quantization parameters is obtained as follows: based on the index of i, the quantization parameter corresponding to i in the M quantization parameters is retrieved from the query interface used to access the M quantization parameters by using the index of i. In this way, the quantization parameters for each round can be retrieved conveniently.

[0012] In one possible implementation, the preset model and the preset M-term quantization parameters are stored together in binary format as Control.bin. This reduces storage space usage.

[0013] In one possible implementation, i is 1, and the input for the i-th round of processing is the encoded features of the image to be processed; or, i is greater than 1 and less than or equal to N, and the input feature map for the i-th round of processing is the feature map output by the (i-1)-th round of processing. This enables serial image processing, eliminating the need to set up multiple models in electronic devices and saving memory.

[0014] In one possible implementation, before acquiring the image to be processed, the method further includes: displaying a preview interface, the preview interface including a first button and a second button, the first button being selected; acquiring multiple image frames in response to a trigger operation on the second button; acquiring the image to be processed, including: processing the multiple image frames to obtain the image to be processed; and performing N rounds of processing on the image to be processed according to a preset model and preset M quantization parameters, including: when the first button is determined to be selected, performing N rounds of processing on the image to be processed according to the preset model and preset M quantization parameters. In this way, when taking a photo, the user can conveniently select whether to use the image processing method of this application embodiment based on the interface.

[0015] In one possible implementation, before the first round of processing, the image to be processed is encoded using an encoding model. After the Nth round of processing, the result of the Nth round of processing is decoded. This can yield an image of better quality.

[0016] Secondly, embodiments of this application provide an image processing apparatus, which may be an electronic device, a chip or chip system within an electronic device. The apparatus may include a processing unit and a display unit. The processing unit is used to implement any processing-related method executed by the electronic device in the first aspect or any possible implementation of the first aspect. When the apparatus is an electronic device, the processing unit may be a processor. The apparatus may further include a storage unit, which may be a memory. The storage unit is used to store instructions, and the processing unit executes the instructions stored in the storage unit to cause the electronic device to implement the method described in the first aspect or any possible implementation of the first aspect. When the apparatus is a chip or chip system within an electronic device, the processing unit may be a processor. The processing unit executes the instructions stored in the storage unit to cause the electronic device to implement the method described in the first aspect or any possible implementation of the first aspect. The storage unit may be a storage unit within the chip (e.g., a register, cache, etc.), or a storage unit located outside the chip within the electronic device (e.g., a read-only memory, random access memory, etc.).

[0017] For example, the processing unit is used to acquire the image to be processed; and to perform N rounds of processing on the image to be processed according to a preset model and preset M quantization parameters; wherein M and N are both integers greater than 1, and M is less than or equal to N; the M quantization parameters include quantization parameters with different values; the i-th round of processing includes: calling the model, and using the quantization parameter corresponding to i in the M quantization parameters in the model to process the input of the i-th round of processing; the i-th round of processing is any round of processing in the N-th round of processing.

[0018] In one possible implementation, the M quantization parameters include at least a first quantization parameter and a second quantization parameter. The first quantization parameter is used for round a of the N rounds of processing, and the second quantization parameter is used for round b of the N rounds of processing. a and b are different, and both a and b are integers greater than 1.

[0019] In one possible implementation, the M quantization parameters are set based on the importance of each stage in the N rounds of processing. The N rounds of processing include multiple processing stages, and each processing stage includes multiple consecutive rounds of processing. Each processing stage has different importance in the N rounds of processing, and the target value corresponding to the processing stage with higher importance is greater than the target value corresponding to the processing stage with lower importance. The target value includes the ratio between the number of quantization parameters corresponding to any processing stage and the number of processing rounds included in any processing stage.

[0020] In one possible implementation, any one of the M quantization parameters includes a scaling factor and an offset, both of which are related to the data range in each processing stage.

[0021] In one possible implementation, the preset M quantization parameters are stored using an index-value approach; the quantization parameter corresponding to i in the M quantization parameters is obtained as follows: based on the index of i, the quantization parameter corresponding to i in the M quantization parameters is retrieved from the query interface used to access the M quantization parameters by querying the index of i.

[0022] In one possible implementation, the preset model and the preset M-item quantization parameters are stored together in binary format Control.bin.

[0023] In one possible implementation, i is 1, and the input for the i-th round of processing is the encoded features of the image to be processed; or, i is greater than 1 and less than or equal to N, and the input feature map for the i-th round of processing is the feature map output by the (i-1)-th round of processing.

[0024] In one possible implementation, a display unit is used to display a preview interface, which includes a first button and a second button, wherein the first button is in a selected state; a processing unit is also used to acquire multiple image frames in response to a trigger operation on the second button; process the multiple image frames to obtain an image to be processed; and, when the first button is determined to be in a selected state, perform N rounds of processing on the image to be processed according to a preset model and preset M quantization parameters.

[0025] In one possible implementation, before the first round of processing of the image to be processed, an encoding model is used to encode the image to be processed, and after the Nth round of processing is completed, the result of the Nth round of processing is decoded.

[0026] Thirdly, embodiments of this application provide a terminal device, including a processor and a memory, wherein the memory is used to store code instructions and the processor is used to run the code instructions to perform the method described in the first aspect or any possible implementation of the first aspect.

[0027] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program or instructions that, when executed on a computer, cause the computer to perform the methods described in the first aspect or any possible implementation thereof.

[0028] Fifthly, embodiments of this application provide a computer program product including a computer program, which, when run on a computer, causes the computer to perform the methods described in the first aspect or any possible implementation thereof.

[0029] Sixthly, this application provides a chip or chip system including at least one processor and a communication interface. The communication interface and the at least one processor are interconnected via a circuit. The at least one processor is used to run computer programs or instructions to perform the methods described in the first aspect or any possible implementation of the first aspect. The communication interface in the chip can be an input / output interface, pins, or circuits, etc.

[0030] In one possible implementation, the chip or chip system described above in this application further includes at least one memory storing instructions. The memory can be an internal storage unit of the chip, such as a register or cache, or it can be a storage unit of the chip itself (e.g., read-only memory, random access memory, etc.).

[0031] It should be understood that the second to sixth aspects of this application correspond to the technical solutions of the first aspect of this application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation are similar, and will not be repeated here. Attached Figure Description

[0032] Figure 1 A schematic diagram of a model architecture provided for an embodiment of this application;

[0033] Figure 2 A schematic diagram of a data range provided in an embodiment of this application;

[0034] Figure 3 A schematic diagram of grouping provided for an embodiment of this application;

[0035] Figure 4 This is a schematic diagram of the storage method provided in the embodiments of this application;

[0036] Figure 5 This is a schematic diagram illustrating the process of obtaining adaptive quantization parameters in a diffusion model provided in an embodiment of this application;

[0037] Figure 6 A schematic diagram illustrating the reasoning of a diffusion model provided in an embodiment of this application;

[0038] Figure 7 A schematic diagram of an interface for taking photos, provided as an embodiment of this application;

[0039] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0040] Figure 9 A schematic diagram of the software structure of an electronic device provided in an embodiment of this application;

[0041] Figure 10 A schematic diagram illustrating an image processing method provided in an embodiment of this application;

[0042] Figure 11 This is a schematic diagram of the structure of a chip provided in an embodiment of this application. Detailed Implementation

[0043] To facilitate a clear description of the technical solutions in the embodiments of this application, some terms and technologies involved in the embodiments of this application will be briefly introduced below:

[0044] 1. Quantization parameters

[0045] Quantization parameters are parameters used during model quantization. Model quantization can include quantizing the model's weights and / or quantizing the model's inputs and outputs. Model quantization can include symmetric quantization and asymmetric quantization. Symmetric and asymmetric quantization differ in quantization methods, quantization parameters, application scenarios, and potential accuracy losses. This application's embodiments can use either symmetric or asymmetric quantization parameters.

[0046] For example, quantization parameters may include one or more of the following: quantization bit width, quantization range, scaling factor, offset, and quantization strategy.

[0047] Quantization bit width refers to the number of bits occupied by the quantized value; for example, it can be 8 bits, 4 bits, etc. A lower bit width can reduce the storage and computational load of the model, but may result in some loss of accuracy. The choice of quantization bit width needs to be weighed based on the specific application scenario and accuracy requirements.

[0048] The quantization range defines the range of values ​​for the input data; that is, between which values ​​of the unquantized data will be mapped to the finite range of values ​​after quantization. The quantization range can be indicated using a minimum-maximum (min-max) value. The choice of quantization range affects the distribution and accuracy of the quantized data. A reasonable quantization range allows the quantized data to better retain the characteristics of the original data while minimizing accuracy loss.

[0049] The scaling factor is used to map the original value to the quantized numerical range. The scaling factor is typically calculated based on the quantization range and data distribution, and it determines the scaling ratio of the values ​​during quantization. A suitable scaling factor can make the quantized data closer to the original data, thereby reducing precision loss.

[0050] Offset is used to adjust the mapping position of the zeros before and after quantization. In symmetric quantization, the zero z can be set to 0, ensuring consistency between the zeros before and after quantization, without the need for an additional offset. In asymmetric quantization, an offset can be introduced to adjust the mapping position of the zeros before and after quantization.

[0051] Quantization strategy refers to the specific methods and strategies used in the quantization process, such as channel-wise quantization (independent quantization of each channel of the convolution kernel) and tensor-wise quantization (using the same quantization parameters for the entire tensor). Different quantization strategies have different impacts on the accuracy and performance of the model. Channel-wise quantization can preserve more model details, but the computational cost is relatively high; tensor-wise quantization has a lower computational cost, but may sacrifice some accuracy. The choice of a suitable quantization strategy depends on the specific application scenario and model architecture.

[0052] 2. Terminology

[0053] In the embodiments of this application, terms such as "first" and "second" are used to distinguish identical or similar items with substantially the same function and purpose. For example, "first chip" and "second chip" are used only to distinguish different chips and do not limit their order of execution. Those skilled in the art will understand that terms such as "first" and "second" do not limit the quantity or execution order, and that "first" and "second" do not necessarily imply that they are different.

[0054] It should be noted that, in the embodiments of this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0055] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0056] 3. Electronic equipment

[0057] The electronic device in this application embodiment can also be any form of terminal device. For example, the electronic device may include: mobile phone, tablet computer, handheld computer, laptop computer, mobile internet device (MID), virtual reality (VR) device, augmented reality (AR) device, wireless terminal in industrial control, wireless terminal in self-driving, wireless terminal in remote medical surgery, wireless terminal in smart grid, wireless terminal in transportation safety, wireless terminal in smart city, wireless terminal in smart home, cellular phone, cordless phone, session initiation protocol (SIP) phone, wireless local loop (WLL) station, personal digital assistant (PDA), handheld device with wireless communication function, computing device or other processing device connected to a wireless modem, vehicle-mounted device, electronic device in 5G network, or future evolved public land mobile communication network. The embodiments of this application do not limit the scope of electronic devices in a network (PLMN).

[0058] By way of example and not limitation, in this embodiment, the electronic device can also be a wearable device. Wearable devices, also known as wearable smart devices, are a general term for devices that utilize wearable technology to intelligently design and develop everyday wearables, such as glasses, gloves, watches, clothing, and shoes. Wearable devices are portable devices that are worn directly on the body or integrated into the user's clothing or accessories. Wearable devices are not merely hardware devices, but also achieve powerful functions through software support, data interaction, and cloud interaction. Broadly speaking, wearable smart devices include those that are feature-rich, large in size, and can achieve complete or partial functions without relying on a smartphone, such as smartwatches or smart glasses, as well as those that focus on a specific type of application function and require the use of other devices such as smartphones, such as various smart bracelets and smart jewelry for vital sign monitoring.

[0059] Furthermore, in this application embodiment, the electronic device can also be an electronic device in the Internet of Things (IoT) system. IoT is an important part of the future development of information technology. Its main technical feature is to connect objects to the network through communication technology, thereby realizing an intelligent network of human-machine interconnection and object-to-object interconnection.

[0060] The electronic equipment in the embodiments of this application may also be referred to as: user equipment (UE), mobile station (MS), mobile terminal (MT), access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent, or user device, etc.

[0061] In this embodiment, the electronic device or various network devices include a hardware layer, an operating system layer running on top of the hardware layer, and an application layer running on top of the operating system layer. The hardware layer includes hardware such as a central processing unit (CPU), a memory management unit (MMU), and memory (also called main memory). The operating system can be any one or more computer operating systems that implement business processing through processes, such as Linux, Unix, Android, iOS, or Windows. The application layer includes applications such as browsers, address books, word processing software, instant messaging software, cameras, and / or image processing.

[0062] Some electronic devices can support a camera function, which can also provide various shooting modes, such as one or more of the following: large aperture mode, night mode, portrait mode, or professional mode.

[0063] In the shooting modes described above, there may be issues such as the subject being out of focus or lacking sharpness. Therefore, image processing models such as diffusion models can be used to process the captured images, transforming low-resolution images into high-resolution images. For example, low-resolution images can be processed into high-resolution images.

[0064] For example, Figure 1 A schematic diagram of a network architecture employing a diffusion model using a control network is shown. Figure 1 As shown, the diffusion model can include an encoder, an n-step interaction network, and a decoder.

[0065] The encoding module is used to encode the input image to obtain image features.

[0066] A network with n iterations can include n control networks and n UNets. Control networks can introduce additional conditions and precisely control the generation process to make the generated images more aligned with user needs. These additional conditions may include prompts, such as those indicating the image style. UNets handle tasks such as feature extraction, denoising, and image restoration, providing the foundation for generating high-quality images.

[0067] The decoding module is used to decode the output of the last step of n-step iterations to obtain the processed image.

[0068] When the diffusion model processes an image, the encoding module encodes the input image to obtain image features (also called feature maps). These image features are then input into an n-step iteration network, where they undergo n rounds of processing. Each round is processed by the interaction between the control module and the UNet. The input to the first round is the image features encoded by the encoder. In each subsequent round, the input can include the output of the previous round; for example, the input feature map for each round might be the output feature map of the previous round. After n rounds of processing, the output of the nth round is fed into the decoding model for decoding, resulting in the processed image.

[0069] In the relevant implementation, it can be like Figure 1The diffusion model shown is set up in an electronic device, and this diffusion model can be floating-point type. This requires deploying an encoder, decoder, and n floating-point type controllers and UNets on the edge side.

[0070] However, floating-point diffusion models have a large number of parameters and computational complexity, which not only consumes a lot of memory in electronic devices but also results in long processing times when using them for image processing. This is because diffusion models achieve denoising through multi-step inference. For example, with n=50, if the inference model uses 50 steps for denoising, then... Figure 1 The control and UNet modules will run 50 times in a loop, which will result in a long processing time.

[0071] Therefore, in one possible implementation, from the perspective of saving storage space (ROM), a single quantization model can be used for all 50 loops. This quantization model can be the model obtained by quantizing the control and UNet respectively. For example, the control and UNet can be quantized separately to obtain two quantized models, which can be understood as the aforementioned quantization model. Alternatively, the control and UNet can be merged and quantized into a single quantization model, which can also be understood as the aforementioned quantization model. This allows for the deployment of an encoder, decoder, and a single quantization model on the device side, with the quantization model called once per loop, thus avoiding the need for 50 quantization models and reducing memory usage. However, because the input data range varies significantly across different loops, using the same quantization model for all 50 loops will result in lower accuracy. For example, as shown... Figure 2 As shown, the horizontal axis represents the number of iterations, and the vertical axis represents the data values. The thicker curve represents the maximum value (max), and the thinner curve represents the minimum value (min). Taking one iteration as an example... Figure 2 As shown, the input distribution in step 0 of ControlNet is in [-4, 4], while the input distribution in step 49 is in [-2, 2]. In extreme cases, if the min-max of the quantization parameters is set to -4, 4, then one bit of quantization precision will be lost in step 49, reducing the accuracy of the diffusion model and resulting in poor performance at the edge. If the min-max of the quantization parameters is set to -2, 2, then truncating the data to [-2, 2] in step 1 will introduce a large truncation error, which will also reduce the accuracy of the diffusion model and result in poor performance at the edge.

[0072] In another implementation, considering processing efficiency, 50 quantization models are used for 50 iterations. This allows for the deployment of an encoder, decoder, and 50 quantization models on the edge, with each iteration using its own quantization model. This reduces the impact of different data distribution ranges during iterations with varying number of iterations, improving the accuracy of model processing. However, due to the large number of parameters in the diffusion model, this can cause significant memory pressure.

[0073] In another implementation, balancing memory usage and accuracy, an encoder, decoder, and a set of floating-point models are deployed on the device side. These floating-point models could include a control module and a UNet module. The floating-point model is called once during each loop, and quantization parameters are calculated in real-time based on the input of each loop. This allows for real-time quantization of the floating-point model, eliminating the need for 50 quantization models and reducing memory consumption. However, the computational load of real-time quantization parameter calculation is significant, resulting in high model processing latency. For example, when processing images using a diffusion model of the size shown in Table 1, the latency on the device side reaches 95 seconds or even longer, leading to a poor user experience.

[0074] Table 1. Parameters and computational complexity of each module in the diffusion model.

[0075] Module Parameters computational load control 363M 139G unet 866M 414G decoder 66.3M 1278.6G encoder 34.2M 570.1G

[0076] In view of this, the image processing method and related apparatus provided in this application adopt a method of pre-storing models and multiple quantization parameters. During image processing, the model can be called in each round, and the corresponding quantization parameters for that round can be called based on the preset multiple quantization parameters to dynamically realize the quantization of the model input and output. This supports the pre-storage of one set of models, which helps to reduce memory occupation. Moreover, the input and output of the model can be quantized based on the pre-stored quantization parameters, without the need to calculate the quantization parameters in real time, which reduces the amount of computation in model processing and improves the timeliness of model quantization. Furthermore, since the pre-set adapted quantization parameters can be used in each round of calling, it helps to improve the accuracy of the model in each round of processing, thereby achieving a better balance between accuracy, memory occupation, and processing latency.

[0077] The pre-stored model can be any model that needs to be repeatedly run during inference, not limited to a portion of the diffusion model. The number of models that need to be repeatedly run can be one or more, for example, it can include the control and UNet components of the diffusion model. The pre-stored model can be trained based on samples and a neural network model; this application does not limit the specific content of the pre-stored model. Pre-storing the model can be understood as pre-storing the model architecture and its weights in an electronic device.

[0078] In this embodiment of the application, the pre-stored model can be a floating-point model, a quantized model, or any model with high precision that can be further quantized for input and output.

[0079] For example, taking a pre-stored floating-point model as an example, one possible implementation can store the model architecture and floating-point weights of the floating-point model. In subsequent loop processing, the floating-point model can be called, and floating-point weights can be used for model inference. Another possible implementation can store the floating-point model and its corresponding weight quantization parameters. In the image processing method of this application or during the initialization of the electronic device, the floating-point model can be converted to a fixed-point model based on the weight quantization parameters. In subsequent loop processing, when calling the model, the fixed-point model can be called, and the quantized weights can be used for model inference.

[0080] Taking a pre-stored quantized model as an example, the quantized model can include models obtained by quantizing one or more models individually, or it can include models obtained by merging and quantizing multiple models. Taking a diffusion model as an example, the quantized model can be a model obtained by quantizing the control and UNet. For example, quantizing the control and UNet separately yields two quantized models, which can be understood as the pre-stored model mentioned above. Alternatively, the control and UNet can be merged and quantized into a single quantized model, which can also be understood as the pre-stored quantized model mentioned above. In subsequent loop processing, when calling the model, a fixed-point model can be called, using the quantized weights to perform model inference, thus reducing memory usage.

[0081] In this embodiment of the application, multiple quantization parameters can be parameters used to quantize the input and output of the model.

[0082] In one possible implementation, the number of terms for multiple quantization parameters can be the same as the number of loops. The number of loops can be understood as the number of times the model is run repeatedly. This allows each loop to correspond to its own quantization parameters, which is beneficial for adapting appropriate quantization parameters to each loop, thereby improving the accuracy of model inference.

[0083] In another possible implementation, the number of terms for multiple quantization parameters can be less than the number of loops. This allows for multiple loops to correspond to a single quantization parameter, reducing the memory space occupied by the quantization parameters.

[0084] In implementations where the number of multiple quantization parameters can be less than the number of loops, the loop processing can be divided into several stages, each stage including multiple loop processes, and each stage setting one quantization parameter.

[0085] Alternatively, based on the varying importance of each processing round, the multi-round processing can be divided into multiple groups, each corresponding to a quantization parameter. Importance can be understood as the degree of influence on the final output; the greater the influence on the final output, the higher the importance. For example, taking a diffusion model, earlier rounds of processing are beneficial for quickly finding the denoising direction during inference, while later rounds are beneficial for refining the image content during inference. The data range of adjacent steps in the middle rounds is relatively similar. Therefore, the earlier and later rounds are more important, while the middle rounds are less important. Thus, the earlier and later rounds can be subdivided into smaller groups, while the middle rounds can be roughly grouped, with each group sharing a set of quantization parameters. This approach saves memory occupied by quantization parameters while maintaining accuracy. Figure 3 As shown, the first part, steps 0-4, can be divided into three groups: step 0 in one group, steps 1 and 2 in another, and steps 3 and 4 in yet another. The middle part, steps 5-44, can be grouped into sets of 5. Finally, steps 45-49 can be divided into two groups: steps 45 and 49 in one group, and steps 46, 47, and 48 in another.

[0086] Of course, other image processing models may have situations where the front and back parts are of low importance, while the middle part is of high importance. In such cases, the front and back parts can be roughly grouped, and the middle part can be subdivided. This application does not limit the grouping method in its embodiments.

[0087] In this embodiment of the application, one possible implementation of pre-storing the model and multiple quantization parameters is to store the model and multiple quantization parameters in the form of a binary format control.bin, which can reduce the storage space required. Figure 4 The diagram illustrates a control.bin file. A set of floating-point models and weight quantization parameters can be understood as a trained model and its corresponding weight quantization parameters. These weight quantization parameters can be a set of invariant parameters. Multiple quantization parameters (such as...) Figure 4 The quantization parameters shown (e.g., quantization parameter 0, quantization parameter 2, quantization parameter i, quantization parameter n, etc.) can be the quantization parameters corresponding to each step's input and output. The quantization parameters for each step can be different, and multiple quantization parameters can be stored in some addresses, each with an index. When it is necessary to query the quantization parameters for each step, the query interface in control.bin can be used, based on the index of that step (e.g., ...). Figure 4 The index format corresponding to the step shown can be (step value: index value), to query the corresponding quantization parameter.

[0088] It's understandable that different step counts can correspond to the same index or different indices, depending on the grouping of steps. Furthermore, when pre-storing the model and multiple quantization parameters, other methods can be used, such as storing the quantization parameters separately for each step. Alternatively, when storing quantization parameters for steps grouped together, after storing the quantization parameters for one step in that group, an identifier can be added to the other steps in that group, indicating that the other steps in the group share the same quantization parameter.

[0089] The specific values ​​of the quantization parameters for each round of iteration in this application embodiment can be obtained through a limited number of experiments based on the scenario in which the model is to be used. This application embodiment does not impose any limitations. In a possible implementation, the input of the 0th step of the n-step floating-point inference process can be used as the correction dataset for the 0th step quantization, and the quantization parameters for the 0th step can be generated based on this data range. Then, the floating-point model of the 0th step of the inference process is replaced with the quantization model, and the input of the 1st step generated is used as the correction dataset for the 1st step quantization, and the quantization parameters for the 1st step can be generated based on this data range. Similarly, in the i-th step, the floating-point model of the first i-1 steps of the inference process is replaced with the quantization model, and the input of the i-th step generated is used as the correction dataset for the i-th step quantization, and the quantization parameters for the i-th step can be generated based on this data range. Through this serial cumulative quantization error correction, the global quantization accuracy of the n-th step can be relatively high.

[0090] For example, Figure 5 This diagram illustrates the process of obtaining suitable quantization parameters in a diffusion model according to an embodiment of this application. The input to step 0 is the image features encoded by the encoder. In any step after step 0, the input to step i includes the output of step i-1. For step i, the input of floating-point inference step i is selected as the calibration dataset (or activation). The activation data range is the ideal range for step i; therefore, the appropriate quantization parameters for step i can be calculated based on the activation data range. In this implementation, because the quantization parameters for step i are related to the output of step i-1, they are not affected by inconsistent distributions in other steps, for example... Figure 2 The range of the calibration dataset in step 25 is {[-3.8,-2],[2,3]}. This range is not affected by the data range between steps 2 and 23, nor by the data range between steps 26 and 45. It is mainly related to the data range of step 25. Therefore, the min and max values ​​of the symmetric quantization parameters for step 25 can be set to -3.8 and 3.8, respectively. This ensures that each step can obtain quantization parameters that are suitable for that step. When generating the calibration dataset for step i, the control parameters of the first i-1 steps can all use the quantization parameters corresponding to the step number to obtain a suitable quantization model, thus achieving better image processing results.

[0091] In summary, in this embodiment, an encoder, decoder, a set of reusable models, and multiple quantization parameters can be deployed on the edge electronic device. The model is called once in each loop, and the input and output of the model called in that loop are quantized based on the pre-deployed quantization parameters corresponding to each loop. The quantized model can then be used for image processing in that loop. Optionally, the encoder and decoder in this embodiment can also be quantized models, which further saves memory usage.

[0092] Taking the pre-stored model as an example, specifically the control and UNet components in the diffusion model, we can illustrate this. Figure 6 This diagram illustrates the inference process of the diffusion model according to an embodiment of this application. The input for step 0 is the image features encoded by the encoder. In any step after step 0, the input for step i includes the output of step i-1. When implementing the image processing method of this application embodiment or during the initialization of an electronic device, the floating-point model can be converted into a fixed-point model based on the weight quantization parameters of the model weights. In n loops of n steps, the fixed-point model can be called, and the encodings step mapping can be switched through the query interface of control.bin. After obtaining the input and output quantization parameters of the corresponding step i, the input of step i is converted into fixed-point data, and model inference is performed to obtain the output of step i. The output of step i can be dequantized based on the input and output quantization parameters to obtain floating-point data, which is used as part of the input of the (i+1)th step. Then, the fixed-point model is called again, and the encodings step mapping can be switched through the query interface of control.bin. The input and output quantization parameters of the corresponding step i+1 are obtained, and the input of step i+1 is converted into fixed-point data. Model inference is performed to obtain the output of step i+1, and the output of step i+1 can be dequantized based on the input and output quantization parameters to obtain floating-point data, which is used as part of the input of the (i+2)th step. This process is repeated n times. The output of the nth step can then be processed by the decoder into a high-quality image.

[0093] The following is combined with Figure 7 This application describes one application scenario of an embodiment. Figure 7 This diagram illustrates an interface for taking a photo. (For example...) Figure 7 In the interface of the electronic device shown in (a), multiple application icons can be displayed on the desktop, such as camera icon 300, address book icon, phone icon, message icon, clock icon, calendar icon, gallery icon, memo icon, file manager icon, email icon, music icon, and / or calculator icon.

[0094] After receiving a trigger operation on the camera icon 300, you can enter the preview interface. The preview interface offers a variety of shooting modes, such as professional mode, aperture mode, night mode, portrait mode, photo mode, and / or video mode. Users can slide the control where the shooting mode is located to select the desired shooting mode.

[0095] For example, such as Figure 7 As shown in (b), the telephoto mode 301 can be slid to the selected position. After receiving a trigger on the shutter button 304, the electronic device can capture an image and process the image using the image processing algorithm of this application embodiment to obtain an image of better quality.

[0096] Or, such as Figure 7 As shown in (c), the portrait mode 303 can be slid to the selected position, and then... Figure 7 The interface shown in (c) also includes a button 302 for enabling or disabling the image processing algorithm of this embodiment. When button 302 is used to enable the image processing algorithm, after receiving a trigger on the camera button 304, the electronic device can capture an image and process it using the image processing algorithm of this embodiment. When button 302 is used to disable the image processing algorithm, after receiving a trigger on the camera button 304, the electronic device can process the image using the usual portrait mode method, without using the image processing algorithm of this embodiment.

[0097] Of course, in any shooting mode, this application embodiment may also include a button (not shown in the figure) for turning the image processing algorithm of this application embodiment on or off. The image processing method of this application embodiment can also be applied to any scenario that requires image processing, and this application embodiment is not limited thereto.

[0098] For example, Figure 8 A schematic diagram of the electronic device is shown.

[0099] The electronic device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0100] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device. In other embodiments of this application, the electronic device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may include hardware, software, or a combination of software and hardware.

[0101] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0102] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0103] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the aforementioned memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0104] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a SIM card interface, and / or a USB interface, etc.

[0105] It is understood that the interface connection relationships between the modules illustrated in the embodiments of the present invention are merely illustrative and do not constitute a limitation on the structure of the electronic device. In other embodiments of this application, the electronic device may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0106] Internal memory 121 can be used to store executable program code, including instructions. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of the electronic device (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. Processor 110 executes various functional applications and data processing of the electronic device by running instructions stored in internal memory 121 and / or instructions stored in memory located within the processor.

[0107] Camera 193 is used to capture still images or videos. In some embodiments, the electronic device may include one or N cameras 193, where N is a positive integer greater than 1.

[0108] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. In some embodiments, the electronic device may include one or N displays screens 194, where N is a positive integer greater than 1. The electronic device implements display functions through a GPU, display screen 194, and application processor, etc. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor.

[0109] For example, in this embodiment of the application, the internal memory 121 may store a program for executing the image processing method of this embodiment of the application, and the display screen 194 may display, as shown in the example. Figure 7 The interface shown includes shooting modes. Upon receiving a signal, such as... Figure 7 After the user operation described, the processor 110 can implement the image processing in the embodiment of this application and improve the image quality based on the program for executing the image processing method of this application stored in the internal memory 121.

[0110] Figure 9 This is a software structure block diagram of an electronic device according to an embodiment of this application. A layered architecture can divide the software system of an electronic device into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system can be divided into five layers, from top to bottom: applications, application framework, hardware abstraction layer (HAL), kernel, and hardware layer.

[0111] The application layer can include a series of application packages. The application layer runs applications by calling the application programming interface (API) provided by the application framework layer. For example... Figure 9 As shown, the application package can include applications such as camera and gallery.

[0112] The application framework layer provides APIs and a programming framework for applications within the application layer. The application framework layer includes predefined functions. For example... Figure 9 As shown, the application framework layer may include a camera access interface and a view system. The camera access interface provides an application programming interface and programming framework for camera applications. The view system includes visual controls, such as controls for taking photos, as described above. Figure 7The various controls shown in the interface.

[0113] like Figure 9 As shown, the HAL layer may include a camera hardware abstraction layer and a camera algorithm library. The camera hardware abstraction layer can provide virtual hardware for the camera device. The camera algorithm library may include runtime code and data that implement the image processing methods provided in the embodiments of this application.

[0114] The kernel layer is the layer between hardware and software. For example... Figure 9 As shown, this kernel layer may include one or more of the following: camera device driver, digital signal processor driver, and image processor driver. The camera device driver is used to drive the camera's sensor to acquire images. The digital signal processor driver is used to drive the digital signal processor to process images. The image processor driver is used to drive the graphics processor to process images.

[0115] The hardware layer can include hardware such as cameras and displays.

[0116] It should be understood that in some embodiments, layers that perform the same function may be called by other names, or layers that can perform the functions of multiple layers may be considered as one layer, or layers that can perform the functions of multiple layers may be divided into multiple layers. This application does not impose any limitations on this.

[0117] The following is in conjunction with the above. Figure 9 The software structure shown below provides a detailed description of the image processing method in this application embodiment:

[0118] In response to a user's action of opening the camera application, such as clicking the camera application icon, the camera application calls the camera access interface of the application framework layer to launch the camera application, and then sends a command to start the camera by calling the camera hardware abstraction layer. The camera hardware abstraction layer sends this command to the camera device driver in the kernel layer. The camera device driver can then start the corresponding camera to capture images. After the captured image is processed by image stabilization, fusion, etc., a color image can be obtained. This color image is then processed by the image processing method provided in the embodiments of this application, which is set in the algorithm library, to obtain a high-quality image. The camera algorithm library can transmit the processed image to the camera hardware abstraction layer. The camera hardware abstraction layer can then transmit the processed image to the display screen for display.

[0119] The image processing method of this application will be described in detail below through specific embodiments. The following embodiments can be implemented in combination with each other or independently, and the same or similar concepts or processes may not be described again in some embodiments.

[0120] Figure 10 An image processing method according to an embodiment of this application is illustrated. The method includes:

[0121] S101. Obtain the image to be processed.

[0122] In this embodiment of the application, the image to be processed can be any color image. The method of acquiring the image to be processed can include, for example... Figure 7 The images described in the embodiments are obtained during the photo-taking process. Images selected from a gallery may also be included, but this application does not limit the scope of the embodiments.

[0123] S102. Based on the preset model and the preset M quantization parameters, perform N rounds of processing on the image to be processed; where M and N are both integers greater than 1, and M is less than or equal to N; the M quantization parameters include quantization parameters with different values; the i-th round of processing includes: calling the model and using the quantization parameter corresponding to i in the M quantization parameters in the model to process the input of the i-th round of processing; the i-th round of processing is any round of processing in the N-th round of processing.

[0124] In this embodiment, the preset model can be a pre-stored floating-point model or a quantized model, or it can be a fixed-point model obtained by weight quantization of a pre-stored model. For details, please refer to the description of the pre-stored model in the foregoing embodiments. The preset M quantization parameters can refer to the description of the multiple quantization parameters in the foregoing embodiments. The N rounds of processing can refer to the description of the n rounds of processing in the foregoing embodiments, and will not be repeated here.

[0125] In this embodiment, a pre-stored model and M quantization parameters are used. During image processing, the model can be called in each of the N rounds of processing, and the corresponding quantization parameters for that round are called based on the preset multiple quantization parameters. This dynamically realizes the quantization of the model's input and output. This supports the pre-stored model, which helps reduce memory usage. Furthermore, the model's input and output can be quantized based on the pre-stored quantization parameters, eliminating the need for real-time calculation of quantization parameters, reducing the computational load of model processing, and improving the timeliness of model quantization. Since the pre-set adaptive quantization parameters can be used in each round of calling, it helps improve the accuracy of the model in each round of processing. Thus, it can better balance accuracy, memory usage, and processing latency.

[0126] In one possible implementation, the M quantization parameters include at least a first quantization parameter and a second quantization parameter. The first quantization parameter is used for round a of the N rounds of processing, and the second quantization parameter is used for round b of the N rounds of processing. a and b are different, and both a and b are integers greater than 1.

[0127] In this embodiment, the M quantization parameters may include quantization parameters for one round of processing or quantization parameters for multiple rounds of processing. The number of rounds of processing can also be different. For specific correspondence, please refer to the descriptions related to the number of quantization parameter items in the above embodiments, which will not be repeated here. This allows for multiple loops corresponding to a single quantization parameter, reducing the memory space occupied by the quantization parameters.

[0128] In one possible implementation, the M quantization parameters are set based on the importance of each stage in the N rounds of processing. The N rounds of processing include multiple processing stages, and each processing stage includes multiple consecutive rounds of processing. Each processing stage has different importance in the N rounds of processing, and the target value corresponding to the processing stage with higher importance is greater than the target value corresponding to the processing stage with lower importance. The target value includes the ratio between the number of quantization parameters corresponding to any processing stage and the number of processing rounds included in any processing stage.

[0129] In this embodiment of the application, the stage may correspond to... Figure 3 The corresponding embodiments describe the front, middle, and rear parts. Of course, in specific application scenarios, the stages can be further subdivided or broadly divided according to the actual situation. The target value is defined to illustrate the degree of refinement of the processing grouping for each stage; in actual implementation, it may not be necessary to calculate the target value. The adaptation of quantification parameters based on the importance of each stage can be referred to... Figure 3 The description of the corresponding embodiments will not be repeated. In this way, the quantization parameters of each round in the important stages can be set as close as possible to the appropriate round to improve accuracy, while fewer quantization parameters can be set for the less important stages to save memory. Thus, while maintaining accuracy, the memory occupied by quantization parameters can be saved.

[0130] In one possible implementation, any one of the M quantization parameters includes a scaling factor and an offset, both of which are related to the data range in each processing stage. Of course, any quantization parameter can also include any parameter used for quantization. Because the scaling factor and offset have a significant impact on quantization accuracy, this can help improve quantization accuracy, thereby improving image processing accuracy.

[0131] In one possible implementation, the preset M quantization parameters are stored using an index-value pair. The quantization parameter corresponding to i in the M quantization parameters is obtained as follows: based on the index of i, the quantization parameter corresponding to i in the M quantization parameters is retrieved from the query interface used to access the M quantization parameters by using the index of i. For a detailed implementation, please refer to [link to implementation details]. Figure 4 The description of the corresponding embodiments will not be repeated. In this way, the quantization parameters for each round can be easily retrieved.

[0132] In one possible implementation, the preset model and the preset M-term quantization parameters are stored together in binary format as Control.bin. For a detailed implementation, please refer to [link / reference needed]. Figure 4 The description of the corresponding embodiments will not be repeated. In this way, less storage space is required.

[0133] In one possible implementation, i is 1, and the input for the i-th round of processing is the encoded features of the image to be processed; or, i is greater than 1 and less than or equal to N, and the input feature map for the i-th round of processing is the feature map output by the (i-1)-th round of processing. This enables serial image processing, eliminating the need to set up multiple models in electronic devices and saving memory.

[0134] In one possible implementation, before acquiring the image to be processed, the method further includes: displaying a preview interface, the preview interface including a first button and a second button, the first button being selected; acquiring multiple image frames in response to a trigger operation on the second button; acquiring the image to be processed, including: processing the multiple image frames to obtain the image to be processed; and performing N rounds of processing on the image to be processed according to a preset model and preset M quantization parameters, including: when the first button is determined to be selected, performing N rounds of processing on the image to be processed according to the preset model and preset M quantization parameters.

[0135] In this embodiment of the application, the preview interface can be referred to as follows: Figure 7 In the interfaces shown in (b) and (c), the first button can correspond to button 302, and the second button can correspond to the camera button 304. Specific details are omitted. In this way, when taking a photo, users can conveniently select whether to use the image processing method of this application embodiment based on the interface.

[0136] In one possible implementation, before the first round of processing of the image to be processed, an encoding model is used to encode the image to be processed, and after the Nth round of processing is completed, the result of the Nth round of processing is decoded.

[0137] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0138] The foregoing primarily describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the aforementioned functions, it includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, based on the method steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0139] This application embodiment can divide the apparatus for implementing the method into functional modules based on the above method examples. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0140] like Figure 11 The diagram shows a chip structure according to an embodiment of this application. Chip 110 includes one or more processors 111, communication lines 112, communication interfaces 113, and memory 114.

[0141] In some implementations, memory 114 stores elements such as executable modules or data structures, or subsets thereof, or extended sets thereof.

[0142] The methods described in the embodiments of this application can be applied to or implemented by processor 111. Processor 111 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuit in the hardware of processor 111 or by instructions in software form. The processor 111 may be a general-purpose processor (e.g., a microprocessor or conventional processor), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates, transistor logic devices, or discrete hardware components. Processor 111 can implement or execute the various processing-related methods, steps, and logic block diagrams disclosed in the embodiments of this application.

[0143] The steps of the method disclosed in the embodiments of this application can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can be located in mature storage media in the art, such as random access memory, read-only memory, programmable read-only memory, or electrically erasable programmable read-only memory (EEPROM). This storage medium is located in memory 114, and the processor 111 reads the information in memory 114 and, in conjunction with its hardware, completes the steps of the above method.

[0144] The processor 111, memory 114 and communication interface 113 can communicate with each other via communication line 112.

[0145] In the above embodiments, the instructions stored in the memory for execution by the processor can be implemented in the form of a computer program product. This computer program product can be pre-written into the memory, or it can be downloaded and installed into the memory as software.

[0146] This application also provides a computer program product comprising one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions may be transmitted from a website site, computer, server, or data center to another website site, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. For example, available media may include magnetic media (e.g., floppy disk, hard disk, or magnetic tape), optical media (e.g., digital versatile disc (DVD)), or semiconductor media (e.g., solid-state disk (SSD)).

[0147] This application also provides a computer-readable storage medium. The methods described in the above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any combination thereof. The computer-readable medium may include computer storage media and communication media, and may also include any medium capable of transferring a computer program from one place to another. The storage medium can be any target medium accessible by a computer.

[0148] As one possible design, computer-readable media may include compact disc read-only memory (CD-ROM), RAM, ROM, EEPROM, or other optical disc storage; computer-readable media may also include disk storage or other disk storage devices. Furthermore, any connecting cable may also be appropriately referred to as computer-readable media. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of media. As used herein, disks and optical discs include optical discs (CD), laser discs, optical discs, digital versatile discs (DVD), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs optically reproduce data using lasers.

[0149] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

Claims

1. An image processing method, characterized in that, The method includes: Obtain the image to be processed; The image to be processed is subjected to N rounds of processing based on a preset model and M preset quantization parameters. Wherein, M and N are both integers greater than 1, and M is less than or equal to N; the M quantization parameters include quantization parameters with different values; the i-th round of processing includes: calling the model, and using the quantization parameter corresponding to i in the M quantization parameters in the model to process the input of the i-th round of processing; the i-th round of processing is any round of processing in the N-th round of processing.

2. The method according to claim 1, characterized in that, The M quantization parameters include at least a first quantization parameter and a second quantization parameter. The first quantization parameter is used for round a of the N rounds of processing, and the second quantization parameter is used for round b of the N rounds of processing. The first and second quantization parameters are different, and both the first and second parameters are integers greater than 1.

3. The method according to claim 1 or 2, characterized in that, The M quantization parameters are set based on the importance of each stage in the N rounds of processing. The N rounds of processing include multiple processing stages, and each processing stage includes multiple consecutive rounds of processing. Each processing stage has a different importance in the N rounds of processing, and the target value corresponding to the processing stage with higher importance is greater than the target value corresponding to the processing stage with lower importance. The target value includes the ratio between the number of quantization parameters corresponding to any processing stage and the number of processing rounds included in any processing stage.

4. The method according to any one of claims 3, characterized in that, The quantization parameters in the M terms include a scaling factor and an offset, both of which are related to the data range in each of the processing stages.

5. The method according to any one of claims 1-4, characterized in that, The preset M quantization parameters are stored using an index-value method; the quantization parameter corresponding to i in the M quantization parameters is obtained in the following way: Based on the index of i, the quantization parameter corresponding to i in the M quantization parameters is obtained from the query interface used to access the M quantization parameters by querying the index of i.

6. The method according to any one of claims 1-5, characterized in that, The preset model and the preset M-item quantization parameters are stored together in binary format Control.bin.

7. The method according to any one of claims 1-6, characterized in that, The value of i is 1, and the input to the i-th round of processing is the encoded features of the image to be processed; Alternatively, i is greater than 1 and less than or equal to N, and the input feature map of the i-th round of processing is the feature map output by the (i-1)-th round of processing.

8. The method according to any one of claims 1-7, characterized in that, The process of acquiring the image to be processed includes: Display a preview interface, which includes a first button and a second button, wherein the first button is in a selected state; In response to the triggering operation of the second button, multiple image frames are acquired; The step of obtaining the image to be processed includes: processing the multiple image frames to obtain the image to be processed; The step of processing the image to be processed in N rounds according to a preset model and preset M quantization parameters includes: when the first button is determined to be selected, processing the image to be processed in N rounds according to the preset model and preset M quantization parameters.

9. The method according to any one of claims 1-8, characterized in that, Before the first round of processing of the image to be processed, the image to be processed is encoded using an encoding model. After the Nth round of processing is completed, the result of the Nth round of processing is decoded.

10. An electronic device, characterized in that, include: A memory and a processor, the memory for storing a computer program and the processor for executing the computer program to perform the method as described in any one of claims 1-9.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed, cause a computer to perform the method as described in any one of claims 1-9.

12. A computer program product, characterized in that, Includes a computer program that, when run, causes an electronic device to perform the method as described in any one of claims 1-9.

Images