Image processing method and related device

Abstract

The present application provides an image processing method and related devices, which are applied to electronic devices with cameras and providing shooting and display functions, such as smart phones, tablet computers, large-screen devices, etc. The electronic device divides the image to be processed into multiple regions. Next, the electronic device can send each region of interest to the corresponding deblurring model for deblurring according to the classification label. For example, regions of interest with the same degree of blur are sent to the same deblurring model, and regions of interest with different degrees of blur are sent to different deblurring models, so as to obtain a clear image of the image to be processed. It can be seen that the present application uses multiple "dedicated deblurring models" to deblur regional images with different degrees of blur, which can also avoid pseudo-textures caused by inconsistent image blur levels, improve deblurring consistency, and effectively improve image quality.

Description

Technical Field

[0001] The present application relates to the field of terminals, and in particular to image processing methods and related devices. Background Art

[0002] When using electronic devices such as mobile phones and tablets to shoot moving objects, or when there is relative motion between the subject and the electronic device, the image or video (including multiple image frames) captured by the electronic device will be blurred to a certain extent. In order to improve the imaging quality of the image, a deblurring algorithm is usually used to deblur the blurred image.

[0003] At present, in order to reduce processing delay, electronic devices divide blurred images into multiple image sub-blocks and perform deblurring on the multiple image sub-blocks through parallel computing. However, there may be deblurring differences between different image sub-blocks, which is not conducive to improving image quality. Summary of the invention

[0004] An image processing method and related device provided in an embodiment of the present application can deblur areas of different blur levels in an image according to classification labels used to indicate the blur level, thereby obtaining a clear image and improving image quality.

[0005] In a first aspect, the present application provides an image processing method, which is applied to an electronic device, wherein the electronic device includes a camera, and the method may include: acquiring a first image captured by the camera; dividing the first image into L types of regions of interest, wherein each of the L types of regions of interest corresponds to a classification label, and the classification label is used to indicate the blur degree of the region of interest, and L is a positive integer;

[0006] Based on the classification labels, the L types of regions of interest are respectively input into corresponding deblurring models to obtain clear images corresponding to the L types of regions of interest, wherein the deblurring model is a trained deep learning model.

[0007] In the above embodiment, the present application divides the first image (i.e., the image to be processed) according to the degree of blur, and then sends the divided areas into the corresponding deep learning model according to the degree of blur for parallel deblurring processing, which can reduce the model processing delay. By using multiple "dedicated deblurring models" to deblur regional images with different blur degrees, pseudo textures (artifacts) caused by inconsistent image blur levels can be avoided, and the deblurring consistency can be improved, thereby effectively improving the image quality.

[0008] In a possible implementation of the first aspect, dividing the first image into L types of regions of interest includes:

[0009] Divide the first image into N image sub-blocks, where N is a positive integer;

[0010] Determine a classification label corresponding to each image sub-block in the N image sub-blocks, wherein the classification label is used to indicate a blur degree of the image sub-block;

[0011] The N image sub-blocks are clustered based on the classification labels to divide the first image into L types of regions of interest.

[0012] It can be seen that in order to improve the accuracy of the blur degree of the region of interest, the blur degree of each image sub-block in the first image can be determined first. Since the resolution of the image sub-block is low, the blur degree corresponding to each image sub-block can be obtained quickly. Then, based on the blur degree of the image sub-block, the image sub-blocks are clustered, so that the first image can be divided into multiple regions of interest in an adaptive manner.

[0013] In a possible implementation manner of the first aspect, the classification label is further used to indicate clarity, and the classification label includes one or more of a clear label, a first fuzzy label, and a second fuzzy label;

[0014] The L types of regions of interest include one or more of the region of interest corresponding to the clear label, the region of interest corresponding to the first fuzzy label, and the region of interest corresponding to the second fuzzy label, wherein the degree of blur represented by the first fuzzy label is less than the degree of blur represented by the second fuzzy label.

[0015] It can be seen that in order to perform targeted or refined deblurring processing on the first image, the first image can be divided into multiple regions according to classification labels. This division method can reduce the performance delay of model inference.

[0016] In a possible implementation of the first aspect, inputting the L types of regions of interest into corresponding deblurring models based on the classification labels to obtain clear images corresponding to the L types of regions of interest respectively includes:

[0017] Inputting the region of interest corresponding to the first fuzzy label into a first deblurring model, and performing deblurring processing on the region of interest corresponding to the first fuzzy label based on the first deblurring model to obtain a clear image of the region of interest corresponding to the first fuzzy label; wherein the first deblurring model is trained based on sample data corresponding to the first fuzzy label;

[0018] The region of interest corresponding to the second fuzzy label is input into a second deblurring model, and the region of interest corresponding to the second fuzzy label is deblurred based on the second deblurring model to obtain a clear image of the region of interest corresponding to the second fuzzy label; wherein the second deblurring model is trained based on sample data corresponding to the second fuzzy label.

[0019] It can be seen that there is blur in the area of ​​interest corresponding to the first fuzzy label and the area of ​​interest corresponding to the second fuzzy label. For areas with different degrees of blur, the present application can use different deblurring models to perform deblurring processing in a targeted manner, which can improve the effect of the deblurring processing and improve the image quality.

[0020] In a possible implementation of the first aspect, the region of interest corresponding to the first fuzzy label or the second fuzzy label includes an image sub-block corresponding to the clear label, and after inputting the L types of regions of interest into corresponding deblurring models based on the classification labels, and before obtaining clear images corresponding to the L types of regions of interest, the method further includes:

[0021] Determine whether there is texture in the output image sub-block of the input image sub-block corresponding to the clear label; wherein the input image sub-block is the image sub-block corresponding to the clear label input into the deblurring model, and the output image sub-block is the image sub-block corresponding to the input image sub-block output by the deblurring model;

[0022] In the case where there is texture in the output image sub-block, the output image sub-block is replaced with the input image sub-block, thereby obtaining clear images corresponding to the L types of regions of interest respectively;

[0023] In the case that there is no texture in the output image sub-block, clear images corresponding to the L types of regions of interest are obtained.

[0024] It is understandable that there may be image sub-blocks corresponding to clear labels in the regions of interest corresponding to different classification labels. Since the image sub-blocks corresponding to clear labels are not blurred, in order to avoid the texture caused by the processing of the deblurring model, the output image sub-blocks of the deblurring model can be identified. If there is texture, it needs to be processed to avoid affecting the deblurring quality due to the presence of texture.

[0025] In a possible implementation manner of the first aspect, determining whether there is texture in the output image sub-block of the input image sub-block corresponding to the clear label includes:

[0026] It is determined that there is texture in the output image sub-block by that a mean of pixel difference values ​​between the output image sub-block and the input image sub-block corresponding to the clear label is greater than a first threshold or a variance of the pixel difference values ​​is greater than a second threshold.

[0027] It can be seen that by comparing the difference between the output image sub-block and the input image sub-block, the difference between the output image sub-block and the input image sub-block can be determined. When the difference is large, it can indicate that a large change has occurred in the output image sub-block, and it needs to be processed to ensure the image quality.

[0028] In a possible implementation of the first aspect, dividing the first image into L types of regions of interest based on the classification labels includes:

[0029] The first image is divided into L types of regions of interest according to the same classification label in a nearest neighbor clustering manner, wherein the value of L is the same as the number of types of the classification label.

[0030] It can be seen that the same classification labels can be grouped together by clustering, so that the classification labels of most of the image sub-blocks in the divided region of interest can be guaranteed to be the same, thereby improving the accuracy of the first image division.

[0031] In a possible implementation of the first aspect, in a deblurring model corresponding to the first fuzzy label, a weight corresponding to the first fuzzy label is greater than a weight corresponding to the second fuzzy label;

[0032] In the deblurring model corresponding to the second fuzzy label, the weight corresponding to the second fuzzy label is greater than the weight corresponding to the first fuzzy label.

[0033] It can be seen that in order to improve the processing capability of the deblurring model corresponding to the classification label for the image area corresponding to the classification label, the weight corresponding to the classification label in the model can be increased, and the weight corresponding to the non-classification label can be reduced.

[0034] In a possible implementation manner of the first aspect, there are overlapping areas among the L types of regions of interest, and after obtaining clear images respectively corresponding to the L types of regions of interest, the method further includes:

[0035] The overlapping parts are processed by weighted average fusion to splice the clear images corresponding to the L types of regions of interest, thereby obtaining a clear image of the first image.

[0036] It can be seen that in order to ensure a natural edge transition of the region of interest during processing, the regions of interest can overlap with each other and there are overlapping parts. In this way, each region of interest can be spliced ​​together through weighted average fusion, which can avoid splicing marks and improve image quality.

[0037] In a possible implementation of the first aspect, the method further includes:

[0038] saving a clear image of the first image;

[0039] A clear image of the first image is displayed.

[0040] It can be seen that the user can view the saved clear images, which improves the user experience.

[0041] In a second aspect, an embodiment of the present application provides an electronic device, the electronic device comprising: a camera; one or more processors; a memory; wherein the camera is used to capture a first image; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the image processing method described in the first aspect or any possible implementation of the first aspect.

[0042] In a third aspect, the present application provides a chip or a chip system, the chip or chip system comprising at least one processor and a communication interface, the communication interface and the at least one processor are interconnected by a line, and the at least one processor is used to run a computer program or instruction to execute the image processing method described in the first aspect or any possible implementation of the first aspect. The communication interface in the chip can be an input / output interface, a pin or a circuit, etc.

[0043] In a possible implementation, the chip or chip system described above in the embodiment of the present application further includes at least one memory, in which instructions are stored. The memory may be a storage unit inside the chip, such as a register, a cache, etc., or a storage unit of the chip (e.g., a read-only memory, a random access memory, etc.).

[0044] In a fourth aspect, an embodiment of the present application provides a computer storage medium, which stores a computer program. When the computer program is executed by a processor, the computer executes the image processing method described in the first aspect or any possible implementation of the first aspect.

[0045] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed on a communication device, enables the communication device to execute an image processing method as described in the first aspect or any possible implementation of the first aspect.

[0046] It should be understood that the description of technical features, technical solutions, beneficial effects or similar language in this application does not imply that all features and advantages can be realized in any single embodiment. On the contrary, it is understood that the description of features or beneficial effects means that specific technical features, technical solutions or beneficial effects are included in at least one embodiment. Therefore, the description of technical features, technical solutions or beneficial effects in this specification does not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions and beneficial effects described in the present embodiment can also be combined in any appropriate manner. Those skilled in the art will understand that the embodiment can be realized without one or more specific technical features, technical solutions or beneficial effects of a specific embodiment. In other embodiments, additional technical features and beneficial effects can also be identified in a specific embodiment that does not embody all embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The following is an introduction to the drawings used in the embodiments of the present application.

[0048] Figure 1 is a schematic diagram of a communication system provided in an embodiment of the present application;

[0049] Figure 2 This is an example diagram of a knowledge distillation scenario provided in an embodiment of the present application;

[0050] FIG. 3A to FIG. 3F The electronic device 100 is shown as an example of a UI in which a clear image is displayed while shooting a person's long jump action;

[0051] Figure 4 is a schematic diagram of an image processing scenario provided by an embodiment of the present application;

[0052] Figure 5 It is a flowchart of an image processing method provided in an embodiment of the present application;

[0053] Figure 6 is a schematic diagram of identifying the degree of blur provided in an embodiment of the present application;

[0054] Figure 7 is a schematic diagram of a region of interest provided in an embodiment of the present application;

[0055] Figure 8 is a schematic diagram of replacing a static area provided in an embodiment of the present application;

[0056] Fig. 9is a structural diagram of an electronic device 100 provided in an embodiment of the present application;

[0057] Fig.10 It is a schematic diagram of the software architecture of the electronic device 100 provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification and appended claims of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include plural expressions, unless there is a clear indication to the contrary in the context. It should also be understood that the term "and / or" used in the present application refers to and includes any or all possible combinations of one or more listed items.

[0059] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0060] The term "user interface (UI)" in the following embodiments of the present application refers to a medium interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The user interface is a source code written in a specific computer language such as Java and extensible markup language (XML). The interface source code is parsed and rendered on an electronic device, and finally presented as content that can be recognized by the user. The commonly used form of user interface is a graphical user interface (GUI), which refers to a user interface related to computer operation displayed in a graphical manner. It can be a visual interface element such as text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, etc. displayed on the display screen of an electronic device.

[0061] The electronic device (such as a mobile phone, tablet computer, etc.) in the embodiment of the present application is equipped with a camera, and the electronic device can capture images through the camera. In the process of capturing images, the relative motion between the camera and the photographed object will cause the image output by the camera to be blurred. In order to improve the quality of the output image, the electronic device needs to use a deblurring algorithm to process the image output by the camera, improve the clarity of the image, and enhance the user's shooting experience.

[0062] In one implementation, the electronic device can directly input the blurred image (i.e., blurred image) output by the camera into the deblurring model, and after being processed by the deblurring model, the electronic device can obtain a clear image. Since the blurred image is a full-resolution image, the calculation amount and delay of the model will be increased, which will cause certain program performance pressure on the electronic device.

[0063] In one implementation, in order to reduce latency, the electronic device can divide the blurred image into multiple image sub-blocks, input the multiple image sub-blocks into a deblurring model, and after the deblurring model processes the multiple image sub-blocks in parallel, the electronic device can obtain a clear image. The latency of the model can be reduced by parallel processing, but when the blur degree of the blurred image is inconsistent, the way of dividing the image sub-blocks may lead to deblurring differences between different image sub-blocks, which may eventually lead to the introduction of boundary artifacts (also called pseudo-textures) around each image sub-block, which does not improve the image quality. In addition, it is difficult for a single deep learning model to perform consistent deblurring on image sub-blocks with inconsistent blur degrees.

[0064] From the above analysis, we can see that the images captured in the following scenarios may be blurred, including but not limited to the following three situations:

[0065] Case 1: The object being photographed is moving, such as using a mobile phone to shoot a person's long jump.

[0066] Case 2: Moving the shooting device (such as a mobile phone, tablet or other electronic device) during shooting. For example, changing the shooting angle to shoot a static scene.

[0067] Case 3: The above cases 1 and 2 work together. For example, in the process of a person long jumping, the long jumping person is photographed by constantly changing the shooting angle.

[0068] In the above three situations, when photographing the object, as the object and the photographing device move relative to each other, the object will form a series of continuously changing images on the imaging plane. This series of continuously changing information may cause different degrees of image blur. For example, if the movement amplitude is small, the blur is also small; if the movement amplitude is large, the blur is also large.

[0069] In view of this, the present application provides an image processing method and related apparatus, which is applied to electronic devices with a camera and providing shooting and display functions, such as smart phones, tablet computers, large-screen devices, etc.

[0070] Exemplarily, the electronic device divides the image to be processed into N image sub-blocks, and determines the classification label corresponding to each of the N image sub-blocks, where the classification label is used to indicate the blur degree of the image sub-block. Then, the electronic device can cluster the N image sub-blocks according to the classification label, thereby dividing the image to be processed into multiple regions. Next, the electronic device can send each region of interest to a corresponding deblurring model for deblurring according to the classification label, for example, regions of interest with the same blur degree are sent to the same deblurring model, and regions of interest with different blur degrees are sent to different deblurring models, thereby obtaining a clear image of the image to be processed.

[0071] It can be seen that the present application divides the image to be processed according to the degree of blur, and then sends the divided areas into the corresponding deep learning model according to the degree of blur for parallel deblurring, which can reduce the model processing delay. By using multiple "dedicated deblurring models" to deblur regional images with different blur levels, pseudo-textures (artifacts) caused by inconsistent image blur levels can be avoided, and the deblurring consistency can be improved, thereby effectively improving the image quality.

[0072] First, the system provided by the embodiment of the present application is introduced. Figure 1 is a schematic diagram of a communication system provided in an embodiment of the present application, such as Figure 1 As shown, the communication system includes an electronic device 100 and a first device 200 .

[0073] Among them, the first device 200 is used to collect training samples, and different deep learning models are obtained by training using different training samples. Exemplarily, the training samples include blurred images, blurred images with classification labels, and clear images corresponding to the blurred images. Among them, the classification labels are used to indicate the blurriness of the image. The first device 200 can use blurred images with different blurriness to train a deep learning model for identifying the blurriness (referred to as the recognition model). The first device 200 can use blurred images and clear images with classification labels to train a deep learning model for deblurring (referred to as the deblurring model). The first device 200 is usually a device with strong computing power. Exemplarily, the first device 200 can be a server, a server cluster, a personal computer (PC), etc.

[0074] The first device 200 and the electronic device 100 are connected in communication. Exemplarily, the first device 200 and the electronic device 100 can be connected in communication by wired means such as a universal serial bus (USB). Alternatively, the first device 200 and the electronic device 100 can be connected in communication by wireless means, and the wireless means may include but are not limited to global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), Bluetooth, wireless fidelity (Wi-Fi), NFC, voice over Internet protocol (VoIP), communication protocols supporting network slicing architecture, etc.

[0075] In the embodiment of the present application, through the communication connection between the first device 200 and the electronic device 100, it can be achieved that: the first device 200 transmits the recognition model and the deblurring models corresponding to different blur levels to the electronic device 100.

[0076] In a possible implementation, the electronic device 100 identifies the degree of blur of each image sub-block in the image to be processed based on the recognition model, and redivides the image to be processed into regions according to the degree of blur, and inputs the multiple regions obtained after the division into corresponding deblurring models respectively, and the deblurring model deblurs the images of the corresponding regions, thereby obtaining a clear image.

[0077] Of course, in other embodiments, the model training and recognition process can also be completed independently by one device (i.e., the first device 200 and the electronic device 100 are the same device). For example, a deblurring model is obtained by PC training, and the deblurring model is used to deblur the blurred image. This application does not impose specific restrictions on this.

[0078] In a possible implementation, the first device 200 may utilize a deep neural network (DNN) or a convolutional neural network (CNN) to train a recognition model and a deblurring model.

[0079] Some terms related to neural networks are explained below to facilitate understanding by those skilled in the art.

[0080] (1) Deep neural network DNN is also called multi-layer neural network, which can be understood as a neural network with multiple hidden layers. According to the position of different layers, deep neural network DNN can be divided into three categories: input layer, hidden layer and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. There can be fully connected layers between layers, that is, any neuron in the i-th layer can be connected to any neuron in the i+1-th layer.

[0081] (2) Convolutional neural network (CNN) is a deep neural network with a convolutional structure. Convolutional neural network (CNN) contains a feature extractor consisting of a convolutional layer and a subsampling layer, which can be regarded as a filter. The convolutional layer refers to the neuron layer in the convolutional neural network (CNN) that performs convolution processing on the input signal. In the convolutional layer of the convolutional neural network (CNN), a neuron can only be connected to some neurons in the adjacent layers. A convolutional layer usually contains several feature planes, and each feature plane can be composed of some neural units arranged in a rectangular shape.

[0082] (3) The convolution kernel can be initialized in the form of a matrix of random size, and the convolution kernel can obtain reasonable weights through learning during the training process of the convolutional neural network. In addition, the direct benefit of shared weights is to reduce the connections between the layers of the convolutional neural network, while reducing the risk of overfitting.

[0083] (4) A filter is a series connection of multiple convolution kernels, each of which is assigned to a specific channel of the input. When the number of channels is 1, the filter is the convolution kernel. When the number of channels is greater than 1, the filter refers to the series connection of multiple convolution kernels. For example, if an image is stored as a tensor in RGB format, the input includes three channels, namely the R matrix, the G matrix, and the B matrix (red, green, and blue, corresponding to three images of the same size). The matrix of each channel is convolved with a corresponding convolution kernel, and all the convolution kernels corresponding to all channels form a filter. Each filter is used to extract different feature data. For another example, a picture has four channels ARGB (transparency and red, green and blue, corresponding to four images of the same size). Assuming the convolution kernel size is 100*100, a total of 16 convolution kernels w1 to w16 are used, among which convolution kernels w1 to w4 form the first filter, convolution kernels w5 to w8 form the second filter, convolution kernels w9 to w12 form the third filter, and convolution kernels w13 to w6 form the fourth filter. Different filters are used to extract different feature data of the input image. The first filter is used to perform convolution operations on the ARGB image, that is, w1 to w4 are used to perform convolution operations on the four images on the four channels, and the first image can be obtained; the first pixel in the upper left corner of this image is the weighted sum of the pixels in the 100*100 area in the upper left corner of the four input images, and so on. Similarly, including other filters, the output of this layer corresponds to 4 "images". Each image pair is a response to a different feature in the original image.

[0084] (5) The convolutional neural network can use the error back propagation (BP) algorithm to correct the size of the parameters in the initial neural network model during the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, the forward transmission of the input signal to the output will generate error loss, and the parameters in the initial neural network model are updated by back propagating the error loss information, so that the error loss converges. The back propagation algorithm is a back propagation movement dominated by error loss, which aims to obtain the optimal parameters of the super-resolution model, such as the weights and attention vectors in the embodiments of the present application.

[0085] (6) Convolution is to extract feature data from the original input. In short, feature data extraction is to extract features from the original input in a small area-by-area manner. Expressed in mathematical terms, convolution is the operation of the convolution kernel and the input matrix of the convolution layer. Usually, the input matrix is ​​a matrix extracted from the image matrix according to the stride of the convolution kernel during convolution. The convolution kernel is a small window that records weights. The convolution kernel slides on the image matrix according to the stride. Each sliding convolution kernel corresponds to a submatrix of the image matrix. The weights in the convolution kernel and the values ​​contained in the submatrix are multiplied and added, and assigned to an element corresponding to the current output feature map (output matrix) of the convolution kernel. Among them, convolution is not limited to convolution of the original input, but also includes re-using convolution on the output result after convolution, which is not limited in the embodiments of the present application. For example, the first convolution extracts low-level feature data, the second convolution extracts mid-level feature data, the third convolution extracts high-level features, and so on. Features can be continuously extracted and compressed, and the higher-level features finally extracted can be understood as further concentration of the original features, making the final features more reliable. The last layer of features can be used to process various tasks, such as classification and regression.

[0086] Exemplarily, the neural network includes at least an input layer, a hidden layer and an output layer, wherein the hidden layer may include one or more of the following: a fully connected layer, a convolutional layer, a pooling layer, an activation layer, a normalization layer, etc., which play different roles in feature extraction. The process of the first device 200 training the neural network to obtain a deep learning model mainly includes: providing a sample image, obtaining a basic neural network, inputting the sample image into the neural network for iterative training, and gradually optimizing the weight matrix in the neural network according to the loss function until the loss function converges, so that the weight matrix in the neural network reaches the optimal value, and the neural network can be used as a recognition model or a defuzzification model.

[0087] Next, the training process is introduced. Model training usually requires multiple sets of sample data, each set of sample data includes two types of images, one is the first sample image containing the features to be processed, and the other is the second sample image containing the target features. It should be noted that the features to be processed here refer to the features that need to be identified, classified, labeled, etc., and the target features refer to one or several types of features that need to be identified or processed.

[0088] For example, if a recognition model is trained, the first sample image of the feature to be processed includes image sub-blocks with different blur levels, such as clear image sub-blocks, image sub-blocks corresponding to a first blur level, and image sub-blocks corresponding to a second blur level, and the first blur level is less than the second blur level. It should be noted that the collected image may have more or fewer blur levels, and the present application does not impose any restrictions on this. The second sample image of the target feature includes image sub-blocks carrying classification labels for reflecting blur levels, such as a clear label for reflecting clarity, a first fuzzy label for reflecting the first blur level, and a second fuzzy label for reflecting the second blur level.

[0089] If the deblurring model is trained, the first sample image of the feature to be processed includes a blurred image with a classification label, for example, a blurred image corresponding to the first blurred label and a blurred image corresponding to the second blurred label. The second sample image of the target feature includes a clear image, for example, a clear image corresponding to the blurred image with the first blurred label and a clear image corresponding to the blurred image with the second blurred label.

[0090] Exemplarily, when the neural network is iteratively trained using the input first sample image and the second sample image, the following steps are generally included:

[0091] Initialization parameters: Initialize the parameters of the neural network, such as weights and biases.

[0092] Forward propagation: Through forward transmission, the first sample image and the second sample image are input into the neural network and processed by the activation function to obtain the output of the network.

[0093] Calculate the loss function: A loss function can be calculated based on the difference between the network's output and the expected value to measure the network's prediction error.

[0094] Back propagation: Through the back propagation algorithm, the gradient for each parameter is calculated, that is, the derivative of the loss function with respect to the parameter.

[0095] Update parameters: Based on the gradient information, an optimization algorithm (such as stochastic gradient descent) can be used to update the network parameters (i.e., the values ​​of weights and biases) to reduce the loss function.

[0096] Repeat the steps of forward propagation, calculating the loss function, back propagation, and updating parameters until a predetermined number of training times or a certain accuracy rate is reached, such as the loss function converges. Through this training process, the neural network can gradually learn the mapping relationship between the features of the first sample image and the features of the second sample image, and make accurate predictions based on these features.

[0097] In a possible implementation, when training different deblurring models, in addition to using different sample data, a model compression method called knowledge distillation can also be used. For example, for a network model with a weak blur level, the model calculation amount can be compressed. For specific examples, see Figure 2 . Figure 2 This is an example diagram of a knowledge distillation scenario provided by the embodiment of the present application. It can be understood that the process of knowledge distillation is divided into two stages: 1. Original model training, training the "Teacher model", referred to as Net-T, which is characterized by a relatively complex model; 2. Simplified model training, training the "Student model", referred to as Net-S, which is a single model with a small number of parameters and a relatively simple model structure. Figure 2 As shown, the network model 301 with a strong degree of fuzziness can be used as a "Teacher model". As a model with strong generalization ability, the "Teacher model" can be used to distill and train the "Student model", and the "Student model" can be directly allowed to learn the generalization ability of the "Teacher model". The resulting "Student model" can be used as a network model 302 with a weak degree of fuzziness.

[0098] Exemplarily, in the present application, it is assumed that the degree of fuzziness corresponding to the first fuzzy label is less than the degree of fuzziness corresponding to the second fuzzy label, so the network model 302 with a weak fuzziness degree may be the model corresponding to the first fuzzy label, and the network model 301 with a strong fuzziness degree may be the model corresponding to the second fuzzy label.

[0099] In one possible implementation, during the training of the model, the L1 paradigm and perceptual loss (Loss) can be used as loss functions. Among them, the L1 paradigm refers to the sum of absolute values, which can be used to measure the difference between the predicted value and the true value. Perceptual Loss is a loss function based on human eye perception, which can help better handle image blur. Then, during the training process, the weighted Loss can be passed back to the network and the parameters of the model can be updated. In this way, the performance and learning ability of the model can be continuously improved through iterative optimization of the network.

[0100] Exemplarily, the loss function Loss=λ0L0+λ1L1+λ2L2(1), where L represents the perceptual Loss, L(1) represents the L1 paradigm, and λ represents the weight.

[0101] In one implementation, in the process of training a network model with a strong degree of fuzziness, the loss weight of the strong fuzzy area can be increased. For example, L0 represents the loss function of the area corresponding to the clear label, L1 represents the loss function of the area corresponding to the first fuzzy label, and L2 represents the loss function of the area corresponding to the second fuzzy label. In the present application, it is assumed that the degree of fuzziness indicated by the first fuzzy label is less than the degree of fuzziness indicated by the second fuzzy label. Therefore, the value of λ2 can be larger, λ2 is greater than λ0, λ2 is greater than λ1, and illustratively, λ0 takes a value of 0.1, λ1 takes a value of 0.2, and λ2 takes a value of 0.7.

[0102] In one implementation, in the process of training the network model of weak fuzziness, the loss weight of the weak fuzzy area is increased. For example, L0 represents the loss function of the area corresponding to the clear label, L1 represents the loss function of the area corresponding to the first fuzzy label, and L2 represents the loss function of the area corresponding to the second fuzzy label. The fuzziness indicated by the first fuzzy label is less than the fuzziness indicated by the second fuzzy label. Therefore, the value of λ1 can be larger, λ1 is greater than λ0, λ1 is greater than λ2, and λ0 is 0.1, λ1 is 0.7, and λ2 is 0.1.

[0103] Taking the above-mentioned situation one (the photographed object is moving, such as using a mobile phone to shoot a person's long jump action) as an example, combined with the UI interface description provided on the electronic device 100, the blurred image is deblurred by the image processing method provided in the embodiment of the present application to obtain a clear image.

[0104] FIG. 3A to FIG. 3F The electronic device 100 is shown as an example of a UI that captures a person's long jump action and displays a clear image.

[0105] in:

[0106] Figure 3A The desktop 111 designed in the embodiment of the present application is exemplarily shown.

[0107] like Figure 3AAs shown, a page with application icons is displayed in the desktop 111, and the page includes multiple application icons (for example, a settings application icon, an application market application icon, a gallery application icon, a browser application icon, etc.). A page indicator is also displayed below the multiple application icons to indicate the positional relationship between the currently displayed page and other pages. A tray area is displayed below the page indicator. Among them, the tray area includes multiple tray icons, for example, a camera application icon, an address book application icon, a phone application icon, and a message application icon. The tray area remains displayed when the page is switched. In some embodiments, the above-mentioned page may also include multiple application icons and a page indicator. The page indicator may not be part of the page and exist independently. The above-mentioned tray icon is also optional, and the embodiments of the present application are not limited to this.

[0108] The electronic device 100 may receive a user operation to open a camera application, such as clicking a camera application icon on the desktop 111. In response to the operation, the electronic device 100 may display a Figure 3B The user interface shown.

[0109] Figure 3B The user interface of the shooting and display service provided in the embodiment of the present application is exemplarily shown, also called a preview interface.

[0110] like Figure 3B As shown, the preview interface may include a mode bar 121, a shooting control 122, a preview interface 123, a review control 124 and a quick function area 126. Among them:

[0111] The mode bar 121 may include multiple camera mode options, such as "night scene", "portrait", "photo", "video recording", etc. Different camera modes can provide users with camera services with different effects. Users can select any of the multiple camera modes to take photos according to different needs. For example, "photo" can be the default camera mode for taking photos. "Video recording" is used to record videos. The "night scene" mode is suitable for photo scenes with dim light, such as at night. The "portrait" mode is suitable for photo scenes where the subject of the photo is a person. The electronic device 100 can also provide more shooting modes, such as "large aperture", "movie", "professional", etc., which are not listed one by one here.

[0112] The electronic device 100 may receive a user operation for switching modes, such as sliding left / right in the mode bar 121, and change the currently used photo mode according to the operation. For example, when receiving a leftward sliding of the mode bar 121 (left sliding operation) and causing the float to stop at the "portrait" option, the electronic device 100 may switch to the "portrait" mode. By default, the electronic device 100 first uses the "photo" mode.

[0113] When the "photographing" mode is used, the shooting control 122 is used to trigger photographing; when the "recording" mode is used, the shooting control 122 can be switched to a recording control, and the recording control is used to trigger recording. The electronic device 100 can receive a user operation acting on the shooting control 122, such as a click operation. After receiving the user operation acting on the shooting control 122, the electronic device 100 can generate a shooting instruction. The electronic device 100 can obtain the image reported by the camera at the corresponding timestamp according to the shooting instruction, and then save it as a photo.

[0114] The preview interface 123 displays the image frames captured by the camera. In the "photographing" mode, the preview interface 123 is a photo-taking preview interface, and in the "recording" mode, the preview interface 123 is a recording preview interface. Here, the preview interface 123 can display in real time the images processed by the image processing algorithms corresponding to different modes, so that the user can perceive the photographing effects corresponding to different photographing modes in real time.

[0115] The review control 124 can be used to browse thumbnails of the photos / videos that have been taken. When receiving a user operation acting on the review control 124, the electronic device 100 can also display the wonderful photos corresponding to the thumbnails.

[0116] The quick function area 126 may include an HDR function option 126A, an AI function option 126B, a flash function option 126C, a color mode function option 126D, a setting function option 126E, etc. The HDR function option 126A can be used to trigger the electronic device 100 to use the HDR algorithm to perform image fusion processing when it is turned on. The AI ​​function option 126B can be used to trigger the electronic device 100 to identify the photo scene in the preview screen when it is turned on. The current AI function option 126B is in the off state. The flash function option 126C can be used to trigger the electronic device 100 to turn on or off the flash. The color mode function option 126D can be used to trigger the electronic device 100 to use a color filter to process the image collected by the camera. The setting function option 126E can be used to set the photo parameters of the electronic device 100 (for example, image size, image storage format, etc.).

[0117] When the camera is aimed at a person who is doing a long jump, the electronic device 100 can sequentially display the images of the long jump captured by the camera at different times in the preview interface 123. For example, at the first time, Figure 3B The preview interface 123 shown shows the character's preparation action before jumping. At the second moment, Figure 3C The preview interface 123 shown shows the squatting action of the character when jumping. The second moment is the moment after the first moment.

[0118] Assume that at the second moment, Figure 3CWhen the preview screen shows a picture of a person squatting when jumping, the user clicks the shooting control 122 and the electronic device 100 can obtain the picture.

[0119] However, since the person doing the long jump is always in motion, there is a relative motion between the photographed object (i.e., the person doing the long jump) and the electronic device 100 from the first moment to the second moment. Then the instantaneous speeds of the pixels of various parts of the photographed object on the imaging plane are different. For example, the instantaneous speeds of the pixels corresponding to the arms and legs of the person on the imaging plane may be greater than the instantaneous speeds of the pixels corresponding to the head or body on the imaging plane, and the pixels of the photographed object on the imaging plane will also change accordingly.

[0120] Therefore, at the second moment, the electronic device 100 responds to the user operation of the shooting control 122 and obtains the first image (ie Figure 3C There may be different levels of blur in the image (as shown in the figure). For example, the arms and legs of a person may be more blurred than the head or body.

[0121] In one implementation, the electronic device 100 identifies the blurred area in the first image based on the image processing method provided by the present application, and determines the blur degree corresponding to the blurred area. Then, the electronic device 100 inputs the image areas corresponding to different blur degrees into the corresponding deblurring model for deblurring, thereby obtaining a clear image of the first image, and saves the clear image. The process example can be seen below Figure 4 .

[0122] Figure 4 is a schematic diagram of an image processing scenario provided by an embodiment of the present application, such as Figure 4 As shown, the electronic device 100 responds to the user operation of the shooting control, obtains the first image captured by the camera, cuts the first image into N small image sub-blocks, identifies the blur degree of each small image sub-block based on the recognition model 300, and marks each image sub-block according to a preset classification label (for example, a clear label 0, a first fuzzy label 1, and a second fuzzy label 2). Among them, the clear label is used to indicate that the image sub-block is clear, the first fuzzy label is used to indicate that the image sub-block is weakly blurred, and the second fuzzy label is used to indicate that the image sub-block is strongly blurred.

[0123] Then, the recognition model 300 can output image sub-blocks carrying classification labels, and the electronic device 100 clusters the image sub-blocks with the same classification labels according to the nearest neighbor method, and divides the original first image into multiple regions of interest in an adaptive manner, for example, a static area (carrying a clear label, for example, obtained by clustering multiple image sub-blocks with a clear label 0), a weakly blurred area (carrying a first blurred label, for example, obtained by clustering multiple image sub-blocks with a first blurred label 1), and a strongly blurred area (carrying a second blurred label, for example, obtained by clustering multiple image sub-blocks with a second blurred label 2).

[0124] Next, the electronic device 100 inputs each region into the corresponding deep learning model according to the classification label for deblurring. For example, the electronic device 100 inputs the strongly blurred region into the network model 301 with a strong blur level for deblurring, and the network model 301 with a strong blur level outputs a clear image corresponding to the strongly blurred region. The electronic device 100 inputs the weakly blurred region into the network model 302 with a weak blur level for deblurring, and the network model 302 with a weak blur level outputs a clear image corresponding to the weakly blurred region. Therefore, the electronic device 100 can obtain a clear image after deblurring the first image.

[0125] like Figure 3D As shown, the electronic device 100 can generate a thumbnail based on the clear image and update the thumbnail in the review control 124. Subsequently, the user can view the clear image through the updated thumbnail. The electronic device 100 can detect the operation on the review control 124 and, in response to the operation, can display Figure 3E The interface 125 shown is a large picture display interface of the gallery, in which a clear image obtained by deblurring the blurred image after acquisition is displayed. Figure 3F The user interface shown in the example, then choose to view the clear image, Figure 3E The user interface shown is used to display thumbnails of large images stored in the gallery. Figure 1 They are usually larger than the thumbnails shown in the preview interface.

[0126] Not limited to the above preview scene, the electronic device 100 can deblur the collected image in the video recording scene to obtain a clear image. For example, the electronic device 100 records the long jump process of a long jump athlete, and the preview window of the electronic device 100 can display the motion state of the athlete captured by the camera at different times during the long jump process at one time. During the recording process, the video recorded by the electronic device 100 is composed of N frames of images. Assuming that the 3rd frame image and the 5th frame image are blurred images, the electronic device 100 can first extract the above blurred images from the video, and then deblur the 3rd frame image and the 5th frame image through the image processing method of the present application to obtain a clear 3rd frame image and a clear 5th frame image, and finally insert the clear 3rd frame image and the clear 5th frame image into the recorded video. When the user ends the recording by ending the recording control, the image in the saved video is clear.

[0127] See also Figure 5 , Figure 5 : is a flowchart of an image processing method provided by an embodiment of the present application, including but not limited to the following steps:

[0128] Step S501: the electronic device obtains a first image captured by a camera.

[0129] Specifically, the electronic device 100 runs an application that can start a camera, for example, when running a camera application, an instruction to instruct the acquisition of image data can be generated according to business needs, thereby acquiring an image (i.e., a first image) acquired by the camera. Exemplarily, the electronic device 100 responds to a user operation of starting a camera application, acquires an image in real time through the camera, and responds to a user's shooting control to acquire the first image from the image acquired in real time by the camera.

[0130] It is understandable that a circular buffer of a fixed size may be preset in the electronic device 100. After each frame of image is generated, the camera may send the frame of image to the above buffer, and the image stored in the buffer may be referred to as the original image. Exemplarily, the buffer in the electronic device 100 may be used to cache the original image reported by the camera, and after each frame of original image reported by the camera is received, the electronic device 100 may write it into the buffer. When the buffer is full, after receiving a new frame of image, the earliest written frame of original image may be read from the buffer, and then the newly received image may be stored. Based on different uses, the electronic device 100 may divert the original image stored in the buffer to obtain different image streams, such as a photo stream, a preview stream, a video stream, and the like.

[0131] In one implementation, the image processing method provided in the embodiment of the present application can be applied to any of the above-mentioned image streams, and the electronic device 100 can obtain the first image from any of the above-mentioned image streams.

[0132] Step S502: Divide the first image into L types of regions of interest.

[0133] Wherein, L is a positive integer, and each of the L ROIs corresponds to a classification label, and the classification label is used to indicate the blur degree of the ROI. The region of interest (ROI) is an area to be processed that is outlined in a square, circle, ellipse or irregular polygon from the processed image (i.e., the first image). This area is the area that is focused on during image analysis.

[0134] In a possible implementation, the electronic device 100 performs image patching on the first image to obtain N image patches, then determines a classification label corresponding to each of the N image patches, and finally clusters the N image patches based on the classification labels to divide the first image into L types of regions of interest.

[0135] In the process of image segmentation, the first image is divided into a series of small patches of equal or different sizes, which are usually called image patches or simply patches. A "patch" refers to a small area or fragment in an image. This concept is usually used to decompose a large image into smaller parts so that each patch can be processed, analyzed or feature extracted separately.

[0136] In one implementation, the electronic device 100 inputs N image sub-blocks into a recognition model, and the recognition model outputs image sub-blocks carrying classification labels, thereby determining the classification label corresponding to each image sub-block.

[0137] The recognition model is a deep neural network that is trained to accurately identify the degree of blur. The model is composed of several convolutional layers, pooling layers, and fully connected layers. In one scenario, different regions in the first image have different blur procedures. The electronic device 100 can identify the degree of blur of each of the N image sub-blocks through the recognition model.

[0138] See also Figure 6 , Figure 6 is a schematic diagram of identifying the degree of blur provided in an embodiment of the present application. Figure 6As shown, the first image is divided into N image sub-blocks, and the electronic device 100 inputs the N image sub-blocks into the recognition model. The recognition model can label each image sub-block according to the degree of blur, and output the classification label corresponding to each image sub-block. The classification label can be used to indicate the degree of blur or clarity of the image sub-block. For example, the clarity label 0 is used to indicate that the image sub-block belongs to a static area and is clear; the first blur label 1 is used to indicate that the image sub-block belongs to a weak blur area and the degree of blur is relatively blurry; the second blur label 2 is used to indicate that the image sub-block belongs to a strong blur area and the degree of blur is very blurry. It should be noted that since the image resolution of the image sub-block is low and the recognition model is a lightweight model, the delay for the recognition model to identify the degree of blur of each image sub-block is short.

[0139] In one implementation, the electronic device 100 divides the first image into L regions of interest (ROI) based on the classification label corresponding to each image sub-block through a classification network. Wherein L is a positive integer, and the value of L is the same as the number of classification labels. For example, if there are three classification labels, the first image is divided into three regions of interest. In one implementation, the electronic device can reallocate image sub-blocks according to the same classification label based on a clustering method such as the nearest neighbor domain, so that the first image can be divided into regions of interest corresponding to multiple classification labels in an adaptive manner, such as a region of interest corresponding to a first fuzzy label, a region of interest corresponding to a second fuzzy label, and a region of interest corresponding to a clear label.

[0140] Among them, the nearest neighbor domain is a commonly used clustering method, which clusters based on the similarity or distance between samples. Each sample is assigned to a cluster or a group according to its distance or similarity with other samples.

[0141] See also Figure 7 , Figure 7 is a schematic diagram of a region of interest provided in an embodiment of the present application, such as Figure 7 As shown, the electronic device 100 allocates image sub-blocks with the same classification label or adjacent image sub-blocks to a cluster or a group according to the nearest neighbor clustering method. This cluster or this group can be considered as the region of interest corresponding to the classification label. Figure 7It can be seen that the electronic device 100 can divide the first image into a region of interest corresponding to a clear label 0, a region of interest corresponding to a first fuzzy label 1, and a region of interest corresponding to a second fuzzy label 2 according to the classification labels. Exemplarily, the region of interest corresponding to the clear label 0 includes a plurality of image sub-blocks corresponding to the clear label 0. The region of interest corresponding to the first fuzzy label 1 includes a plurality of image sub-blocks corresponding to the first fuzzy label 1, and may also include an image sub-block corresponding to the second fuzzy label 2 and an image sub-block corresponding to the clear label 0. The region of interest corresponding to the second fuzzy label 2 includes a plurality of image sub-blocks corresponding to the second fuzzy label 2, and may also include an image sub-block corresponding to the first fuzzy label 1 and an image sub-block corresponding to the clear label 0.

[0142] It is understandable that in order to ensure a natural edge transition of the region of interest during processing, the regions of interest may overlap with each other, and there may be overlapping regions. For example, the region of interest corresponding to the first fuzzy label 1 includes all or part of the image sub-blocks in the region of interest corresponding to the second fuzzy label 2, and / or the region of interest corresponding to the second fuzzy label 2 includes all or part of the image sub-blocks in the region of interest corresponding to the first fuzzy label 1.

[0143] Step S503: inputting L types of regions of interest into corresponding deblurring models respectively based on the classification labels to obtain clear images corresponding to the regions of interest.

[0144] Specifically, when the electronic device 100 performs deblurring processing on the first image, different regions of interest in the first image are input into the deblurring model corresponding to the classification label of the region of interest according to the classification label, and the region of interest is deblurred by the deblurring model, so that an estimation result of a clear image can be output. It can be understood that there may be an area corresponding to a clear label in the first image, and the region of interest corresponding to the clear label is a static area. Because the static area belongs to a clear image and there is no blurred picture, the electronic device 100 does not need to perform deblurring processing on the region of interest corresponding to the clear label, and can directly output it. Finally, the electronic device 100 splices each obtained clear image together to obtain a clear image of the first image.

[0145] In one scenario, the region of interest corresponding to the fuzzy label (such as the first fuzzy label or the second fuzzy label) may contain one or more image sub-blocks of a static region (such as a clear label 0). Figure 7As shown, there is an image sub-block with a clear label 0 in the region of interest corresponding to the first fuzzy label 1, and there is an image sub-block with a clear label 0 in the region of interest corresponding to the second fuzzy label 2. Because the image sub-block corresponding to the clear label 0 is clear, abnormal texture may appear in the image sub-block after being processed by the deblurring model. In order to solve this problem, the electronic device can compare the pixel values ​​of the output image sub-block and the input image sub-block of the image sub-block. If the comparison result shows that there are more abnormal textures in the output image sub-block, the image sub-block input to the deblurring model will replace the output image sub-block.

[0146] For example, see Figure 8 , Figure 8 Schematic diagram of a static area replacement provided by an embodiment of the present application. Figure 8 As shown, after the input image sub-block 401 is input into the deblurring model, an output image sub-block 402 can be obtained. The electronic device can obtain the output image sub-block 402 according to the formula R=I Get -I inpun Calculate the pixel difference, for example Figure 8 Pixel I of the input image sub-block 401 is shown with a clear label 0 Gen and the pixel I of the corresponding output image sub-block 402 input If the mean σ=var(R) of the difference R is greater than the threshold or the variance μ=mean(R) of the difference R is greater than the threshold, it indicates that texture appears in the output image sub-block 402 due to the deblurring process of the deblurring model. The electronic device can replace the output image sub-block 402 with the input image sub-block 401 of the original input level, thereby ensuring that the static area is not over-deblurred.

[0147] In a possible implementation, after acquiring clear images of L regions of interest, the electronic device can process the overlapping parts of the L regions of interest by weighted average fusion, thereby splicing the clear images corresponding to the L regions of interest to obtain and save a clear image of the first image.

[0148] Exemplarily, the electronic device 100 in the embodiment of the present application can be a smart screen device, a smart television (television, TV), a mobile phone, a tablet computer, an ultra-mobile personal computer (UMPC), a netbook, as well as a cellular phone, a personal digital assistant (personal digital assistant, PDA), a wearable device (such as: a smart watch, a smart bracelet), and other devices with a shooting function. The embodiment of the present application does not impose any special restrictions on the specific form of the electronic device 100.

[0149] For example, taking the electronic device 100 as a mobile phone, Fig. 9 is a schematic diagram of the structure of an electronic device 100 provided in an embodiment of the present application. That is, exemplary, Figure 1 The electronic device 100 shown may be a mobile phone.

[0150] like Fig. 9 As shown, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 221, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0151] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0152] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0153] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0154] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present invention is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0155] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0156] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0157] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0158] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0159] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, demodulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0160] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0161] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0162] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0163] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0164] ISP is used to process data fed back by camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise and brightness of the image. For example, ISP deblurs a blurred image based on the image processing method provided in this application to obtain a clear image. ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0165] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0166] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0167] The internal memory 221 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM).

[0168] Random access memory may include static random-access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM, for example, the fifth generation DDR SDRAM is generally referred to as DDR5 SDRAM), etc.; non-volatile memory may include disk storage devices and flash memory.

[0169] Flash memory can be divided into NOR FLASH, NAND FLASH, 3D NAND FLASH, etc. according to the operating principle; single-level cell (SLC), multi-level cell (MLC), triple-level cell (TLC), quad-level cell (QLC), etc. according to the storage unit potential level; universal flash storage (UFS), embedded multi media card (eMMC), etc. according to the storage specification.

[0170] The random access memory can be directly read and written by the processor 110, and can be used to store executable programs (such as machine instructions) of the operating system or other running programs, and can also be used to store user and application data.

[0171] The non-volatile memory may also store executable programs and user and application data, etc., and may be loaded into the random access memory in advance for direct reading and writing by the processor 110 .

[0172] The external memory interface 120 can be used to connect to an external non-volatile memory to expand the storage capacity of the electronic device 100. The external non-volatile memory communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, files such as music and videos are stored in the external non-volatile memory.

[0173] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0174] The software system of the electronic device 100 (such as a mobile phone) can adopt a layered architecture, a transaction-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. This embodiment of the application takes the Android system with a layered architecture as an example to illustrate the software architecture of the mobile phone. Fig.10 , Fig.10 It is a schematic diagram of the software architecture of the electronic device 100 provided in an embodiment of the present application.

[0175] The layered architecture divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom, namely, the application layer, the application framework layer (frame), the hardware abstraction layer (HAL), the driver layer, and the hardware layer. In an embodiment of the present application, when the application layer interacts with the user and triggers the scene of starting the first image acquisition, the application layer can call the camera API and pass the control instruction indicating the image acquisition to the camera of the hardware layer through the application framework layer, the HAL layer, and the driver layer. The camera can respond to the control instruction indicating the acquisition of the first image, collect image data, and pass the obtained image data to the application layer through the kernel layer, the HAL layer, and the framework layer.

[0176] like Fig.10As shown, the application may include camera, gallery and other applications. Among them, the camera application may include but is not limited to: UI module, camera module, gallery module, etc. Among them, the UI module may be a cameraUI module, which is mainly responsible for the human-computer interaction of the camera application, such as controlling the display of the preview interface and the preview screen therein, receiving and responding to user operations occurring in the preview interface. The camera module provides camera functions, focus functions, etc. The gallery module can be used to store clear photos taken by the user in the file system or a specific database of the electronic device for retrieval by applications such as the gallery.

[0177] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0178] like Fig.10 As shown, the application framework layer mainly involves the camera framework, which may include camera management and camera device and other camera access interfaces, playing a role of connecting the upper and lower parts, and can interact with the camera application through the application API, and can interact with HAL through the HAL interface definition language (HIDL). It may also include a window manager, and the camera application and the gallery application can display the photos taken to the user with the support of the window manager. In addition, the application framework layer may also include a window manager, a content provider, a view system, a resource manager, a notification manager, etc.

[0179] The hardware abstraction layer (HAL) is an interface layer located between the application framework layer and the driver layer, and provides a virtual hardware platform for the operating system. Exemplarily, the hardware abstraction layer may include a camera hardware abstraction layer and an image processing module. Among them, the camera hardware abstraction layer may provide virtual hardware for camera device 1 (first camera), camera device 2 (second camera) and more camera devices.

[0180] The image processing module stores a variety of models. For example, in an embodiment of the present application, the image processing module may include a recognition model, a classification model, a first deblurring model, a second deblurring model, etc. Among them:

[0181] The recognition model is used to identify a classification label of each image sub-block in the first image, and the classification label is used to indicate the blur degree or clarity of the image sub-block.

[0182] In one scenario, the classification label includes a clear label, a first fuzzy label, and a second fuzzy label.

[0183] The classification model is used to divide the first image based on the classification label to obtain a region of interest corresponding to the first fuzzy label, a region of interest corresponding to the second fuzzy label, and a region of interest corresponding to the clear label.

[0184] The first deblurring model is used to perform deblurring processing on the region of interest corresponding to the first fuzzy label to obtain a clear image.

[0185] The second deblurring model is used to perform deblurring processing on the region of interest corresponding to the second fuzzy label to obtain a clear image.

[0186] The driver layer is a layer between hardware and software, which includes drivers for various hardware. The driver layer may include camera device drivers, digital signal processor drivers, and image processor drivers. Among them, the camera device driver is used to drive the image sensor of one or more cameras in the camera to collect images and drive the image signal processor to pre-process the image. The digital signal processor driver is used to drive the digital signal processor to process images. The image processor driver is used to drive the graphics processor to process images.

[0187] The hardware layer may include a camera, an image signal processor, a digital signal processor, and an image processor. The camera may include one or more image sensors of the camera (e.g., image sensor 1, image sensor 2, etc.). Optionally, the camera may also include a camera motor, a lens, a TOF sensor, and the like.

[0188] It should be understood that each step in the above method embodiment provided by the present application can be completed by an integrated logic circuit of hardware in a processor or by instructions in the form of software. The method steps disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware processor, or by a combination of hardware and software modules in a processor.

[0189] The present application also provides a chip system, which includes at least one processor for implementing the functions involved in the method performed by the electronic device 100 in any of the above embodiments.

[0190] In one possible design, the chip system also includes a memory, which is used to store program instructions and data, and the memory is located inside or outside the processor.

[0191] The chip system may be composed of the chip, or may include the chip and other discrete devices.

[0192] Optionally, the processor in the chip system may be one or more. The processor may be implemented by hardware or by software. When implemented by hardware, the processor may be a logic circuit, an integrated circuit, etc. When implemented by software, the processor may be a general-purpose processor implemented by reading software code stored in a memory.

[0193] Optionally, the memory in the chip system may also be one or more. The memory may be integrated with the processor or may be separately arranged with the processor, which is not limited in the embodiments of the present application. Exemplarily, the memory may be a non-transient processor, such as a read-only memory ROM, which may be integrated with the processor on the same chip or may be arranged on different chips respectively. The embodiments of the present application do not specifically limit the type of memory and the arrangement of the memory and the processor.

[0194] Exemplarily, the chip system can be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD) or other integrated chips.

[0195] The present application also provides a computer program product, which includes: a computer program (also referred to as code, or instruction), which, when executed, enables a computer to execute the method executed by the electronic device 100 in any of the above embodiments.

[0196] The present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program (also referred to as code or instruction). When the computer program is executed, the computer executes the method executed by the electronic device 100 in any of the above embodiments.

[0197] The various implementation modes of the present application can be combined arbitrarily to achieve different technical effects.

[0198] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in this application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integration. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk), etc.

[0199] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be implemented by a computer program to instruct the relevant hardware to complete the process, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-mentioned method embodiments. The aforementioned storage medium can be a non-volatile memory, and specifically can include: ROM or random access memory RAM, a magnetic disk or an optical disk and other media that can store program codes. In short, the above is only an embodiment of the technical solution of the present invention, and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made according to the disclosure of the present invention should be included in the scope of protection of the present invention.

Claims

1. An image processing method, characterized in that: Applied to an electronic device, the electronic device includes a camera, and the method includes: Acquire a first image captured by the camera; Performing image segmentation on the first image to obtain N image sub-blocks, determining classification labels corresponding to the N image sub-blocks, clustering the N image sub-blocks based on the classification labels, and dividing the first image into L types of regions of interest, wherein each of the L types of regions of interest corresponds to a classification label, and the classification label is used to indicate a degree of blurriness of the region of interest, and the classification label includes a clear label, a first fuzzy label, and a second fuzzy label, the degree of blurriness indicated by the first fuzzy label is less than the degree of blurriness indicated by the second fuzzy label, the L types of regions of interest include a region of interest corresponding to the first fuzzy label and a region of interest corresponding to the second fuzzy label, the region of interest corresponding to the first fuzzy label includes an image sub-block corresponding to the first fuzzy label, an image sub-block corresponding to the second fuzzy label, and an image sub-block corresponding to the clear label, and the region of interest corresponding to the second fuzzy label includes an image sub-block corresponding to the first fuzzy label, an image sub-block corresponding to the second fuzzy label, and an image sub-block corresponding to the clear label, and N is a positive integer, and L is a positive integer; Based on the classification labels, the L types of regions of interest are respectively input into corresponding deblurring models to obtain output images corresponding to the L types of regions of interest, wherein the deblurring model is a trained deep learning model, and the deblurring model includes a deblurring model corresponding to a first fuzzy label and a deblurring model corresponding to a second fuzzy label, and in the deblurring model corresponding to the first fuzzy label, the weight corresponding to the first fuzzy label is greater than the weight corresponding to the second fuzzy label, and in the deblurring model corresponding to the second fuzzy label, the weight corresponding to the second fuzzy label is greater than the weight corresponding to the first fuzzy label; In the case where there is texture in an output image sub-block in the output image, the output image sub-block is replaced with an input image sub-block to obtain a clear image corresponding to the output image, wherein the input image sub-block is an image sub-block corresponding to the clear label in the L types of regions of interest input into the deblurring model, and the output image sub-block is an image sub-block corresponding to the input image sub-block output by the deblurring model.

2. The method according to claim 1, characterized in that The deblurring model corresponding to the first fuzzy label is obtained by training based on the sample data corresponding to the first fuzzy label; The deblurring model corresponding to the second fuzzy label is obtained by training according to the sample data corresponding to the second fuzzy label.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: Determining whether there is texture in the output image sub-block; In the case that there is no texture in the output image sub-block, clear images corresponding to the L types of regions of interest are obtained.

4. The method according to claim 3, characterized in that The determining whether there is a texture in the output image sub-block comprises: It is determined that there is texture in the output image sub-block by that a mean of pixel difference values ​​between the output image sub-block and the input image sub-block corresponding to the clear label is greater than a first threshold or a variance of the pixel difference values ​​is greater than a second threshold.

5. The method according to claim 1, characterized in that The clustering of the N image sub-blocks based on the classification labels to divide the first image into L regions of interest includes: The N image sub-blocks are clustered according to the same classification label in a nearest neighbor clustering manner, and the first image is divided into L types of regions of interest, wherein the value of L is the same as the number of types of the classification labels.

6. The method according to claim 1, characterized in that There are overlapping regions among the L regions of interest, and the method further includes: The overlapping parts are processed by weighted average fusion to splice the clear images corresponding to the L types of regions of interest, thereby obtaining a clear image of the first image.

7. The method according to claim 6, characterized in that The method further comprises: saving a clear image of the first image; A clear image of the first image is displayed.

8. An electronic device, characterized in that: The electronic device comprises: one or more processors; a memory; wherein the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code comprises computer instructions, and the one or more processors call the computer instructions so that the electronic device executes the method as described in any one of claims 1-7.

9. A chip system, characterized in that: The chip system is applied to an electronic device, and the chip system includes one or more processors, and the processor is used to call computer instructions so that the electronic device executes the method as described in any one of claims 1-7.

10. A computer program product comprising instructions, characterized in that When the computer program product is executed on an electronic device, the electronic device is enabled to execute the method according to any one of claims 1 to 7.

11. A computer-readable storage medium comprising instructions, characterized in that: When the instructions are executed on an electronic device, the electronic device is caused to execute the method as claimed in any one of claims 1 to 7.

Images