Image processing model training methods, electronic devices, and computer storage media

Abstract

This application provides a training method for an image processing model, an electronic device, and a computer storage medium. The training method for the image processing model includes: obtaining a sample image group, the sample image group including a first image and a second image with the same image content; dividing the first image into regions according to image features and corresponding preset region division parameters using an image processing model to obtain at least two types of regions to be blurred; constructing blur degree data corresponding to each type of region to be blurred based on image features and corresponding blur parameters, thereby obtaining blur degree data corresponding to the first image; deblurring the second image using the image processing model based on the blur degree data corresponding to the first image to obtain a deblurred image; calculating a loss value based on the first image and the deblurred image; and adjusting the image processing model based on the loss value to deblur the image using the adjusted image processing model.

Description

Technical Field

[0001] This application relates to the field of video image processing, and more particularly to a training method for an image processing model, an electronic device, and a computer storage medium. Background Technology

[0002] In scenarios such as conference room audio and video capture, online live streaming, real-time conferencing, and real-time communication (RTC), the image quality is prone to blurring and various distortions / artifacts due to limitations of the camera and video capture environment, as well as the effects of downsampling and encoding / decoding compression.

[0003] Generally, image quality can be improved by methods such as deblurring, resolution enhancement (super-resolution), and blemish removal. Among these three methods, deblurring is the most effective in improving image quality. However, commonly used deblurring schemes require estimating the corresponding blur kernel based on the image content and then performing deblurring based on the kernel, resulting in high computational costs and poor performance. Summary of the Invention

[0004] In view of this, embodiments of this application provide a training scheme for an image processing model to at least partially solve the above-mentioned problems.

[0005] According to a first aspect of the embodiments of this application, a method for training an image processing model is provided, comprising: obtaining a sample image group, the sample image group including a first image and a second image with the same image content, wherein the clarity of the first image is greater than that of the second image; extracting image features from the first image using an image processing model to obtain image features; dividing the first image into regions using the image processing model according to the image features and their corresponding preset region division parameters to obtain at least two types of regions to be blurred; constructing blur degree data corresponding to each type of region to be blurred according to the image features of each type of region to be blurred and their respective corresponding blur parameters to obtain blur degree data corresponding to the first image; deblurring the second image using the image processing model according to the blur degree data corresponding to the first image to obtain a deblurred image; calculating a loss value based on the first image and the deblurred image; and adjusting at least one of the following in the image processing model according to the loss value: preset region division parameters and deblurring parameters, so as to deblur the image using the adjusted image processing model.

[0006] According to a second aspect of the embodiments of this application, a method for deploying an image processing model is provided, comprising: sending an image to be processed to extract image features from the image to be processed using multiple pre-trained image processing models to obtain image features; dividing the image to be processed into regions based on the matching degree between the image features and preset region division parameters using the image processing models to obtain at least two types of regions to be processed; and deblurring the at least two types of regions based on deblurring parameters corresponding to each type of region to be processed using the image processing models to obtain deblurred images output by the multiple image processing models respectively, wherein the image parameters of the multiple image processing models are different, and the image parameters include preset region division parameters and / or deblurring parameters; displaying the multiple deblurred images to a user; sending a deblurred image selected by the user from the multiple deblurred images to receive and deploy the image processing model corresponding to the selected deblurred image, and performing deblurring processing using the image processing model.

[0007] According to a third aspect of the embodiments of this application, an image processing method is provided, comprising: determining an image to be processed; extracting image features from the image to be processed using an image processing model to obtain image features; dividing the image to be processed into regions using the image processing model based on the degree of matching between the image features and preset region division parameters to obtain at least two types of regions to be processed; and deblurring the at least two types of regions using the image processing model based on deblurring parameters corresponding to each type of region to be processed to obtain a deblurred image corresponding to the image to be processed.

[0008] According to a fourth aspect of the present application, an electronic device is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; the memory is used to store at least one executable instruction, wherein the executable instruction causes the processor to perform an operation corresponding to the method described above.

[0009] According to a fifth aspect of the present application, a computer storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the method described above.

[0010] According to the training scheme of the image processing model provided in the embodiments of this application, during the training process, at least two types of regions to be blurred can be obtained by dividing the first (high-definition) image of the sample image group according to image features and preset region division parameters. Then, corresponding blur degree data is constructed for each of the at least two types of regions to be blurred according to the blur parameters corresponding to each type. The image processing model can then perform deblurring on the second (low-definition) image of the sample image group based on the blur degree data. Furthermore, the image processing model is adjusted based on the loss value calculated from the deblurred image and the first (high-definition) image. This allows the image processing model to learn different types of image region division methods and their corresponding deblurring schemes. Therefore, when performing deblurring through the trained image processing model, the image can be divided into different types of regions, and deblurring can be performed on each type of region according to the corresponding deblurring parameters. This greatly improves the quality of deblurring. Moreover, during the deblurring process, the trained parameters can be directly used, eliminating the need for computationally intensive operations such as estimating the blur kernel. This reduces the computational load on the electronic devices deploying the image processing model and improves the real-time performance of deblurring. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0012] Figure 1 A schematic diagram of an exemplary system for training an image processing model applicable to embodiments of this application;

[0013] Figure 2A A flowchart illustrating the steps of a training method for an image processing model provided in this application embodiment;

[0014] Figure 2B for Figure 2A A schematic diagram of a scenario example in the illustrated embodiment;

[0015] Figure 3A A flowchart illustrating the steps of another image processing model training method provided in this application embodiment;

[0016] Figure 3B for Figure 3A A schematic diagram of a scenario example in the illustrated embodiment;

[0017] Figure 4 A flowchart illustrating the steps of a method for deploying an image processing model, as provided in an embodiment of this application.

[0018] Figure 5 This is a flowchart illustrating the steps of a deployment method for an image processing model according to Embodiment 4 of this application;

[0019] Figure 6 This is a schematic diagram of the structure of an electronic device according to Embodiment 4 of this application;

[0020] Figure 7 This is a schematic diagram of a usage scenario according to Embodiment 5 of this application. Detailed Implementation

[0021] To enable those skilled in the art to better understand the technical solutions in the embodiments of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application should fall within the protection scope of the embodiments of this application.

[0022] The specific implementation of the embodiments of this application will be further described below with reference to the accompanying drawings.

[0023] Figure 1 An exemplary system is shown that provides a training method for an image processing model applicable to embodiments of this application. For example... Figure 1 As shown, the system 100 may include a cloud server 102, a communication network 104, and / or one or more user devices 106. Figure 1 The example in the text shows multiple user devices.

[0024] The cloud server 102 can be any suitable device for storing information, data, programs, and / or any other suitable type of content, including but not limited to distributed storage system devices, server clusters, computing cloud server clusters, etc. In some embodiments, the cloud server 102 can perform any suitable function. For example, in some embodiments, the cloud server 102 can be used to train an image processing model. As an optional example, in some embodiments, the cloud server 102 can be used to train an image processing model based on user selection. As another example, in some embodiments, the cloud server 102 can be used to send an image processing model to a user device.

[0025] In some embodiments, communication network 104 may be any suitable combination of one or more wired and / or wireless networks. For example, communication network 104 may include any one or more of the following: the Internet, intranet, wide area network (WAN), local area network (LAN), wireless network, digital subscriber line (DSL) network, frame relay network, asynchronous transfer mode (ATM) network, virtual private network (VPN), and / or any other suitable communication network. User equipment 106 may be connected to communication network 104 via one or more communication links (e.g., communication link 112), and communication network 104 may be linked to cloud server 102 via one or more communication links (e.g., communication link 114). Communication links may be any communication link suitable for transmitting data between user equipment 106 and cloud server 102, such as network links, dial-up links, wireless links, hardwired links, any other suitable communication links, or any suitable combination of such links.

[0026] User equipment 106 may include any one or more user devices suitable for presenting images, compressed images, image sequences, and / or sequences of compressed images. In some embodiments, user equipment 106 may include any suitable type of device. For example, in some embodiments, user equipment 106 may include mobile devices, tablet computers, laptop computers, desktop computers, wearable computers, game consoles, media players, vehicle entertainment systems, and / or any other suitable type of user equipment.

[0027] Based on the above system, this application first provides a training method for an image processing model, which will be described below through several embodiments.

[0028] Figure 2A This is a flowchart illustrating a training method for an image processing model provided in an embodiment of this application, as shown below. Figure 2A As shown, it includes:

[0029] S201. Obtain a sample image group, which includes a first image and a second image with the same image content, wherein the clarity of the first image is greater than that of the second image.

[0030] In this embodiment, "identical image content" means that the objects, their positions, and their colors are the same in both the first and second images. This can also be understood as images of different resolutions captured from the same scene. The second image can be obtained directly or by compressing the first image; both are within the scope of this application.

[0031] In this embodiment, the first image can be a high-definition large image, and the second image can be a normal resolution image or a low resolution image.

[0032] S202. Extract image features from the first image using an image processing model to obtain image features;

[0033] In this embodiment, the image features can be any image features, as long as they can be used to determine the areas that need to be focused on deblurring during image quality enhancement. For example, image features can be image edge features, image color features, texture features, depth features, semantic features, etc. The areas that need to be focused on deblurring during image quality enhancement can be, for example, the edges of objects in the image, faces in the image, etc.

[0034] S203. Using the image processing model, the first image is divided into regions according to the image features and their corresponding preset region division parameters to obtain at least two types of regions to be blurred.

[0035] The preset region division parameters can be the threshold corresponding to the extracted image features, such as the edge region threshold corresponding to the image edge features. The region to be blurred obtained by dividing it can be, for example, the pixel identified as the image edge and the part of its surrounding area within the edge region threshold. The image edge threshold can be, for example, the pixel distance between the pixel and the pixel of the image edge is 10 pixels, or the pixel value difference between the pixel and the pixel corresponding to the image edge is 10, etc. This embodiment does not limit this.

[0036] In this embodiment, the first image can be divided into regions based on image features. Regions that match the image features can be classified as first-class regions to be blurred, and regions outside the first-class regions to be blurred can be classified as second-class regions to be blurred.

[0037] If an image has multiple features, a type of region to be blurred can be defined based on at least one image feature, thus obtaining multiple types of regions to be blurred. The region other than the region to be blurred defined based on the image features is then taken as another type of region to be blurred.

[0038] S204. Based on the image features of various regions to be blurred and their corresponding blur parameters, construct blur degree data corresponding to each type of region to be blurred, and obtain blur degree data corresponding to the first image.

[0039] Since the regions to be blurred are divided according to image features, the various regions to be blurred can be blurred according to the image features, and a good blurring effect can be achieved. Therefore, based on the image features of various regions to be blurred and their corresponding blurring parameters, blurring degree data corresponding to each type of region to be blurred can be constructed, thereby obtaining the blurring degree data corresponding to the entire first image.

[0040] For example, if the region where the image edge is located is divided into a type of region to be blurred based on image features, then the edge can be downsampled and then the blurring can be achieved. The specific blurring level data can be the downsampling level corresponding to each pixel. If the region with dense texture is divided into a type of region to be blurred based on image features, then the region with dense texture can be downsampled and the blurring can be achieved. The specific blurring level data can be the downsampling parameters.

[0041] In this embodiment, since the second image and the first image have the same image content, theoretically, the deblurred image obtained after deblurring the second image should be quite consistent with the first image. However, this is an ideal situation. In reality, there will be differences between the two. In this embodiment, model training can be performed based on this difference. The specific training process is described in step S205 below.

[0042] S205. The second image is deblurred according to the blur level data corresponding to the first image by the image processing model to obtain a deblurred image. A loss value is calculated based on the first image and the deblurred image. At least one of the following in the image processing model is adjusted according to the loss value: preset region division parameters and deblurring parameters, so as to deblur the image by the adjusted image processing model.

[0043] In this embodiment, during the training process, the image processing model deblurs the second image based on the blur level data corresponding to the first image. Specifically, the blur level data can be used as the training input sample of the image processing model, and the first image corresponding to the blur level data can be used as the learning target to train the image processing model. Since the image processing model is trained based on the blur level data, the image processing model can deblur the second image based on the blur level data.

[0044] In this embodiment, since the second image and the first image have the same image content, theoretically, the deblurred image obtained after deblurring the second image should be quite consistent with the first image. Based on this, in this embodiment, a loss value can be calculated based on the first image and the deblurred image, and the preset region segmentation parameters and deblurring parameters in the image processing model can be adjusted according to the loss value, so that the deblurred image output by the adjusted image processing model is as close as possible to the first image.

[0045] According to the training scheme of the image processing model provided in the embodiments of this application, during the training process, at least two types of regions to be blurred can be obtained by dividing the first (high-definition) image of the sample image group according to image features and preset region division parameters. Then, corresponding blur degree data is constructed for each of the at least two types of regions to be blurred according to the blur parameters corresponding to each type. The image processing model can then perform deblurring on the second (low-definition) image of the sample image group based on the blur degree data. Furthermore, the image processing model is adjusted based on the loss value calculated from the deblurred image and the first (high-definition) image. This allows the image processing model to learn different types of image region division methods and their corresponding deblurring schemes. Therefore, when performing deblurring through the trained image processing model, the image can be divided into different types of regions, and deblurring can be performed on each type of region according to the corresponding deblurring parameters. This greatly improves the quality of deblurring. Moreover, during the deblurring process, the trained parameters can be directly used, eliminating the need for computationally intensive operations such as estimating the blur kernel. This reduces the computational load on the electronic devices deploying the image processing model and improves the real-time performance of deblurring.

[0046] See Figure 2B The diagram illustrates a usage scenario of this embodiment.

[0047] See Figure 2B The upper part is a schematic diagram of training the image processing model. The first image can be input into the image processing model, which divides the first image into multiple types of regions to be blurred according to preset region division parameters. It can also construct blur degree data corresponding to each type of region to be blurred according to the image features of each type of region to be blurred (A, B, C) and their corresponding blur parameters through a preset construction program, thereby obtaining the blur degree data corresponding to the first image. Then, the image processing model can continue to deblur the second image according to the blur degree data and output a deblurred image. The loss value is calculated based on the deblurred image, and the preset region division parameters, deblurring parameters and other parameters in the image processing model are adjusted according to the loss value.

[0048] See Figure 2B The lower part is a schematic diagram of using an image processing model. After the image processing model is deployed on an electronic device, the electronic device can input the image to be processed. The image processing model can divide the image to be processed into regions according to the adjusted preset region division parameters, and perform deblurring processing on different categories of regions according to the adjusted deblurring parameters to obtain an image with enhanced image quality.

[0049] It should also be noted that since the deblurring process using the image processing model mainly relies on the adjusted preset region segmentation parameters and deblurring parameters, the image processing model itself does not need to perform complex data processing. Therefore, any lightweight model can be used, as long as it can perform image feature extraction and deblurring, which reduces the performance requirements of the electronic devices that deploy the image processing model.

[0050] The solution provided in this embodiment can be executed by any electronic device with data processing capabilities, including but not limited to servers, computers, mobile phones, etc.

[0051] Figure 3A This is a flowchart illustrating another image processing model training method provided in this application embodiment. In this embodiment, image features include edge features, and preset region segmentation parameters include image edge segmentation thresholds, such as... Figure 3A As shown, it includes:

[0052] S301. Obtain a sample image group, which includes a first image and a second image with the same image content, wherein the clarity of the first image is greater than that of the second image.

[0053] The specific implementation method of this step can be referred to step S201 in the above embodiment, and will not be repeated here.

[0054] S302. The first image is processed by an image processing model to extract image features, which may specifically include edge features.

[0055] The specific implementation method of this step can be referred to step S202 in the above embodiment, and will not be repeated here.

[0056] S303. Using an image processing model, the image is divided according to edge features and image edge segmentation thresholds to obtain the first region corresponding to the image edge.

[0057] Optionally, in this embodiment, the image features include texture features, the edge features are the gradient values ​​of the texture features, the image edge segmentation threshold is the image gradient threshold, and the image is segmented according to the edge features and the image edge segmentation threshold by the image processing model to obtain the first region corresponding to the image edge, including: segmenting the region with a texture gradient greater than the texture gradient threshold into the first region by the image processing model.

[0058] For ease of explanation, the first image will be referred to as HR, meaning high-resolution image, and the second image as LR, meaning low-resolution image. Specifically, it can be:

[0059] 1. Obtain L by applying the Laplace gradient to HR (i.e., the first image):

[0060]

[0061] Wherein, HR is the first image. It is a convolution symbol. is the convolution kernel, and L is the image gradient.

[0062] It should be noted that Laplace gradient is one method for finding image edges. In other implementations of this embodiment, other methods can also be used to find image edges, which are also within the scope of protection of this application.

[0063] For ease of handling, the letter L can be split into two parts: Lwhite (white border) and Lblack (black border).

[0064] Lblack=Clip(L,0.0,10000.0)>0

[0065] Lwhite=-Clip(L,-10000.0,0.0)>0

[0066] In the above formula, the function Clip(A, B, C) means to cut off the value of A to the range [b, c], and 10000.0 in the above formula is the gradient threshold of the Laplace gradient.

[0067] After separating the white border (Lwhite) and the black border (Lblack), the first region and mask can be determined for each of the two borders, and then they can be merged.

[0068] 2. The strength of image edges is measured using the Laplace gradient result to find the first region map0(x,y), where x,y are pixel spatial coordinates. Specifically, the first region can be an image edge and its neighborhood. In this embodiment, the image edge segmentation threshold can be the Laplace intensity threshold lapmax, thus the determined first region can be:

[0069]

[0070] Pixels whose image gradient is greater than or equal to lapmax are set to 1, and pixels whose gradient is less than lapmax are set to 0. The positions of the pixels set to 1 are the image edges and their neighborhoods.

[0071] To further improve the accuracy of the identified first region, map0(x, y) can be iterated multiple times:

[0072] map0(x,y)=Dilate(map0,kernel_dilate)

[0073] Here, `dilate()` indicates that a dilation operation is performed on `map0`, and `kernel_dilate` is the window function for the dilation operation. Specifically, the dilation operation involves using the window function to sweep across `map0` and assigning the maximum value of the pixels in the area covered by the window function to the anchor pixel of the window function, thereby expanding the region. In this embodiment, the size of the window function can be 7*7.

[0074] In addition, if we separate the white border Lwhite and the black border Lblack, we can determine the corresponding map0(x, y) for the white border Lwhite and the black border Lblack respectively.

[0075] S304. Divide the area other than the first area into the second area.

[0076] If we separate the white border Lwhite and the black border Lblack, and determine the corresponding map0(x,y) for the white border Lwhite and the black border Lblack respectively, then we can also remove the corresponding map0(x,y) for the white border Lwhite and the black border Lblack respectively to obtain the second region.

[0077] Optionally, in this embodiment, if the preset region division parameters include a texture density threshold, the method further includes: dividing regions with texture density greater than the texture density threshold into a third region. Correspondingly, step S304 may also include: dividing regions other than the first and third regions into a second region. The second region can be other regions besides image edges and texture-dense regions. The third region generally refers to small and dense texture regions such as grass, salt and pepper, foliage, and pollen. For such textures, it is difficult to learn a good deblurring effect. The result of deblurring is often that the region condenses into a "block or worm-like structure," so a special deblurring method is required. Therefore, in this embodiment, it is divided into a third region separately and targeted blurring is performed, so that the image processing model learns a deblurring scheme with a better effect.

[0078] Optionally, in this embodiment, the third region can be identified by a pre-trained texture-dense region recognition network. The texture-dense region recognition network is trained by the following method: obtaining a third image that does not contain texture-dense regions and a fourth image that only contains texture-dense regions; performing image fusion processing on the third image and the fourth image to obtain a dense sample image, and using the region in the dense sample image that belongs to the fourth image as an image annotation; and training the texture-dense region recognition network based on the dense sample image and the image annotation.

[0079] Therefore, the texture density region recognition network trained can identify regions in the first image whose texture density is greater than the texture density threshold and label them as the third region.

[0080] Specifically, in this embodiment, an image set A that does not contain any dense texture / weak texture regions can be pre-collected as the third image and an image set B that contains only pure dense texture / weak texture regions can be pre-collected as the fourth image. For any pair of images in the third and fourth image sets, imgA and imgB, a 0-1 map "mapR" can be randomly generated based on imgA and imgB. The values ​​in mapR are 0, 1, 0 indicating that the image content comes from imgA, and 1 indicating that the image content comes from imgB.

[0081] Then, based on imgA, imgB, and mapR, the dense sample image “imgA(1-mapR)+imgB*mapR” input to the texture dense region recognition network can be determined, and mapR can be used as the annotation image, which annotates the regions in the dense sample image that belong to the fourth image.

[0082] The above method can directly generate dense sample images with dense textures and automatically generate image annotations without manual processing, thus saving training costs.

[0083] In this embodiment, the texture-dense region recognition network can be any image semantic segmentation network, preferably Unet.

[0084] For the first region corresponding to the image edge, a main strategy can be adopted: "the higher the laplace intensity of a pixel, the stronger the blur (and the greater the corresponding deblur intensity)." That is, pixels with higher image gradients in the first region correspond to larger blur weights, causing the image processing model to strengthen blurring at the image edges. Conversely, when using the image processing model for deblurring, it can enhance sharpening at the image edges. Therefore, based on the first blur window parameter group in the blur parameters, the blur weights corresponding to each pixel in the first region can be determined, ensuring that pixels with higher image gradients in the first region have larger blur weights, resulting in stronger deblurring.

[0085] However, if the pixels in the neighborhood of the image edge are deblurred using the same deblurring parameters as the image edge, distortion or defects may easily occur. Therefore, in this application, when performing deblurring processing through the image processing model, the deblurring processing of the neighborhood of the image edge can be skipped to avoid distortion or defects.

[0086] Therefore, in this embodiment, before executing step S305, it may further include: adding a blur mask to the first image, wherein the blur mask is used to achieve at least one of the following: reducing the blur degree of pixels adjacent to the pixel position corresponding to the image edge in the first region, while maintaining the blur degree of pixels corresponding to the image edge; reducing the blur degree of pixel positions in the first image where the image gradient is greater than the image gradient threshold; and reducing the blur degree of pixel positions in the first image where the pixel value is greater than the pixel threshold.

[0087] Firstly, during training, a blur mask is used to reduce the blurriness of pixels adjacent to the image edge in the first region, while maintaining the blurriness of pixels corresponding to the image edge. When using an image processing model for deblurring, an image mask can be generated to reduce the deblurring of pixels adjacent to the image edge in the image to be processed, while maintaining the deblurring of pixels corresponding to the image edge. This avoids deblurring the neighborhood of the image edge (i.e., pixels adjacent to the image edge), preventing over-sharpening or white edges, and improving image quality. It should be noted that the deblurring process for the first region can specifically be a sharpening process.

[0088] Secondly, during training, blurring masks can reduce the blurring of pixels in the first image whose image gradient is greater than the image gradient threshold. When using an image processing model for deblurring, an image mask can be generated to reduce the deblurring of pixels in the image to be processed that are greater than the image gradient threshold, thereby preventing distortion or defects caused by excessive image gradients and improving image quality.

[0089] Thirdly, during training, blurring masks can reduce the blurring of pixels with pixel values ​​greater than the pixel threshold in the first image. When using an image processing model for deblurring, an image mask can be generated to reduce the deblurring of pixels with pixel values ​​greater than the pixel threshold, preventing distortion or defects caused by excessively large pixel values ​​and improving image quality.

[0090] The following examples illustrate specific methods of suppression. It should be noted that any other schemes that can achieve the above-mentioned suppression effect are within the scope of protection of this application.

[0091] In this embodiment, regarding the first aspect mentioned above, the pixel position corresponding to the image edge can be determined by the image gradient. Of course, other schemes for determining the pixel position corresponding to the image edge are all within the protection scope of this application.

[0092] Specifically, in this embodiment, a blur mask can be used to reduce the blurring of pixels adjacent to the image edge in the first region while maintaining the blurring of pixels corresponding to the image edge. To more accurately determine the pixel positions and gradient values ​​to be reduced, thereby improving the masking effect, the image gradient values ​​of the first region can be segmented according to preset segmentation parameters. Bilateral filtering is performed on the image region corresponding to each segment to reduce the image gradient adjacent to the image edge while maintaining the image gradient of pixels at the image edge. Based on the filtering results corresponding to multiple image gradient segments, a blur mask is determined and added to the first region to reduce the blurring of pixels adjacent to the image edge while maintaining the blurring of pixels corresponding to the image edge.

[0093] In this embodiment, the preset segmentation parameter can be thr1. Then, the method of performing bilateral filtering on the image region corresponding to each image gradient segment to reduce the image gradient of pixels corresponding to image edges in the image region, while maintaining the image gradient of pixels other than image edges, can be as follows:

[0094] 1. Determine the image region map1(x, y) corresponding to the gradient of the image segment according to the segmentation parameter thr1:

[0095]

[0096] In map1, the regions where the value is 0 are the regions that need to be filtered bilaterally.

[0097] 2. Determine the regions that need filtering, excluding the first region (map2):

[0098] map2 = 1 - map1 + map1(1 - map0);

[0099] Here, map0 is the first region determined in the above steps.

[0100] 3. Determine the first region (map10) that requires bilateral filtering:

[0101] map10=1-map2=1-(1-map1+map1(1-map0))=map1*map2.

[0102] 4. Perform bilateral filtering on map10 to obtain the filtered region map11:

[0103] map11=bilateralFilter(map10, kernelSizeBF, sigmaColor, sigmaSpace)

[0104] Here, bilateralFilter() is the bilateral filtering function, kernelsizebf is the filtering window for bilateral filtering, sigmacolor is the color difference range for bilateral filtering, and sigmaSpace is the sigma value in the filtering window. The larger the sigma value, the more pixels participate in the filtering calculation.

[0105] 5. Smooth the bilaterally filtered map11 to obtain map12, map12b, and map13:

[0106]

[0107] map12b=GaussianBlur(map12, kernelSizeGB, sigmaGB)

[0108] map13=map11(1-map12b)+map12*map12b

[0109] In this diagram, map12 is the part of map11 that needs to be smoothed based on the threshold thr2; map12b is the filtered image obtained by applying Gaussian filtering to map12; gaussianbilr() is the Gaussian filtering function; kernelsizeGB is the filtering window for Gaussian filtering; sigmaGB is the radius range for Gaussian filtering; and map13 is nap13 obtained by merging the Gaussian-filtered region map12b and the un-Gaussian-filtered region in map11.

[0110] Of course, it should be noted that in other implementations of this application, step 5 may not be executed, which is also within the scope of protection of this application. If step 5 is not executed, map13 in step 6 can be replaced with map11.

[0111] 6. After merging the first region that has already undergone bilateral filtering with the regions (excluding the first region) that need filtering (map2), we obtain map3, which has undergone bilateral filtering on the image region corresponding to the gradient of that image segment:

[0112] map3 = map2 + map13.

[0113] Then, map3 can be merged into the image gradient L to update L, i.e., L = L * map3. The aforementioned thr1 and thr2 can also be updated, and bilateral filtering can be performed again. Assigning different values ​​to thr1 enables multi-segment filtering.

[0114] It should be noted that if the white edge Lwhite and the black edge Lblack are separated, then the white edge Lwhite and the black edge Lblack can be segmented separately, and bilateral filtering can be performed on the image region corresponding to the gradient of each segment. Of course, the following is only an example of segmenting the white edge Lwhite and the black edge Lblack and performing bilateral filtering. The specific number of segments and the parameters thr1 and thr2 for bilateral filtering of each segment are not limited in this embodiment.

[0115] Let the process of steps 1-6 above be denoted as the function Func_suppression: map3, map0 = Func_suppression(X),

[0116] The process of segmenting the white border Lwhite and performing bilateral filtering on the image region corresponding to the gradient of each segment can be described as follows:

[0117] map3white,map0white=Func_suppression(Lwhite,thr1=30,thr2=0.3)

[0118] Lwhite = Lwhite * map3white

[0119] map3white,map0white=Func_suppression(Lwhite,thr1=20,thr2=0.3)

[0120] Lwhite = Lwhite * map3white

[0121] map3white,map0white=Func_suppression(Lwhite,thr1=10,thr2=0.3)

[0122] Lwhite = Lwhite * map3white

[0123] map3white,map0white=Func_suppression(Lwhite,thr1=5,thr2=0.4)

[0124] The above formula sets four segments: (thr1 = 30, thr2 = 0.3), (thr1 = 20, thr2 = 0.3), (thr1 = 10, thr2 = 0.3), and (thr1 = 5, thr2 = 0.3). Lwhite = Lwhite * map3. The white flag updates the white border Lwhite with the map3 of the region where bilateral filtering has been completed, until the last bilateral filtering is completed.

[0125] The process of segmenting the black border Lblack and performing bilateral filtering on the image region corresponding to the gradient of each segment can be described as follows:

[0126] map3black,map0black=Func_suppression(Lblack,thr1=20,thr2=0.3)

[0127] Lblack = Lblack * map3black

[0128] map3black,map0black=Func_suppression(Lblack,thr1=10,thr2=0.3)

[0129] Lblack = Lblack * map3black

[0130] map3black,map0black=Func_suppression(Lblack,thr1=5,thr2=0.4)

[0131] The above formula sets three segments: (thr1 = 20, thr2 = 0.3), (thr1 = 10, thr2 = 0.3), and (thr1 = 5, thr2 = 0.3). Lblack = Lblack * map3. The black flag updates the region map3 (where bilateral filtering has been completed) to the black border Lblack until the last bilateral filtering is completed.

[0132] After bilateral filtering, the white edge Lwhite and the black edge Lblack can be merged to obtain an image lap that reduces the image gradient of pixels adjacent to the image edge in the first region while maintaining the image gradient of the pixels at the image edge.

[0133] Lap=Lblack*map3black+Lwhite*map3white

[0134] The resulting blurred mask can be further optimized by bilateral filtering. The formula for bilateral filtering of the image lap is as follows:

[0135] Lap=bilateralFilter(Lap, kernelSizeBF, sigmaColor, sigmaSpace)

[0136] Here, bilateralFilter() is the bilateral filtering function, kernelsizeBF is the filtering window for bilateral filtering, sigmaColor is the color difference range for bilateral filtering, and sigmaSpace is the sigma value in the filtering window. The larger the sigma value, the more pixels participate in the filtering calculation.

[0137] Regarding the second aspect mentioned above, in order to reduce the blurriness of pixels in the first image where the image gradient is greater than the image gradient threshold, the obtained blur mask can be segmented and mapped according to multiple preset gradient thresholds lapTHR, lapmax, and lapmax1 to obtain lap1:

[0138]

[0139] Here, `lap` is the image obtained through the above steps, or it can be the image obtained after calculating the image gradient of the first image. `x` and `y` are used to indicate the pixel position, and `lap(x, y)` is used to indicate the Laplacian gradient value corresponding to that pixel position in the blurred mask. Multiple preset gradient thresholds `lapTHR`, `lapmax`, and `lapmax1` can be adjusted later according to the loss function. `Clip(A, B, C)` means truncating the value of A to the range [b, c], that is, truncating the value of "Lapmax + Lapmax1 - lap(x, y)" to [0, Lapmax]. Through piecewise mapping, the image gradient values ​​of pixels in the image `lap` after adding the blurred mask that have an image gradient greater than the image gradient threshold can be reduced, thereby reducing its blurriness.

[0140] Specifically, in the piecewise function described above, by setting the pixels whose image gradient is less than the preset gradient threshold lapTHR to 0 (corresponding to the first segmented region in lap1 above), the blur weight of the pixels whose image gradient is less than the preset gradient threshold lapTHR can be reset to 0, that is, they are not blurred. When performing deblurring processing through the trained image processing model, pixels whose image gradient is less than the preset gradient threshold lapTHR are also not deblurred, so as to prevent imperfections caused by over-sharpening in areas with relatively flat image gradients.

[0141] Regarding the third aspect mentioned above, in order to reduce the blurriness of pixels with pixel values ​​greater than a pixel threshold in the first image, a segmented mapping can be performed according to a preset pixel value `highpixelsuppressjoint` to obtain the gain:

[0142]

[0143] in,

[0144]

[0145] Furthermore, gain can be merged with lap1, i.e., Lap2 = Lap1 * gain, to obtain a new image lap2 after lowering the pixel values ​​of the pixel positions in the first image where the pixel value is greater than the pixel threshold.

[0146] Specifically, the final blur mask can be obtained from lap1 using the following formula:

[0147] mask=[GaussianBlur(Lap1, kernelSizeGB, sigmaGB) / Lapmax]gamma

[0148] `gaussianbilr()` is the Gaussian filtering function, `kernelsizeGB` is the filtering window for Gaussian filtering, and `sigmaGB` is the radius range for Gaussian filtering. The above formula calculates the quotient of the Gaussian filtering result and the gradient threshold `lapmax` corresponding to the image edge after applying Gaussian filtering to `lap1`, and then calculates the power of `gamma` to obtain the final blurred mask used. It should be noted that the image gradient `lap1` of the first image, which reduces the image gradient of pixels with a gradient greater than the image gradient threshold, can be replaced with "image `lap` obtained by reducing the image gradient of pixels corresponding to the image edge in the first region and maintaining the image gradient of pixels other than the image edge" or "image `lap2` obtained by reducing the image gradient of pixels with a pixel value greater than the pixel threshold in the first image", both of which are within the protection scope of this application.

[0149] After determining and adding a blur mask to the first image, at least two types of regions to be blurred can be blurred according to the blur parameters corresponding to each type of region to be blurred, so as to obtain the blurred image corresponding to the first image, and then step S305 can be executed.

[0150] S305. For the first region corresponding to the image edge, determine the blur weight corresponding to each pixel in the first region based on the first blur window parameter group in the blur parameters and the image value of each pixel in the first region, and construct a first blur window based on the blur weight corresponding to each pixel in the first region.

[0151] It should be noted that this step can redefine the first region. The lapmax used to determine the first region can be the same as or different from the aforementioned lapmax. It should also be noted that if the first region has been redefined, the second region and the remaining regions should also be redefined accordingly.

[0152] Specifically, experiments show that in a blur window (when blurring an image, for each pixel in the image, a ksize x ksiize sub-region of the image is drawn with the pixel as the center, where ksize is an odd number. Each pixel in the sub-region corresponds to a coefficient, i.e., a blur weight, and these coefficients form a ksize x ksize matrix, i.e., the blur window), the larger the blur weight assigned to the pixels surrounding the center pixel that have a large pixel difference from the center pixel, the more the network's learning of deblurring ability will be biased towards highlighting details and sharpness. However, the network effect is also prone to artifacts. For the first region, the difference between the surrounding pixels and the center pixel is already large overall, the need for deblurring is weak, and the need to suppress distortion is strong. Therefore, the blur window for the first region (called the first blur window) is: assigning relatively smaller blur weights to the pixels surrounding the center pixel that have a large pixel difference from the center pixel. Based on this idea, the first blur window can be constructed. It should be noted that any first blur window constructed based on the above idea is within the protection scope of this application, and this embodiment does not limit the specific first blur window. For example, the first blur window can be a bilateral filtering window (but is not limited to) to obtain the new pixel values ​​of each pixel in the first region based on the bilateral filtering result. Specifically, the weights expressed by the elements in the matrix of the first blur window can be inversely proportional to |HR(x,y)-HR(i,j)|.

[0153] The first blurred window constructed can then be:

[0154]

[0155] in,

[0156] Win1 is the first blur window (ksize1 x ksize1) for the pixel at coordinates x, y. i and j are the pixel coordinates in the first blur window corresponding to the pixel at coordinates x and y. HR(x, y) is the pixel value of the pixel at position x, y in the first image. HR(i, j) is the pixel value of the pixel at position i, j in the first image. sigma1 is a parameter used to control the blur intensity. STD is used to control the weight of pixels with large differences from the center point pixel when blurring. When STD is fixed, the pixel with the larger difference from the center point pixel has a smaller blur weight.

[0157] After normalizing Win1, we can obtain:

[0158] win1(i,j,x,y)=win1(i,j,x,y) / sum_i,j(win1(i,j,x,y)). Normalization is the process of summing the values ​​of win1 according to the coordinates ij and then normalizing them.

[0159] S306. For the second region, based on the second blur window parameter group in the blur parameters and the pixel value of each pixel in the second region, determine the blur weight corresponding to each pixel in the second region, and construct a second blur window based on the blur weight corresponding to each pixel in the second region.

[0160] In this embodiment, the second region excludes image edges and their neighborhoods. Therefore, unlike the first region, when blurring the second region, it is necessary to increase the blur weight of pixels with a large difference from the center pixel, while decreasing the blur weight of pixels with an excessively large difference from the center pixel. Based on this, a pixel threshold `divide` can be set to distinguish between pixels with an excessively large difference from the center pixel and pixels with a large difference. For pixels with a large difference, the higher the pixel difference between the pixel value and the center pixel, the greater the corresponding blur weight, i.e., increasing the blur weight of pixels with a large difference from the center pixel; for pixels with an excessively large difference, the higher the pixel difference between the pixel value and the center pixel, the smaller the corresponding blur weight, i.e., decreasing the blur weight of pixels with an excessively large difference from the center pixel. Based on this idea, a second blur window can be constructed. It should be noted that any second blur window constructed based on the above idea is within the scope of protection of this application, and this embodiment does not limit the specific blur window.

[0161] For example, the constructed second blurred window in Win3 can be:

[0162]

[0163] in, Win3 is a second blur window ("blur window") with coordinates x, y, defined by ksize3 x ksize3. i and j are the pixel coordinates within the second blur window corresponding to the xy pixel. HR(x, y) is the pixel value of the pixel at position x, y in the first image; HR(i, j) is the pixel value of the pixel at position i, j in the first image. sigma2 is a parameter used to control the blur intensity. By setting the pixel threshold divide, pixel values ​​can be divided into two segments: |HR(x, y) - HR(i, j)| ≤ divide represents pixels with a large difference from the center pixel, and |HR(x, y) - HR(i, j)| > divide represents pixels with an excessively large difference from the center pixel. Win3 ensures that when pixel values ​​are less than divide, the blur weight increases monotonically, thus raising the blur weight of pixels with a large difference from the center pixel; conversely, it decreases monotonically, thus lowering the blur weight of pixels with an excessively large difference from the center pixel. The blur weight can be represented by the exponent of the second "e" in the window function. Lines A and B need to be solved.

[0164] 1. To ensure that Win3 is continuous across adjacent pixels, i.e., the window function is continuous when the pixel value is the pixel threshold `didide`, it is necessary to ensure that the exponents of the two `e` values ​​are equal when |HR(x,y)-HR(i,j)|=divide.

[0165]

[0166] 2. When |HR(x,y)-HR(i,j)| is large enough, the absolute value of win3(i,j,x,y) approaches 0.

[0167] Based on points 1 and 2 above, lineA and lineB can be obtained.

[0168] After normalizing Win3, we can obtain:

[0169] win3(i,j,x,y) = win3(i,j,x,y) / sum_i,j(win3(i,j,x,y)). Normalization is the process of summing the values ​​of win3 according to the coordinates ij and then normalizing them.

[0170] In addition, if there is a third region corresponding to dense texture, a blur window for image enhancement processing can be constructed for the third region based on the image enhancement parameters in the blur parameters and the pixel values ​​of each pixel in the third region, so as to enhance the third region through the blur window.

[0171] This blurred window can be used in Win2:

[0172]

[0173] in, Win2 is the second blur window "blur window" with coordinates x and y, which is ksize3 x ksize3. i and j are the pixel coordinates in the second blur window corresponding to the pixel with coordinates x and y. HR(x, y) is the pixel value of the pixel with position x and y in the first image. HR(i, j) is the pixel value of the pixel with position i and j in the first image. STD1 is a parameter used to control the intensity of the blur.

[0174] After normalizing Win2, we can obtain:

[0175] Win2(i,j,x,y) = win2(i,j,x,y) / sum_i,j(win2(i,j,x,y)). Normalization is the process of summing the values ​​of win2 according to the coordinates ij and then normalizing them.

[0176] S307. The first image is blurred using an image processing model based on at least the first blur window, the second blur window, and the first image to obtain a blurred image. The image processing model is trained based on the blurred image. The second image is deblurred based on the trained image processing model to obtain a deblurred image.

[0177] Optionally, in this embodiment, if a third region exists, the blurring process of the first image in step S307 includes: blurring the first region in the first image according to a first blur window; blurring the second region in the first image according to a second blur window; and enhancing the third region in the first image, wherein the enhanced third region can be used for subsequent HR. enh The first image, after blurring and image enhancement, is downsampled and encoded to obtain the blurred image.

[0178] In this embodiment, for convenience, the first region is labeled as HR*mapS, the third region is labeled as HR*(1-mapS)*mapR, and the second region is labeled as HR*(1-mapS)*(1–mapR).

[0179] After combining and blurring the first blurred window (win1), the second blurred window (win3), and all maps (e.g., mapS, mapR) and masks, we can obtain:

[0180]

[0181] in It is the convolution symbol.

[0182] Additionally, the image enhancement processing performed on the third region corresponding to the dense texture using the Win2 convolution window can be denoted as HR. enh :

[0183] Where p is the intensity of the enhancement applied to the second region corresponding to the dense texture in HR;

[0184] The first image HR, after processing, can be obtained as follows:

[0185] HR=HR*[mapS+(1-mapS)*(1-mapR)]+HR enh *(1-mapS)*mapR.

[0186] For dense textures, HRB inherits from HR and does not perform blurring; HR itself sharpens and enhances dense textures.

[0187] Specifically, "deblurring the second image using an image processing model" can be achieved by using the blur level data corresponding to the first image as the training input of the image processing model, using the first image as the training ground truth of the image processing model, blurring the first image to obtain a blurred image (i.e., blur level data) as the training input data of the network, training to obtain an image processing model, and then using the trained image processing model to deblurr the second image to obtain a deblurred image.

[0188] S308. Calculate the loss value based on the first image and the deblurred image, and adjust at least one of the following in the image processing model based on the loss value: preset region segmentation parameters and deblurring parameters, so as to perform deblurring processing on the image through the adjusted image processing model.

[0189] In this embodiment, to ensure that the deblurring process in the image processing model focuses on image edges, the weight of the loss value corresponding to the image edges can be increased when calculating the loss value. For example, the power of the loss component corresponding to the edge region can be increased; the larger the power, the greater the weight.

[0190] Specifically, the loss value can be:

[0191] Loss = |HR - netout| W

[0192] HR is the first image, netout is the blurred image, and the power can be denoted as W. W can be specifically defined as:

[0193] W = max{|Laplace(HR)|, |Laplace(UPSAMPLE(LR))|}, where laplace(HR) is the Laplacian gradient calculated on the first image, and laplace(UPSAMPLE(LR)) is the Laplacian gradient calculated after upsampling the second image. The maximum value between the two is taken as W, which can ensure that W is larger for the image edges.

[0194] To control the range of values ​​for w, a linear transformation can be performed on W:

[0195]

[0196] Specifically, the loss value is calculated based on the first image and the deblurred image. Based on the loss value, at least one of the following in the image processing model is adjusted: preset region segmentation parameters and deblurring parameters. When the adjusted image processing model deblurs the image, it can regenerate the blur level data corresponding to the first image, thereby retraining the image processing model to achieve the purpose of "adjusting at least one of the following in the model: preset region segmentation parameters and deblurring parameters". Furthermore, the loss function used to train the image processing model can also use the aforementioned loss value.

[0197] It should be noted that in other implementations of this application, the segmentation can be based on other image features.

[0198] In one feasible approach, if the image features include semantic features and the preset region segmentation parameters include an image semantic segmentation threshold, the first image is segmented into regions based on the image features according to the preset region segmentation parameters to obtain at least two types of regions to be blurred, including: using an image processing model, the region in the image corresponding to the semantic features is divided into a fourth region according to the image semantic segmentation threshold; and at least the regions other than the fourth region are divided into a fifth region.

[0199] For example, the semantic features can specifically be facial features. Based on facial region information, the facial features and cheek skin areas can be divided into different types of regions to be blurred as a fourth region. Then, according to beautification schemes, corresponding blur windows are designed for different regions, and relative deblurring intensity is set for different regions, thereby obtaining a better experience of facial clarity. In addition, semantic features can also be object region information, plant region information, animal region information, etc., which are not limited in this embodiment.

[0200] In another feasible approach, if the image features include color features and texture features, and the preset region segmentation parameters include texture intensity threshold and color gradient threshold, the first image is segmented into regions according to the image features based on the preset region segmentation parameters to obtain at least two types of regions to be blurred, including: using an image processing model, regions with texture gradients less than the texture gradient threshold and color gradients less than the preset color gradient are classified as the sixth region; and regions other than the sixth region are classified as the seventh region.

[0201] For example, in a football sports scenario, the area corresponding to the lawn can be determined based on dense texture information and color (green) information, and corresponding blur windows or convolution windows can be designed accordingly.

[0202] In another feasible approach, if the image features include depth features, the preset region segmentation parameters include a first depth threshold and a second depth threshold, where the first depth threshold is greater than the second depth threshold. Based on the preset region segmentation parameters, the first image is segmented according to the image features to obtain at least two types of regions to be blurred, including: using an image processing model, regions greater than the first depth threshold are segmented into an eighth region, regions less than the second depth threshold are segmented into a ninth region; and regions other than the eighth and ninth regions are segmented into a tenth region.

[0203] In this embodiment, blurring, sharpening, or even softening can be performed on the foreground area to blunt the sharp edges in the foreground area, making the scene more visually three-dimensional.

[0204] See Figure 3B This diagram illustrates a usage scenario of an embodiment of this application. Referring to the upper part, it shows a schematic diagram of training a deep learning network. The deep learning network can be any lightweight deep learning network capable of image deblurring, such as... Figure 3B As shown, the specific process of constructing training data for training deep learning networks includes:

[0205] Regional blurring process: Divide the image into different types of regions to be blurred, and construct blur windows corresponding to each region. The blur windows can be at least one of the first blur window, the second blur window, and the convolution window mentioned above. Then, blur the first image according to the blur windows to obtain the blurred image.

[0206] Downsampling process: The blurred image is downsampled to make its pixels consistent with the second image.

[0207] Encoding process: Used to encode the downsampled image to simulate the image or video transmission process.

[0208] The image obtained from the image encoding is a blurred image used for comparison with the second image. Based on the comparison result, it is determined whether to use the blurred window constructed during the regional blurring process as data for training.

[0209] When training a deep learning network, the network can be configured to deblur the second image based on the blur window constructed during the region blurring process, resulting in a deblurred image. The loss value is then calculated based on the deblurred image and the first image, and the parameters in the deep learning network are adjusted according to the loss value. Specifically, the preset region segmentation parameters and deblurring parameters used in the deep learning network can be adjusted.

[0210] It should be noted that in this embodiment, the downsampling factor is 1 / f, where f is the super-resolution factor set by the user when performing image quality enhancement; crf is the image compression sharpness parameter set by the user, and the compression method can be ffmpeg compression; in addition, the input to the regional blurring module can be a video, and the compressed encoding is several downsampled video frames.

[0211] See Figure 3B As shown in the lower part, the deep learning network may include: a decoding module corresponding to the encoding process, used to decode the image to be processed, and a descratching module may be loaded in the decoding module; an upsampling module corresponding to the downsampling process, used to upsample the decoded image to be processed (super-resolution); and a region-based deblurring module corresponding to the region-based blurring process, used to divide the upsampled image according to preset region division parameters to obtain at least two types of regions to be blurred, and to perform deblurring processing using corresponding deblurring parameters, and output an image with enhanced image quality. Thus, the deep learning network can simultaneously achieve the effects of image decoding, descratching, super-resolution, and deblurring, greatly improving image quality.

[0212] Figure 4 A flowchart illustrating a deployment method for an image processing model provided in an embodiment of this application is shown in the figure, which includes:

[0213] S401. Send the image to be processed so that multiple pre-trained image processing models can be used to deblur the image and obtain multiple deblurred images.

[0214] The image parameters of multiple image processing models differ, including preset region segmentation parameters and / or deblurring parameters.

[0215] Multiple image processing models can process the image to be processed through the following steps, and output deblurred images respectively:

[0216] Image features are extracted from the image to be processed using an image processing model.

[0217] The image processing model is used to divide the image to be processed into regions based on the degree of matching between the image features and the preset region division parameters, resulting in at least two types of regions to be processed.

[0218] Using the image processing model, at least two types of processing regions are deblurred according to the deblurring parameters corresponding to each type of region to be processed, resulting in a deblurred image.

[0219] In this embodiment, the user provides the materials to be processed (image samples / videos) to the cloud, and selects the super-resolution ratio f requirement, encoding quality (crf) requirement, and the model of the end-side device used.

[0220] The server can pre-train and store m batches of image processing models with different structures / complexities in the cloud, based on different super-resolution ratios f and different CRF ranges. Each batch contains n_m image processing models with identical structures, and the sharpening effect increases progressively. The training method can refer to the above embodiment. From the batches of f and CRF selected by the user, the most complex image processing model batch that the user's device can handle is selected. Each image processing model processes the user-provided image, and the resulting deblurred image is uploaded to the cloud for the user to choose from.

[0221] S402, Display multiple deblurred images to the user.

[0222] S403. Send a deblurred image selected by the user from multiple deblurred images, receive and deploy the image processing model corresponding to the selected deblurred image, and perform deblurring processing on the image through the image processing model.

[0223] If the user is satisfied with the result of the i-th network out of n_m networks, and clicks the button to confirm, the on-premises deployment of that network will be completed.

[0224] Optionally, in this embodiment, the method further includes: sending two deblurred images selected by the user from multiple deblurred images, performing a step transition on the two sets of image parameters corresponding to the image processing models of the two selected deblurred images respectively, obtaining multiple sets of image parameters and their corresponding image processing models, performing deblurring processing on the images to be processed through the newly obtained multiple image processing models, obtaining multiple deblurred images, and continuing to execute the above step S402, that is, displaying multiple deblurred images to the user.

[0225] If the user selects the i-th and j-th networks out of n_m networks, and believes that the desired sharpness effect is between these two networks, after receiving the message, the cloud will subdivide the model parameters (such as the region segmentation parameters and deblurring parameters) of the i-th to j-th networks by K subdivisions, train K networks using the above method, and output the deblurred image again through the k networks, and continue to execute the above step S402 for the user to choose and judge.

[0226] If the user selects one of the K deblurred images and clicks the confirmation button, the network is deployed to complete the service; otherwise, if the user selects two of the K steps, K1 steps are transitioned, and the two deblurred images selected by the user from the multiple deblurred images are sent, and step S402 above is executed again. This process iterates continuously as long as the user wants to further subdivide the images.

[0227] If the end user selects K*n stepped networks, and switches between them in real time according to the user's sharpness needs in different scenarios, then these K*n sets of network parameters are retained, and real-time switching is achieved through asynchronous loading.

[0228] The solution provided in this embodiment can be deployed for communication between cloud and on-premises users. Based on the user's customized needs, a deep learning super-resolution network is generated in the cloud, transmitted to the on-premises user terminal (mobile / computer), and automatically configured for user use. It achieves agile and high-quality adaptation to customer scenarios and device models, demonstrating high practicality.

[0229] See Figure 5 Another embodiment of this application also provides an image processing method, including:

[0230] S501. Determine the image to be processed.

[0231] In this embodiment, the image to be processed can be any image. For example, the image to be processed can be a video frame from a real-time video conference, a video frame from an online live broadcast, a video frame from a recorded video, an image stored in the device, etc.

[0232] When the image to be processed is any video frame, in general, video frames from real-time video conferences are decoded and played by an audio / video decoder. In this embodiment, after the audio / video decoder decodes the video frame, it can be input to the image processing model, so that the image processing model can determine the image to be processed. Alternatively, the image processing model can embed an audio / video decoder. After an electronic device, such as a mobile phone or computer, obtains the video stream to be played, it can input the video stream to the image processing model, so that the image processing model decodes the video stream to obtain the image to be processed, and then continues to execute step S502.

[0233] Specifically, in this embodiment, step S501 may include: decoding the received video stream data to obtain video frame images, and upsampling the video frame images to obtain the image to be processed. The parameters used in the decoding and upsampling processes in this embodiment can be determined according to the description in the training method of the above embodiments, and will not be repeated here.

[0234] S502. The image features are extracted from the image to be processed using an image processing model to obtain image features.

[0235] The specific implementation method of this step can be found in the embodiment of the training method of the above image processing model, and will not be repeated here.

[0236] In this embodiment, the image features can be any image features, as long as they can be used to determine the areas that need to be focused on deblurring during image quality enhancement. For example, image features can be image edge features, image color features, texture features, depth features, semantic features, etc. The areas that need to be focused on deblurring during image quality enhancement can be, for example, the edges of objects in the image, faces in the image, etc.

[0237] S503. Using the image processing model, the image to be processed is divided into regions according to the degree of matching between the image features and the preset region division parameters, to obtain at least two types of regions to be processed.

[0238] The specific implementation method of this step can be found in the embodiment of the training method of the above image processing model, and will not be repeated here.

[0239] Optionally, in this embodiment, if the image features include edge features and the preset region division parameters include an image edge division threshold, step S503 includes: using the image processing model to divide the image according to the edge features and the image edge division threshold to obtain a first region corresponding to the image edge; and dividing the region other than the first region into a second region.

[0240] Optionally, in this embodiment, the image features include texture features, and the edge features are the gradient values ​​of the texture features; the preset region division parameters include a texture density threshold, and the image edge division threshold is an image gradient threshold; the step of dividing the first image into regions using the image processing model according to the image features and their corresponding preset region division parameters to obtain at least two types of regions to be processed includes: dividing regions with texture gradients greater than the texture gradient threshold into the first region using the image processing model; the method further includes: dividing regions with texture densities greater than the texture density threshold into a third region; the step of dividing regions other than the first region into a second region includes: dividing regions other than the first region and the third region into the second region.

[0241] Optionally, in this embodiment, dividing the region with a texture density greater than the texture density threshold into a third region includes: identifying the third region through a pre-trained texture-dense region recognition network, wherein the texture-dense region recognition network is trained by: obtaining a third image that does not contain texture-dense regions and a fourth image that only contains texture-dense regions; performing image fusion processing on the third image and the fourth image to obtain a dense sample image, and using the region in the dense sample image that belongs to the fourth image as an image annotation; and training the texture-dense region recognition network based on the dense sample image and the image annotation.

[0242] Optionally, in this embodiment, before performing deblurring processing on at least two types of regions according to the deblurring parameters corresponding to each type of region to be processed, the method further includes:

[0243] A blur mask is added to the first image, wherein the blur mask is used to achieve at least one of the following: reducing the deblurring degree of pixels adjacent to the pixel position corresponding to the image edge in the first region, while maintaining the deblurring degree of pixels corresponding to the image edge; reducing the deblurring degree of pixel positions in the first image where the image gradient is greater than the image gradient threshold; and reducing the deblurring degree of pixel positions in the first image where the pixel value is greater than the pixel threshold.

[0244] Optionally, in this embodiment, adding a blur mask to the first image includes: segmenting the image gradient value of the first region according to preset segmentation parameters; performing bilateral filtering on the image region corresponding to each segment of the image gradient to reduce the image gradient of pixels in the image region corresponding to the image edge, while maintaining the image gradient of pixels other than the image edge; determining and adding a blur mask to the first region based on the filtering results corresponding to the multiple segments of the image gradient, so as to reduce the deblurring degree of pixels in the first region adjacent to the pixel position corresponding to the image edge, while maintaining the deblurring degree of pixels corresponding to the image edge.

[0245] Optionally, in this embodiment, if the image features include semantic features, and the preset region division parameters include an image semantic segmentation threshold, the step of dividing the first image into regions based on the image features and their corresponding preset region division parameters to obtain at least two types of regions to be processed includes: using the image processing model, according to the image semantic segmentation threshold, dividing the region in the image corresponding to the semantic features into a fourth region; and dividing at least the regions other than the fourth region into a fifth region.

[0246] Optionally, in this embodiment, if the image features include color features and texture features, and the preset region division parameters include a texture intensity threshold and a color gradient threshold, the process of dividing the first image into regions based on the image features and their corresponding preset region division parameters to obtain at least two types of regions to be processed includes: using the image processing model to divide regions with texture gradients less than the texture gradient threshold and color gradients less than a preset color gradient into a sixth region; and dividing regions other than the sixth region into a seventh region.

[0247] Optionally, in this embodiment, if the image features include depth features, the preset region division parameters include a first depth threshold and a second depth threshold, wherein the first depth threshold is greater than the second depth threshold, and the first image is divided into regions according to the image features and their corresponding preset region division parameters to obtain at least two types of regions to be processed, including: using the image processing model to divide regions greater than the first depth threshold into an eighth region, and regions less than the second depth threshold into a ninth region; and dividing regions other than the eighth and ninth regions into a tenth region.

[0248] S504. Using the image processing model, at least two types of processing regions are deblurred according to the deblurring parameters corresponding to each type of processing region, to obtain the deblurred image corresponding to the image to be processed.

[0249] Optionally, in this embodiment, the step of deblurring at least two types of regions to be processed according to the deblurring parameters corresponding to each type of region to be processed, to obtain a deblurred image corresponding to the first image, includes: for the first region corresponding to the image edge, determining the blur weight corresponding to each pixel of the first region according to the first blur window parameter group in the deblurring parameters, and constructing a first deblurring window according to the blur weight corresponding to each pixel of the first region, wherein the pixel with a higher image gradient in the first region corresponds to a smaller blur weight; for the second region, determining the blur weight corresponding to each pixel of the second region according to the second blur window parameter group in the deblurring parameters, and constructing a second deblurring window according to the blur weight corresponding to each pixel of the second region, wherein the pixel with a higher pixel value in the second region corresponds to a larger blur weight; and deblurring the first image according to at least the first deblurring window and the second deblurring window to obtain the deblurred image, wherein the blur weight is proportional to the degree of deblurring.

[0250] Optionally, in this embodiment, if the third region exists, the step of deblurring the first image at least according to the first deblurring window and the second deblurring window to obtain the deblurred image includes: deblurring the first region in the first image according to the first deblurring window; deblurring the second region in the first image according to the second deblurring window; performing image enhancement processing on the third region in the first image; and downsampling and encoding the first image after deblurring and image enhancement processing to obtain the deblurred image.

[0251] The following examples illustrate this solution through specific usage scenarios.

[0252] Figure 6 This is a schematic diagram illustrating a usage scenario according to an embodiment of this application, such as... Figure 6 As shown, the usage scenario may include user equipment 1, communication network, and user equipment 2.

[0253] In some embodiments, the communication network can be any suitable combination of one or more wired and / or wireless networks. The communication network may include a transmission intermediary; for example, user equipment 1 can transmit data to a service node via the communication network, and the service node can then forward the data to user equipment 2 via the communication network. For example, communication network 104 can include any one or more of the following: the Internet, intranet, wide area network (WAN), local area network (LAN), wireless network, digital subscriber line (DSL) network, frame relay network, asynchronous transfer mode (ATM) network, virtual private network (VPN), and / or any other suitable communication network. Electronic devices can be connected to the communication network via one or more communication links (e.g., communication links), and the communication network can be linked to a cloud server via one or more communication links (e.g., communication links). Communication links can be any communication link suitable for transmitting data between electronic devices and cloud servers, such as network links, dial-up links, wireless links, hardwired links, any other suitable communication links, or any suitable combination of such links.

[0254] User equipment 1 can be any electronic device capable of running programs, capturing images, encoding images, and transmitting bitstreams. In some embodiments, user equipment 1 may include any suitable type of device. For example, in some embodiments, user equipment 1 may include mobile devices, tablet computers, laptop computers, desktop computers, wearable computers, cloud computing terminals, game consoles, media players, vehicle entertainment systems, and / or any other suitable type of electronic device.

[0255] User device 2 can be any electronic device capable of video decoding and playback. Similar to user device 1, user device 2 can include mobile devices, tablet computers, laptop computers, desktop computers, wearable computers, cloud computing terminals, game consoles, media players, vehicle entertainment systems, and / or any other suitable type of electronic device.

[0256] In a live streaming scenario, user device 1 can be the broadcaster's electronic device, and user device 2 can be a player's electronic device. When user device 1 broadcasts, it generates a video stream and sends it to user device 2 via a communication network. Upon receiving the video stream from the broadcaster, user device 2 inputs it into an image processing model. The model decodes the video stream to obtain the image to be processed, and then performs upsampling (super-resolution) and artifact removal on the image. Afterward, it can be processed... Figure 5 The content of the embodiment is deblurred to obtain a live broadcast with enhanced image quality.

[0257] Reference Figure 7The diagram shows a structural schematic of an electronic device according to this application. The specific embodiments of this application do not limit the specific implementation of the electronic device.

[0258] like Figure 7 As shown, the electronic device may include: a processor 702, a communications interface 704, a memory 706, and a communications bus 708.

[0259] in:

[0260] The processor 702, communication interface 704, and memory 706 communicate with each other via communication bus 708.

[0261] Communication interface 704 is used to communicate with other electronic devices or servers.

[0262] The processor 702 is used to execute program 710, specifically to execute the relevant steps in the above-described image processing model training method embodiment.

[0263] Specifically, program 710 may include program code that includes computer operation instructions.

[0264] The processor 702 may be a CPU, an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The smart device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0265] Memory 706 is used to store program 710. Memory 706 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0266] Specifically, program 710 can be used to cause processor 702 to perform the operation corresponding to the method described in any of the foregoing multiple method embodiments.

[0267] The specific implementation of each step in program 710 can be found in the corresponding steps and units described in the above method embodiments, and has corresponding beneficial effects, which will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.

[0268] This application also provides a computer storage medium storing computer instructions that instruct a computing device to perform an operation corresponding to any of the methods in the above-described multiple method embodiments.

[0269] This application also provides a computer program product, including computer instructions that instruct a computing device to perform an operation corresponding to any of the methods in the above-described multiple method embodiments.

[0270] It should be noted that, depending on the implementation needs, the various components / steps described in the embodiments of this application can be broken down into more components / steps, or two or more components / steps or parts of the operation of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of this application.

[0271] The methods described in the embodiments of this application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code downloaded over a network that is originally stored in a remote recording medium or a non-transitory machine-readable medium and will be stored in a local recording medium. Thus, the methods described herein can be processed by software stored on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., RAM, ROM, flash memory, etc.) capable of storing or receiving software or computer code that, when accessed and executed by the computer, processor, or hardware, implements the methods described herein. Furthermore, when a general-purpose computer accesses code used to implement the methods shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for executing the methods shown herein.

[0272] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0273] The above embodiments are only used to illustrate the embodiments of this application, and are not intended to limit the embodiments of this application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of this application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of this application, and the patent protection scope of the embodiments of this application should be defined by the claims.

Claims

1. A method for training an image processing model, comprising: obtaining a sample image group, the sample image group comprising a first image and a second image with the same image content, the first image having a higher definition than the second image; extracting image features of the first image by an image processing model to obtain image features; performing region division on the first image according to the image features and corresponding preset region division parameters by the image processing model to obtain at least two types of regions to be blurred; constructing blur degree data corresponding to each type of region to be blurred according to the image features of each type of region to be blurred and corresponding blur parameters to obtain blur degree data corresponding to the first image, the blur degree data comprising a first blur window and a second blur window; performing deblurring processing on the second image according to the blur degree data corresponding to the first image by the image processing model to obtain a deblurred image, calculating a loss value according to the first image and the deblurred image, and adjusting at least one of the preset region division parameters and deblurring parameters in the image processing model according to the loss value, so as to perform deblurring processing on an image by the adjusted image processing model; wherein the construction of the blur degree data corresponding to each type of region to be blurred according to the image features of each type of region to be blurred and corresponding blur parameters to obtain the blur degree data corresponding to the first image comprises: for the first region corresponding to an image edge, determining blur weights corresponding to each pixel of the first region according to a first blur window parameter set in the blur parameters and image values of each pixel of the first region, and constructing the first blur window according to the blur weights corresponding to each pixel of the first region; for the second region, determining blur weights corresponding to each pixel of the second region according to a second blur window parameter set in the blur parameters and pixel values of each pixel of the second region, and constructing the second blur window according to the blur weights corresponding to each pixel of the second region.

2. The method of claim 1, wherein, If the image features comprise edge features, the preset region division parameters comprise an image edge division threshold, and the region division on the first image according to the image features and corresponding preset region division parameters by the image processing model to obtain at least two types of regions to be blurred comprises: performing image division according to the edge features and the image edge division threshold by the image processing model to obtain a first region corresponding to an image edge; dividing regions other than the first region into a second region.

3. The method of claim 2, wherein, The image features comprise texture features, the edge features are gradient values of the texture features, the preset region division parameters comprise a texture density threshold, and the image edge division threshold is an image gradient threshold; the region division on the first image according to the image features and corresponding preset region division parameters by the image processing model to obtain at least two types of regions to be blurred comprises: The image processing model divides a region with a texture gradient greater than the texture gradient threshold value into the first region; The method further comprises: dividing a region with a texture density greater than the texture density threshold value into a third region; The dividing of the region other than the first region into the second region comprises: dividing a region other than the first region and the third region into the second region.

4. The method of claim 3, wherein, The dividing of the region with the texture density greater than the texture density threshold value into the third region comprises: identifying the third region by a pre-trained texture dense region identification network, and the texture dense region identification network is trained by the following method: obtaining a third image not containing a texture dense region and a fourth image containing only a texture dense region; performing image fusion processing on the third image and the fourth image to obtain a dense sample image, and regarding a region belonging to the fourth image in the dense sample image as image annotation; training a texture dense region identification network according to the dense sample image and the image annotation.

5. The method of claim 1, wherein, If the third region exists, the blur degree data further comprises a convolution window, and the constructing of the blur degree data corresponding to each type of region to be blurred according to the image features of each type of region to be blurred and the blur parameters corresponding thereto respectively to obtain the blur degree data corresponding to the first image further comprises: for the third region in the first image, constructing the convolution window used for image enhancement processing according to the image enhancement parameter in the blur parameter and the pixel value of each pixel of the third region.

6. The method of claim 1, wherein, Before the constructing of the blur degree data corresponding to each type of region to be blurred according to the image features of each type of region to be blurred and the blur parameters corresponding thereto respectively to obtain the blur degree data corresponding to the first image, the method further comprises: adding a blur mask to the first image, wherein the blur mask is used to at least one of: lowering the blur degree of the pixels adjacent to the pixel position corresponding to the image edge in the first region and maintaining the blur degree of the pixels corresponding to the image edge; lowering the blur degree of the pixels with an image gradient greater than an image gradient threshold value in the first image; lowering the blur degree of the pixels with a pixel value greater than a pixel threshold value in the first image.

7. The method of claim 6, wherein, The determining of the blur mask added to the first image comprises: segmenting the image gradient value of the first region according to a preset segmentation parameter; respectively performing bilateral filtering on the image region corresponding to each segment of image gradient to lower the blur degree of the pixels adjacent to the pixel position corresponding to the image edge in the segment of image gradient and maintain the blur degree of the pixels corresponding to the image edge; determining and adding a blur mask to the first region according to the filtering results corresponding to the multiple segments of image gradient to lower the blur degree of the pixels adjacent to the pixel position corresponding to the image edge in the first region and maintain the blur degree of the pixels corresponding to the image edge.

8. The method of any one of claims 1-7, wherein, If the image feature comprises a semantic feature, the preset region division parameter comprises an image semantic segmentation threshold, and the region division of the first image according to the image feature and the corresponding preset region division parameter to obtain at least two types of regions to be blurred comprises: dividing, by the image processing model, a region corresponding to the semantic feature in the image into a fourth region according to the image semantic segmentation threshold; and dividing a region other than the fourth region into a fifth region. Alternatively, if the image feature comprises a color feature and a texture feature, the preset region division parameter comprises a texture intensity threshold and a color gradient threshold, and the region division of the first image according to the image feature and the corresponding preset region division parameter to obtain at least two types of regions to be blurred comprises: dividing, by the image processing model, a region with a texture gradient less than the texture gradient threshold and a color gradient less than a preset color gradient into a sixth region; and dividing a region other than the sixth region into a seventh region. Alternatively, if the image feature comprises a depth of field feature, the preset region division parameter comprises a first depth of field threshold and a second depth of field threshold, the first depth of field threshold is greater than the second depth of field threshold, and the region division of the first image according to the image feature and the corresponding preset region division parameter to obtain at least two types of regions to be blurred comprises: dividing, by the image processing model, a region greater than the first depth of field threshold into an eighth region, and a region less than the second depth of field threshold into a ninth region; and dividing a region other than the eighth region and the ninth region into a tenth region.

9. A deployment method of an image processing model, comprising: sending a to-be-processed image to a plurality of image processing models pre-trained by the training method of the image processing model in any one of claims 1-8 to perform image feature extraction on the to-be-processed image, to obtain an image feature; and performing region division on the to-be-processed image according to a matching degree of the image feature and a preset region division parameter by the image processing model, to obtain at least two types of to-be-processed regions; and performing deblurring processing on the at least two types of to-be-processed regions respectively according to corresponding deblurring parameters of each type of to-be-processed region by the image processing model, to obtain deblurred images output by the plurality of image processing models respectively, wherein the image parameters of the plurality of image processing models are different, and the image parameters comprise the preset region division parameter and / or the deblurring parameter; showing the plurality of deblurred images to a user; sending a selected deblurred image from the plurality of deblurred images by the user, to receive and deploy the image processing model corresponding to the selected deblurred image, to perform deblurring processing by the image processing model.

10. The method of claim 9, wherein, The method further comprises: The sending user selects two deblurring images from the plurality of deblurring images, so as to perform ladder transition on two sets of image parameters corresponding to the image processing model corresponding to the two selected deblurring images respectively, to obtain a plurality of sets of image parameters and corresponding image processing models, perform deblurring processing on the to-be-processed image through the newly obtained plurality of image processing models, to obtain a plurality of deblurring images, and continue to perform the step of showing the user the plurality of deblurring images.

11. An image processing method, comprising: determining a to-be-processed image; performing image feature extraction on the to-be-processed image through an image processing model to obtain image features, the image processing model being obtained through the training method of the image processing model according to any one of claims 1-8; performing region division on the to-be-processed image according to a matching degree between the image features and preset region division parameters through the image processing model, to obtain at least two types of to-be-processed regions; performing deblurring processing on at least two types of to-be-processed regions respectively through the image processing model according to deblurring parameters corresponding to each type of to-be-processed region, to obtain a deblurring image corresponding to the to-be-processed image.

12. An electronic device comprising: a processor, a memory, a communication interface and a communication bus, the processor, the memory and the communication interface completing communication with each other through the communication bus; the memory is used to store at least one executable instruction, and the executable instruction makes the processor execute the operation corresponding to the training method of the image processing model according to any one of claims 1-11.

13. A computer storage medium having a computer program stored thereon, the program being executed by a processor to implement the method according to any one of claims 1-11.

Images