Model training and image processing method and device, storage medium and program product

Abstract

The embodiment of the invention provides a model training and image processing method and device, a storage medium and a program product. In the embodiment of the invention, the target degradation parameter is obtained by optimizing the initial degradation parameter with the target that the texture loss after image degradation processing is smaller than or equal to the set texture loss threshold value, so that the target degradation parameter is obtained according to the target degradation parameter obtained through optimization. The texture loss of the input image of the model obtained by carrying out degradation processing on the target image is smaller than the texture loss threshold compared with the target image, so that a moderately degraded training sample is generated, and the degraded image still keeps some texture information to provide reliable context clues for the model; and texture reasoning can be carried out based on real observation instead of generating some artifacts, so that texture enhancement is effectively realized.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a model training and image processing method, device, storage medium and program product. Background Technology

[0002] With the rapid development of mobile internet and e-commerce, images and videos have become core media for product display. In e-commerce applications, some product display images or videos suffer from low resolution and blurry details, seriously affecting the visual presentation of products and the user's shopping experience.

[0003] Image restoration can recover or enhance high-frequency details, texture structure, and overall sharpness from degraded, low-quality images, bringing them close to or reaching professional-grade shooting quality. Neural network models are widely used in image restoration due to their ability to automatically perform image restoration. However, existing image restoration models often fail to restore images with texture details and a lack of realism. Therefore, there is an urgent need for an image restoration model that can achieve image texture enhancement. Summary of the Invention

[0004] This application provides a model training and image processing method, device, storage medium, and program product to provide an image restoration model capable of enhancing image texture.

[0005] This application provides a model training method, including: Acquire a target image and a first degraded image; the first degraded image is obtained by degrading the target image according to the target degradation parameters; the target degradation parameters are obtained by optimizing the initial degradation parameters with the goal of the texture loss after image degradation being less than or equal to a set texture loss threshold; Using the first degraded image as the input image and the target image as the reference image, the initial image restoration model is trained to obtain the target image restoration model.

[0006] This application also provides an image processing method, including: Obtain the image to be processed; The image to be processed is restored using a target image restoration model to obtain the target image; The target image is output; wherein the target image restoration model is obtained by training the model using the above-described model training method.

[0007] This application also provides an electronic device, including: a memory and a processor; wherein the memory is used to store computer programs; The processor is coupled to the memory and is used to execute the computer program to perform the steps in the above-described model training method and / or image processing method.

[0008] This application also provides a computer-readable storage medium storing computer instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps in the above-described model training method and / or image processing method.

[0009] This application also provides a computer program product, including a computer program that, when executed by one or more processors, causes the one or more processors to perform the steps in the above-described model training method and / or image processing method.

[0010] In this embodiment, the target degradation parameter is obtained by optimizing the initial degradation parameter with the goal of the texture loss after image degradation processing being less than or equal to a set texture loss threshold. Therefore, based on the optimized target degradation parameter, the input image of the model obtained by degradation processing the target image has a texture loss less than the aforementioned texture loss threshold compared to the target image, thereby generating moderately degraded training samples. This allows the degraded image to still retain some texture information, providing reliable contextual clues for the model, enabling it to perform texture inference based on real observations rather than generating artifacts, thus effectively achieving texture enhancement. Attached Figure Description

[0011] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 A schematic flowchart illustrating the model training method provided in this application embodiment; Figure 2 A schematic diagram illustrating the process of obtaining a target image from an image database, provided for an embodiment of this application; Figure 3 A schematic diagram of a target image obtained from an image database, provided as an embodiment of this application; Figure 4 A schematic diagram illustrating the process of generating a target image using a cross-modal generation model, as provided in an embodiment of this application; Figure 5 A schematic diagram of a target image generated using a cross-modal generation model, provided in an embodiment of this application; Figure 6 A comparison diagram showing the effect of initial degradation parameters and target degradation parameters applied to the same image in an embodiment of this application; Figure 7A schematic flowchart of the image processing method provided in the embodiments of this application; Figure 8 This is a comparison of the effects of an image restoration model trained using an image degraded by initial degradation parameters and an image restoration model trained using a target degradation parameter on the same image, as provided in the embodiments of this application. Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0013] It should be noted that, in the case of user information involved in the embodiments of this application, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0014] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0015] It should be noted that the same reference numerals in the following figures and embodiments denote the same object or the same step. Therefore, once an object or step is defined in one figure or embodiment, it does not need to be discussed further in subsequent figures and embodiments.

[0016] Figure 1 This is a flowchart illustrating the model training method provided in an embodiment of this application. This model training method is primarily used to train an image inpainting model capable of enhancing image texture. For example... Figure 1 As shown, the method mainly includes the following steps: 101. Obtain the target image and the first degraded image. The first degraded image is obtained by degrading the target image according to the target degradation parameters; the target degradation parameters are obtained by adjusting the initial degradation parameters with the goal of the texture loss after image degradation being less than or equal to the set texture loss threshold.

[0017] 102. Using multiple first degraded images as input images and multiple target images as reference images, train the initial image restoration model to obtain the target image restoration model.

[0018] In this embodiment, the image inpainting model refers to a model that performs texture restoration or enhancement on low-quality images to obtain high-quality images. To train an image inpainting model capable of texture enhancement, it is necessary to obtain high-quality images and their corresponding low-quality images. These high-quality and low-quality images form image sample pairs, which are then used to train the initial image inpainting model. In this embodiment, the target image is the high-quality image.

[0019] In this embodiment, the target image refers to an image with high resolution and rich texture. For example, the resolution is greater than or equal to a set resolution threshold, such as 8K or 16K. Generally, there are multiple target images. Multiple means two or more (including two). In this embodiment, the specific method of obtaining the target images in step 101 is not limited. In some embodiments, multiple target images can be obtained from an image database. The image database can be an open-source image library or a self-built image library, etc.

[0020] Specifically, such as Figure 2 As shown in step 201, multiple candidate images are obtained from the image database. The candidate images can be images of their original size from the image database, or partial images extracted from the original images in the image database, i.e., image slices. Optionally, a set number of image slices can be randomly selected from the image database as multiple candidate images. The set number is greater than or equal to 2. For example, the set number can be several thousand or tens of thousands.

[0021] To obtain images with rich texture, the pixel mean and pixel variance of candidate images can be calculated, and so on. Figure 2 As shown in step 202, multiple candidate images are obtained from multiple candidate images whose pixel mean falls within a set pixel value range and whose pixel variance is greater than or equal to a set pixel variance threshold. In this embodiment, for ease of description and differentiation, the candidate images obtained from the image database are defined as first candidate images; and the candidate images in the first candidate images whose pixel mean falls within a set pixel value range and whose pixel variance is greater than or equal to a set pixel variance threshold are defined as second candidate images. The number of second candidate images is less than or equal to the number of first candidate images. The set pixel value range can be [50, 200]; the pixel variance threshold can be 30.

[0022] Since pixel variance reflects the dispersion of pixel grayscale values ​​(or color values) in an image, images with large pixel variance have drastic pixel value changes and contain more texture details. Therefore, pixel variance can be used to filter out images rich in texture details. Pixel mean represents the overall brightness level of an image. Relying solely on variance may select some high-contrast but unusable images, such as overexposed images where most areas are white (mean close to 255). Although edges may cause a large pixel variance, texture details are lost. Similarly, underexposed images where most areas are black (mean close to 0) may also have a large pixel variance due to a few bright spots, but the overall image is unclear. Setting a range for the pixel mean can exclude images that are too dark or too bright, ensuring that the selected images have moderate brightness and reducing the probability of situations where textures cannot be effectively utilized under extreme lighting conditions. Therefore, this embodiment uses a high pixel variance and a reasonable range for the pixel mean to perform dual filtering on multiple first candidate images, effectively eliminating low-quality images that are blurry, smooth, overexposed, or underexposed, thereby improving the overall texture quality and usability of the selected second candidate images.

[0023] To further refine the texture quality of the selected images, such as... Figure 2 As shown in step 203, multiple second candidate images are blurred to obtain multiple blurred images. Each second candidate image corresponds one-to-one with a single blurred image. In some embodiments, Gaussian blurring can be applied to the multiple second candidate images. Specifically, a two-dimensional Gaussian blur kernel of a set size and a set Gaussian standard deviation can be used to perform Gaussian blurring on the multiple second candidate images to obtain multiple blurred images. The set Gaussian kernel size can be 7*7, and the set Gaussian standard deviation can be 3.3, etc., but is not limited to these.

[0024] The more texture an image has, the more image information is lost after blurring. Therefore, in step 204, the image differences between the multiple second candidate images and their respective blurred images can be calculated based on the pixel values ​​of the multiple second candidate images and the pixel values ​​of the multiple blurred images. In this embodiment, the specific implementation of calculating the image differences between the multiple second candidate images and their respective blurred images is not limited.

[0025] In some embodiments, the image difference between a second candidate image and its corresponding blurred image can be represented using the Peak Signal-to-Noise Ratio (PSNR). A higher PSNR indicates lower distortion in the blurred image, meaning a smaller image difference between the second candidate image and its corresponding blurred image. Accordingly, the PSNR between multiple second candidate images and their respective blurred images can be calculated based on the pixel values ​​of multiple second candidate images and multiple blurred images. Specifically, for any second candidate image A, the mean squared error (MSM) between second candidate image A and blurred image B can be calculated based on the pixel values ​​of second candidate image A and its corresponding blurred image B. Furthermore, the maximum pixel value (MAX) supported by the number of bits used to represent pixel brightness or color intensity in the color channel of second candidate image A can be determined. For example, if the number of bits is 8, meaning second candidate image A is an 8-bit image, then the maximum pixel value supported by 8 bits is 255. For example, if the binary bit width is 16 bits and the second candidate image A is a 16-bit image, then the maximum pixel value supported by 16 bits is 65535. Furthermore, the PSNR between the second candidate image A and the blurred image B can be determined based on the mean square error of pixels between the second candidate image A and the blurred image B, and the maximum pixel value supported by the binary bit width. The formula for PSNR can be expressed as: (1).

[0026] In Equation (1), MAX represents the maximum pixel value supported by the number of binary bits representing the pixel brightness or color intensity of the color channel of the second candidate image; MSM represents the pixel mean square error between the second candidate image and its corresponding blurred image.

[0027] In other embodiments, the image difference between the second candidate image and its corresponding blurred image can be represented using the Structural Similarity Index (SSIM). A larger SSIM indicates a closer similarity between the blurred image and the second candidate image, with less distortion in the blurred image, meaning a smaller image difference. Accordingly, the SSIM between multiple second candidate images and their respective blurred images is calculated based on the pixel values ​​of multiple second candidate images and multiple blurred images. Specifically, for any second candidate image A, a sliding window can be used to slide over both the second candidate image A and its corresponding blurred image B, obtaining a local image a of the second candidate image A and a local image b of the blurred image B within the current window. Based on the pixel values ​​of local image a and local image b, the pixel mean and pixel variance of local image a and local image b are calculated. Then, based on the pixel mean and pixel variance of local image a and local image b, the pixel covariance between local image a and local image b can be calculated. Then, based on the pixel mean and pixel variance of local image a, the pixel mean and pixel variance of local image b, and the pixel covariance between local image a and local image b, the SSIM of the second candidate image A and the blurred image B within the current window can be determined; the SSIM of the second candidate image A and the blurred image B is obtained by averaging the SSIMs across all sliding windows. The SSIM of the second candidate image A and the blurred image B within each window can be expressed as: (2).

[0028] In equation (2), and Let a represent the mean and variance of pixel a in the local image 'a' of the second candidate image within the window, respectively. and Let $\mathbf$ and $\mathbf$ represent the mean and variance of pixels in the local image $b$ of the blurred image corresponding to the second candidate image within the window, respectively. This represents the pixel covariance between local image a and local image b. and It is a constant.

[0029] In some other embodiments, the PSNR and SSIM between the second candidate image and its corresponding blurred image can be used to jointly measure the image difference between the second candidate image and its corresponding blurred image. Accordingly, the PSNR and SSIM between multiple second candidate images and their respective blurred images can be calculated based on the pixel values ​​of multiple second candidate images and multiple blurred images. The specific implementation of calculating PSNR and SSIM can be found in the relevant content of the foregoing embodiments, and will not be repeated here.

[0030] After calculating the image difference between the second candidate image and its corresponding blurred image, as follows: Figure 2 As shown in step 205, a second candidate image whose image difference is greater than or equal to a set difference threshold can be selected from multiple second candidate images as the target image.

[0031] In this embodiment, since the richer the texture of an image, the more image information is lost after blurring, the second candidate image selected by the pixel mean and pixel variance can be blurred to obtain the blurred image corresponding to the second candidate image. Then, the image with greater image difference can be selected by the image difference between the blurred image and the second candidate image, so that the selected image has richer texture information.

[0032] In an embodiment where PSNR is used to measure the image difference between a second candidate image and its corresponding blurred image, a second candidate image with a PSNR less than or equal to a set PSNR threshold can be selected from multiple second candidate images as the target image. The PSNR threshold can be 30, etc.

[0033] In an embodiment where SSIM is used to measure the image difference between a second candidate image and its corresponding blurred image, the second candidate image whose corresponding SSIM is less than or equal to a set SSIM threshold can be selected from a plurality of second candidate images as the target image. The SSIM threshold can be 0.82, etc.

[0034] In an embodiment that uses PSNR and SSIM to measure the image difference between a second candidate image and its corresponding blurred image, the second candidate image whose corresponding SSIM is less than or equal to a set SSIM threshold can be selected from multiple second candidate images as the target image. This embodiment combines PSNR and SSIM between the second candidate image and its corresponding blurred image to measure the image difference, which can avoid misjudgment caused by a single indicator due to distortion type or illumination changes, and improve the accuracy of the selection. The target image selected from the image database is as follows: Figure 3As shown, it has rich texture information.

[0035] Since image databases may cover a limited number of scenes or objects, images of some real-world scenes and objects are not included in them. To improve the richness of scenes or objects that image restoration models can learn from and provide them with more abundant learning materials, in some embodiments of this application, a cross-modal generation model can be used to generate target images based on descriptive information from images or videos.

[0036] A modality can be considered a type of data, and can include text, images, audio, and video, among others. These different types of data are considered different modalities. Cross-modal generative models can generate data of another modality using prompts from one modality. For example, a cross-modal generative model could be a text-to-image model, which uses natural language descriptions (speech or text, etc.) of an image as prompts to generate the corresponding image. Another example is a text-to-video model, which uses natural language descriptions (speech or text, etc.) of a video as prompts to generate the corresponding video.

[0037] In this embodiment, natural language description information of the image or video to be generated can be obtained. Specifically, such as... Figure 4 As shown in "Generating Natural Language Description Information Using a Language Model," a language model can be used to generate natural language description information for images and / or videos to be generated. In this embodiment, the topic of the image and / or video description to be generated differs from the topic of the image descriptions in the image database. Thus, the images and videos generated using the natural language description information of the images and / or videos to be generated can supplement the image materials in the image database, improving the diversity and richness of the sample image materials. The language model can be an open-source language model or a self-developed language model.

[0038] To generate high-quality images or videos, the image quality requirements for the generated images and / or videos can be added to the natural language description information to obtain target prompts. These image quality requirements can include positive prompts. Positive prompts describe the positive quality requirements for the generated images and / or videos. For example, positive prompts could be "photorealistic, 8K resolution, highly detailed, ultrahigh-definition, close-up, ultra-detailed" (photorealistic, 8K resolution, highly detailed, ultrahigh-definition, close-up, ultra-detailed). The image quality requirements can include both positive and negative prompts. Negative prompts remind the cross-modal generation model to avoid generating images with specific quality requirements. Negative keywords could include "low resolution, blurry, pixelated, low quality, poor detail, noise, artifacts, warped shaped, fakelighting, cartoon style, oil painting." Correspondingly, such as... Figure 4 As shown in "Add Positive Hints" and "Add Negative Hints", positive and negative hints can be added to the free-language description information to obtain the target hints.

[0039] After obtaining the target cue words, such as Figure 4 As shown in the "Cross-modal generative model," this model can be used to generate images and / or videos based on target prompts. When generating images, the cross-modal generative model can be a text-to-image model; when generating videos, it can be a text-to-video model. The text-to-image and text-to-video models can be open-source or self-developed.

[0040] Furthermore, the target image can be obtained from video frames of the image and / or video to be generated. In this embodiment, a cross-modal generation model can be used to generate images and / or videos with themes different from those in the image database. This can supplement the image materials in the image database, improve the diversity and richness of the sample image materials, and thus help expand the coverage of the trained image restoration model. On the other hand, using a language model to generate natural language description information for the image and / or video to be generated, and optimizing the prompt words for this natural language description information, can improve the accuracy of the prompt words and provide accurate guidance information for the cross-modal generation model to generate high-quality images and / or videos.

[0041] In some embodiments, video frames of the image and / or video to be generated can be determined as target images. In some embodiments, to further improve the quality of sample images, video frames of the image and / or video to be generated can be filtered to obtain target images. Specifically, such as Figure 4 As shown in the "Image Quality Evaluation Model" and "Aesthetic Evaluation Model," these models can be used to evaluate the image quality and aesthetics of video frames in the generated image and / or video, respectively, to obtain the image quality score and aesthetic score of the video frames. Then, as... Figure 4 As indicated by "≥ Image Quality Score Threshold" and "≥ Aesthetic Score Threshold," images with image quality scores greater than or equal to the set image quality score threshold and aesthetic scores greater than or equal to the set aesthetic score threshold are selected from video frames in the image and / or video to be generated as target images. In this embodiment, the image quality evaluation model and aesthetic evaluation model can be open-source neural network models or self-developed neural network models. This embodiment performs image quality and aesthetic screening on video frames of images and / or videos generated by the cross-modal generation model. The selected target images have higher image quality and can provide high-quality training reference images for the image restoration model, thereby helping to improve the quality of the images restored by the image restoration model. Figure 5 This diagram illustrates the target images selected for quality and aesthetic filtering of video frames from images and / or videos generated by a cross-modal generative model. This method can supplement the material in an image database.

[0042] In some embodiments of this application, images and / or videos can be generated directly using a cross-modal generation model without relying on an image database. In this embodiment, a language model is used to generate natural language description information for the images and / or videos to be generated. In this embodiment, the images and / or videos to be generated are not subject-specific, meaning their themes are not limited to differing from those of images in the image database. Then, image quality requirements for the images and / or videos to be generated can be added to the natural language description information to obtain target prompts. The cross-modal generation model is then used to generate the images and / or videos to be generated based on the target prompts. Finally, the target image can be obtained from the video frames of the images and / or videos to be generated. For the specific implementation of each step in this embodiment, please refer to the relevant content of the foregoing embodiments, which will not be repeated here. In this embodiment, the natural language description information for the images and / or videos to be generated can be generated using a language model, and the prompts in this natural language description information can be optimized to improve the accuracy of the prompts, providing accurate guidance information for the cross-modal generation model to generate high-quality images and / or videos.

[0043] The methods for acquiring the target image shown in the foregoing embodiments are merely illustrative and do not constitute a limitation. The specific process for acquiring the target image can be pre-executed before model training begins, and the obtained target image can be stored in a persistent storage medium, such as a database or the disk of the electronic device executing the model training method. Accordingly, the electronic device executing the model training method can read the target image from the persistent storage medium.

[0044] The target image serves as the ground truth (GT) for the image inpainting model's learning process; that is, the target image acts as the reference image for the model's learning. To obtain an image inpainting model capable of texture enhancement, an input image, namely a low-quality (LQ) image, is needed—the image the model needs to inpaint during the learning process. Therefore, in this embodiment, the target image also needs to undergo degradation processing. Thus, in step 101, target degradation parameters can also be obtained. Target degradation parameters refer to the parameters used to perform degradation processing on the target image.

[0045] In this embodiment, the target degradation parameter is obtained by optimizing the initial degradation parameter with the goal of ensuring that the texture loss after image degradation processing is less than or equal to a set texture loss threshold. The optimization process of the initial degradation parameter is described in detail below. The initial degradation parameter is a preset degradation parameter.

[0046] The initial degradation parameters may include the initial degradation parameters corresponding to each of the multiple degradation processes. "Multiple" refers to two or more (including two). The degradation operations included in each degradation process may be the same or different. Generally, the degradation operations included in multiple degradation processes are the same. The degradation operations included in each degradation process may include one or more of the following: scaling, blurring, noise addition, and compression. "Multiple" refers to two or more (including two). Since multiple degradation operations may occur during image transmission over a network, in some embodiments of this application, the same-order degradation processing operations may include scaling, blurring, noise addition, and compression to simulate various degradation operations that occur during image transmission over a network.

[0047] The initial degradation parameters are pre-set multi-order degradation parameters. For example, for a second-order degradation operation, the initial degradation parameters can be shown in Table 1.

[0048] Table 1 shows examples of initial degradation parameters. In Table 1, the execution probabilities of scaling operations [0.2, 0.7, 0.1] refer to the probabilities of performing upsampling, downsampling, and invariance operations on the reference image, which are 0.2, 0.7, and 0.1, respectively.

[0049] In some embodiments, a reference image can be degraded according to initial degradation parameters to obtain a corresponding degradation image. The reference image can be all or part of the aforementioned multiple target images, or other images, without limitation. The reference image can be one or more images. The specific implementation of degrading the reference image according to the initial degradation parameters is the same as the subsequent logical steps of degrading the target image according to the target degradation parameters. The degradation process will be specifically described below in the specific implementation of degrading the target image according to the target degradation parameters, and will not be repeated here. In the embodiments of this application, for ease of description and distinction, the degraded image obtained by degrading the target image according to the target degradation parameters is defined as the first degraded image; and the degraded image obtained by degrading the reference image according to the initial degradation parameters is defined as the second degraded image.

[0050] Subsequently, the texture loss between the degraded image (i.e., the second degraded image) corresponding to the reference image and the reference image can be calculated. In this embodiment, the specific implementation of the texture loss is not limited. In some embodiments, PSNR can be used to measure the texture loss between the second degraded image and the reference image; and / or, SSIM can be used to measure the texture loss between the second degraded image and the reference image; and / or, Learned Perceptual Image Patch Similarity (LPIPS) can be used to measure the texture loss between the second degraded image and the reference image. The larger the LPIPS, the higher the distortion of the degraded image, and the greater the texture loss between the degraded image and the reference image. For specific implementations of calculating the PSNR and SSIM between the second degraded image and the reference image, please refer to the aforementioned content on calculating the PSNR and SSIM between the second candidate image and its corresponding blurred image, which will not be repeated here.

[0051] The texture loss between the second degraded image and the reference image includes an embodiment of LPIPS between the second degraded image and the reference image. A pre-trained feature extraction model can be used to extract features from the second degraded image and the reference image respectively to obtain the image features of the second degraded image and the image features of the reference image. Then, the feature distance (such as Euclidean distance, cosine distance, etc.) between the image features of the second degraded image and the image features of the reference image can be determined as the LPIPS between the degraded image and the reference image.

[0052] In some embodiments, the texture loss between the degraded image and the reference image may include: the PSNR between the degraded image and the reference image, and the weighted sum between SSIM and LPIPS.

[0053] After determining the texture loss between the degraded image corresponding to the reference image and the reference image, if the texture loss between the degraded image corresponding to the reference image and the reference image is less than or equal to the set texture loss threshold, then the initial degradation parameter is determined as the target degradation parameter.

[0054] If the texture loss between the degraded image corresponding to the reference image and the reference image is greater than a set texture loss threshold, the initial degradation parameters are optimized for at least one round until the texture loss is less than or equal to the set texture loss threshold. Then, the degradation parameters obtained when the texture loss is less than or equal to the set texture loss threshold are determined as the target degradation parameters.

[0055] In this embodiment, the initial degradation parameters are optimized to ensure that the texture loss after degradation is less than or equal to a set texture loss threshold, thus obtaining the target degradation parameters. Therefore, the degraded image obtained by using the target degradation parameters to degrade the image has a texture loss no greater than the set texture loss threshold compared to the original image. This process not only degrades the image but also preserves important structural information and details. Consequently, when the degraded image is used as input to train the image restoration model, these preserved texture features provide sufficient clues for subsequent image restoration, enabling the model to learn without losing important information. This helps the model learn how to produce more natural and realistic restoration effects.

[0056] The following is an exemplary description of the specific process of optimizing the initial degradation parameters for at least one round when the texture loss corresponding to the degradation processing of the reference image based on the initial degradation parameters is greater than the set texture threshold.

[0057] Specifically, for the first round of optimization, the goal can be to optimize and adjust the initial degradation parameters directly, with the goal of the degraded image texture loss being less than or equal to the set texture loss threshold.

[0058] In some embodiments, the initial degradation parameters include the execution probability of scaling operations and the scaling ratio range. The gradient can be adjusted according to a set probability to increase the execution probability of downsampling operations in the scaling operations included in the initial degradation parameters, and the gradient can also be adjusted according to this probability to decrease the execution probability of resolution preservation operations in the scaling operations included in the initial degradation parameters; wherein the sum of the execution probabilities of scaling operations is 1. And / or, the scaling ratio range included in the initial degradation parameters can be increased according to a set first adjustment gradient.

[0059] In other embodiments, the initial degradation parameters include: the range of Gaussian blur standard deviation and / or the range of noise parameters and / or the range of image compression parameters. Accordingly, optimizing the initial degradation parameters with the goal of the texture loss being less than or equal to the texture loss threshold can be achieved by: reducing the upper limit of the range of Gaussian blur standard deviation included in the initial degradation parameters according to a set second adjustment gradient; and / or, reducing the range of noise parameters included in the initial degradation parameters according to a set third adjustment gradient; and / or, increasing the lower limit of the range of image compression parameters included in the initial degradation parameters according to a set fourth adjustment gradient.

[0060] After adjusting the initial degradation parameters, the degradation parameters for the second round are obtained. Then, based on these second-round degradation parameters, the reference image is degraded to obtain the second-round degradation image. Next, the texture loss between the second-round degradation image and the reference image is calculated to obtain the second-round texture loss. If the second-round texture loss is greater than a set texture loss threshold, the degradation parameters for the second round are adjusted until the texture loss of the degraded image is less than or equal to the set threshold, resulting in the third-round degradation parameters. This process is repeated until the texture loss of the degraded image is less than or equal to the set texture loss threshold.

[0061] The process of optimizing and adjusting the degradation parameters described above can be summarized as follows: Based on the degradation parameters corresponding to this round, the reference image is degraded to obtain the degraded image corresponding to this round (defined as the third degraded image); the texture loss between the third degraded image and the reference image in this round is calculated; if the texture loss corresponding to this round is greater than a set texture loss threshold, the degradation parameters corresponding to this round are adjusted with the goal of the texture loss of the degraded image being less than or equal to the set texture loss threshold; the degradation processing of the reference image using the degradation parameters corresponding to this round and subsequent operations continue until the texture loss of the degraded image is less than or equal to the set texture loss threshold. Afterwards, the degradation parameters corresponding to when the texture loss of the degraded image is less than or equal to the set texture loss threshold can be determined as the target degradation parameters.

[0062] In some embodiments, the degradation parameters include the execution probability of the scaling operation and the scaling ratio range. The gradient can be adjusted according to a set probability to increase the execution probability of the downsampling operation in the scaling operation included in the initial degradation parameters, and the gradient can be adjusted according to this probability to decrease the execution probability of the resolution preservation operation in the scaling operation corresponding to this round; wherein the sum of the execution probabilities of the scaling operations is 1. And / or, the scaling ratio range corresponding to this round can be increased according to a set first adjustment gradient.

[0063] Since the probability of image downsampling during internet transmission is much greater than the probability of maintaining resolution, optimizing the execution probability of scaling operations by increasing the probability of downsampling and decreasing the probability of resolution preservation can make the image degradation processing more closely approximate the occurrence probability of each scaling operation in actual internet image transmission scenarios. This ensures that the probability distribution of scaling operations in sample images is as consistent as possible with the probability distribution of scaling operations in real-world application scenarios, achieving domain alignment and improving the practicality of the trained image restoration model. Increasing the scaling range expands the selection range of subsequent scaling operations, allowing the image restoration model to learn feature representations of the image across a wider scaling range, thus improving the robustness of the image restoration model.

[0064] In other embodiments, the degradation parameters include: the range of Gaussian blur standard deviation and / or the range of noise parameters and / or the range of image compression parameters. Accordingly, to optimize the initial degradation parameters with the goal of the degraded image texture loss being less than or equal to the texture loss threshold, optimization can be achieved by: reducing the upper limit of the Gaussian blur standard deviation range corresponding to the current round according to a set second adjustment gradient; and / or, reducing the noise parameter range corresponding to the current round according to a set third adjustment gradient; and / or, increasing the lower limit of the image compression parameter range corresponding to the current round according to a set fourth adjustment gradient. In this embodiment, reducing the upper limit of the Gaussian blur standard deviation range, reducing the noise parameter range, and increasing the lower limit of the image compression parameter range can narrow the Gaussian blur standard deviation range, the noise parameter range, and the image compression parameter range, reducing the blur degree, noise intensity, and compression intensity of the degraded image, avoiding excessive loss of image texture and semantic information, thereby reducing the texture loss of the degraded image.

[0065] In this embodiment, during a single round of parameter optimization, some or all of the degradation parameters can be optimized. For example, degradation parameters include: the execution probability and scaling ratio range of the scaling operation, the range of Gaussian blur standard deviation, the range of noise parameters, and the range of image compression parameters. This embodiment does not limit the optimization to optimizing all or some of the following parameters during a single round of parameter optimization: the execution probability of the scaling operation, the scaling ratio range, the range of Gaussian blur standard deviation, the range of noise parameters, and the range of image compression parameters.

[0066] In this embodiment, a texture loss threshold is used as the upper limit of texture loss. If the degradation is too strong (i.e., texture loss > texture loss threshold), the degradation parameters are further adjusted to ensure that the generated degraded image retains sufficient texture information, ensure that the degradation intensity is moderate, and avoid excessive image degradation.

[0067] After multiple rounds of optimization and adjustment of the initial degradation parameters shown in Table 1, the target degradation parameters are shown in Table 2. The target degradation parameters shown in Table 2 are exemplary and do not constitute a limitation.

[0068] Table 2 provides examples of target degradation parameters. Figure 6 The above images, from left to right, show the target image, the target image treated with the initial degradation parameters shown in Table 1, and the image itself (i.e., ...). Figure 6 The leftmost image in the top and bottom images is subjected to first-order and second-order degradation processing to obtain the degradation effect image. Figure 6 The following images, from left to right, show the target image and the target degradation parameters shown in Table 2. Figure 6 The target image shown is subjected to first-order and second-order degradation processing, resulting in degradation effects. According to... Figure 6 The degradation effects of the middle and rightmost images in the top and bottom right figures show that the initial degradation parameters over-degrade the target image, resulting in excessive loss of texture details. Conversely, the target degradation parameters can retain some texture details, providing more image context information for subsequent model training.

[0069] After obtaining the target degradation parameters, these parameters can be stored. For example, they can be stored as a configuration file. Accordingly, the target degradation parameters can be read from the configuration file.

[0070] Subsequently, the target image can be degraded according to the target degradation parameters to obtain a degraded image of the target image (i.e., a first degraded image). In embodiments where there are multiple target images, there are multiple first degraded images. Each of the multiple first degraded images corresponds one-to-one with a multiple target image. In some embodiments, the target image can be pre-processed according to the target degradation parameters to obtain a first degraded image, and multiple target images and their respective first degraded images can be stored. Accordingly, in step 101, the pre-stored target image and its first degraded image can be acquired.

[0071] In other embodiments, after acquiring the target image, the target image can be degraded according to the target degradation parameters to obtain a degraded image of the target image.

[0072] The target image can undergo one-order or multi-order degradation processing. Multi-order refers to two or more orders (including order 2). Degradation operations of the same order can include one or more degradation operations such as scaling, blurring, noise addition, and image compression. If the same-order degradation operation includes multiple degradation operations such as scaling, blurring, noise addition, and image compression, the execution order of these operations is not limited. For example, if the same-order degradation operation includes scaling, blurring, noise addition, and image compression, the degradation can be performed in any order. For example, it can be executed sequentially in the order of scaling, blurring, noise addition, and image compression; or sequentially in the order of scaling, image compression, blurring, and noise addition. The following describes the specific implementation method of performing multi-order degradation processing on multiple target images based on target degradation parameters. Since the degradation process is the same for each level, we will take any one of the multi-level degradation processes as an example to explain the degradation process of the target image.

[0073] Implementation Method 1: The target degradation parameters include the execution probability of the scaling operation and the scaling ratio range. Accordingly, for the image to be scaled corresponding to this level of degradation processing, the target scaling operation to be performed on the image can be determined based on the execution probability of the scaling operation included in the target degradation parameters corresponding to this level of degradation processing; the target scaling operation is either an upsampling operation or a downsampling operation. Then, a target scaling ratio suitable for the target scaling operation can be selected from the scaling ratio range included in the target degradation parameters corresponding to this level of degradation processing. If the target scaling operation is upsampling, the target scaling ratio suitable for the target scaling operation belongs to a sub-range greater than 1 in the scaling ratio range included in the target degradation parameters. Specifically, a scaling ratio can be randomly selected as the target scaling ratio from the sub-range greater than 1 in the scaling ratio range. If the scaling operation is downsampling, the target scaling ratio suitable for the target scaling operation belongs to a sub-range less than 1 in the scaling ratio range included in the target degradation parameters. Specifically, a scaling ratio can be randomly selected as the target scaling ratio from the sub-range less than 1 in the scaling ratio range.

[0074] Then, a target scaling operation can be performed on the image to be scaled according to the target scaling ratio. This embodiment uses the image to be scaled corresponding to any target image as an example to illustrate the execution process of the scaling operation. The scaling process can be performed on the images to be scaled corresponding to each target image using appropriate methods.

[0075] Specifically, if the current degradation process is the first-order degradation process and the first degradation operation executed is scaling, then the image to be scaled is the target image. If the current degradation process is not the first-order degradation process, but the first degradation operation executed is scaling, then the image to be scaled is the degradation image obtained from the previous-order degradation process. If the current degradation process is not the first-order degradation process, and scaling is not the first degradation operation executed in this-order degradation process, then the image to be scaled is the output result of the degradation operation preceding the scaling operation in this-order degradation process. For example, if the processing order of the current-order degradation operations is: image compression operation, scaling operation, blurring operation, noise addition operation, etc., executed sequentially, then the image to be scaled is the output result of the corresponding image compression operation in this order.

[0076] Implementation 1 ensures that the scaling degree of the degraded sample images is within a reasonable range by using a fixed scaling ratio range, avoiding extreme cases that could lead to training instability. Implementation 1 also adopts a scaling operation execution probability similar to the image scaling probability distribution in actual Internet scenarios to simulate the scaling distribution in Internet scenarios, making the training data accurately match the scenario.

[0077] Implementation Method 2: For the image to be blurred corresponding to this level of degradation processing, select the target Gaussian blur standard deviation corresponding to the image to be blurred from the range of Gaussian blur standard deviation values ​​included in the target degradation parameters corresponding to this level of degradation processing. Specifically, a Gaussian blur standard deviation can be randomly selected from the range of Gaussian blur standard deviation values ​​as the target Gaussian blur standard deviation. Further, Gaussian blur processing can be performed on the image to be blurred according to the preset Gaussian kernel and the target Gaussian blur standard deviation. The preset Gaussian kernel can be randomly sampled from the range {7, 9...21}.

[0078] In Implementation 2, if the current degradation process is a first-order degradation process and the first degradation operation executed is a blurring operation, then the image to be blurred is the target image. If the current degradation process is not a first-order degradation process, but the first degradation operation executed is a blurring operation, then the image to be blurred is the degradation image obtained from the previous degradation process. If the current degradation process is not a first-order degradation process, and the blurring operation is not the first degradation operation executed in the current degradation process, then the image to be blurred is the output result of the previous degradation operation before the blurring operation in the current degradation process. For example, if the processing order of the current degradation operations is: image compression operation, scaling operation, blurring operation, noise addition operation, etc., executed sequentially, then the image to be scaled is the output result after the scaling operation corresponding to this order is executed.

[0079] Implementation method 2 enhances the realism of degradation by using the range of Gaussian blur standard deviation values ​​to correspond to the typical defocus or motion blur degree in reality, and avoids excessive information loss due to excessive blur.

[0080] Implementation Method 3: For the image to be denoised corresponding to this level of degradation processing, a target noise parameter value corresponding to the image to be denoised is selected from the range of noise parameter values ​​included in the target degradation parameters corresponding to this level of degradation processing. Specifically, a noise parameter value can be randomly selected from the range of noise parameter values ​​corresponding to this level of degradation processing as the target noise parameter value. Further, noise processing can be performed on the image to be denoised based on the target noise parameter value.

[0081] Implementation method 3 limits the upper limit of noise addition by setting the range of noise parameter values, so as to avoid adding too much noise and causing the image information to be overwhelmed.

[0082] In some embodiments, the noise added during the noise addition operation can be Gaussian noise, and the range of noise parameter values ​​includes the range of Gaussian noise standard deviation values. Accordingly, a Gaussian noise standard deviation can be randomly selected from the range of Gaussian noise standard deviation values ​​corresponding to this level of degradation processing as the target Gaussian noise standard deviation. Then, Gaussian noise can be added to the image to be noised based on the target Gaussian noise standard deviation.

[0083] In other embodiments, the noise added during the noise addition operation can be Poisson noise, and the noise parameter value range includes the Poisson noise amplitude range. Accordingly, a Poisson noise amplitude can be randomly selected from the Poisson noise amplitude range corresponding to the current level of degradation processing as the target Poisson noise amplitude. Then, Poisson noise can be added to the image to be noised according to the target Poisson noise amplitude.

[0084] In some embodiments, the noise parameter range includes the range of Gaussian noise standard deviation and the range of Poisson noise amplitude. The target degradation parameter also includes the application probabilities of Gaussian noise and Poisson noise. The sum of the application probabilities of Gaussian noise and Poisson noise is equal to 1. For example, the application probabilities of Gaussian noise and Poisson noise are 0.5 and 0.5 respectively. Accordingly, the target noise type to be applied to the image to be noisy can be determined from the Gaussian noise and Poisson noise based on their application probabilities. The target noise type is either Gaussian noise or Poisson noise. Then, a target noise parameter value can be selected from the range of noise parameter values ​​corresponding to the target noise type. For example, if the target noise type is Gaussian noise, a Gaussian noise standard deviation can be randomly selected from the range of Gaussian noise standard deviation corresponding to this level of degradation processing as the target Gaussian noise standard deviation. Then, Gaussian noise can be added to the image to be noisy based on the target Gaussian noise standard deviation. If the target noise type is Poisson noise, a Poisson noise amplitude can be randomly selected from the Poisson noise amplitude range corresponding to this level of degradation processing as the target Poisson noise amplitude. Then, Poisson noise can be added to the image to be noisy based on the target Poisson noise amplitude. In this embodiment, the model is exposed to both Gaussian noise with fixed variance and brightness-related Poisson noise during training, which makes the degradation more closely resemble real shooting conditions and helps enhance noise diversity and realism.

[0085] In implementation method 3, if the current degradation process is a first-order degradation process and the first degradation operation executed is a noise addition operation, then the image to be noise-added is the target image. If the current degradation process is not a first-order degradation process, but the first degradation operation executed is a noise addition operation, then the image to be noise-added is the degradation image obtained from the previous degradation process. If the current degradation process is not a first-order degradation process, and the noise addition operation is not the first degradation operation executed in the current degradation process, then the image to be noise-added is the output result of the previous degradation operation before the noise addition operation in the current degradation process. For example, if the processing order of the current degradation operations is: image compression operation, scaling operation, blurring operation, noise addition operation, etc., executed sequentially, then the image to be noise-added is the output result after the corresponding blurring operation of the current order is executed.

[0086] Implementation Method 4: For the image to be compressed corresponding to this level of degradation processing, select the target image compression parameter corresponding to the image to be compressed from the range of image compression parameter values ​​included in the target degradation parameters corresponding to this level of degradation processing. For example, an image compression parameter can be randomly selected from the range of image compression parameter values ​​corresponding to this level of degradation processing as the target image compression parameter. Then, image compression can be performed on the image to be compressed according to the target image compression parameter.

[0087] In implementation method 4, if the current degradation process is a first-order degradation process and the first degradation operation executed is an image compression operation, then the image to be compressed is the target image. If the current degradation process is not a first-order degradation process, but the first degradation operation executed is an image compression operation, then the image to be compressed is the degradation image obtained from the previous degradation process. If the current degradation process is not a first-order degradation process, and the image compression operation is not the first degradation operation executed in the current degradation process, then the image to be compressed is the output result of the previous degradation operation before the image compression operation in the current degradation process. For example, if the processing order of the current degradation operations is: scaling operation, blurring operation, noise addition operation, image compression operation, etc., executed sequentially, then the image to be denoised is the output result after the noise addition operation corresponding to this order is executed.

[0088] Implementation method 4 effectively introduces compression artifacts by using a reasonable range of image compression parameters, without introducing too many artifacts. This allows the subsequent image restoration model to learn to eliminate typical artifacts such as blockiness and ringing, which helps improve the practicality of the image restoration model.

[0089] The aforementioned embodiments 1-4 can be implemented individually or in combination. When multiple embodiments are implemented in combination, the execution order of the degradation operations corresponding to the multiple embodiments is not limited, and in the same level of degradation processing, the input image of the later degradation operation is the processing result of the previous degradation operation.

[0090] After performing multi-stage degradation processing on the target image, the corresponding degraded images (i.e., the first degraded images) are obtained. The degradation processing of the target image based on the target degradation parameters can be either pre-degradation processing or online degradation processing.

[0091] In step 102, the initial image restoration model can be trained using the first degraded image as the input image and the target image as the reference image to obtain the target image restoration model. Here, the target image is the model's reference image and the target for model learning; the input image is the image to be restored input into the model. The purpose of training the initial image restoration model is to use the target image as the learning objective. Step 103 is the process of training the initial image restoration model with the objective of minimizing the image difference between the reconstructed image of the first degraded image and the target image.

[0092] In embodiments where there are multiple target images, the reference image for each first degraded image is the target image corresponding to that first degraded image. For example, the reference image for first degraded image 1 is the target image corresponding to first degraded image 1. Accordingly, the image difference is the image difference between each first degraded image and its corresponding target image.

[0093] In this embodiment, the target degradation parameter is obtained by optimizing the initial degradation parameter with the goal of the texture loss after image degradation processing being less than or equal to the set texture loss threshold. Therefore, based on the optimized target degradation parameter, the input image of the model obtained by degradation processing the target image has a texture loss less than the aforementioned texture loss threshold compared to the target image, thereby generating moderately degraded training samples. This allows the degraded image to still retain some texture information, providing reliable contextual clues for the model, enabling it to perform texture inference based on real observations rather than generating artifacts, thus effectively achieving texture enhancement.

[0094] In this application, the specific implementation of the initial image inpainting model is not limited. In some embodiments, the initial image inpainting model may be a generative model, such as a Convolutional Neural Network (CNN) model, a Recurrent Neural Network (RNN) model, or a self-attention mechanism model. For generative models, the initial image inpainting model can be trained with the goal of minimizing the image difference between the reconstructed image of the first degraded image and the target image to obtain the target image inpainting model.

[0095] The image difference between the reconstructed image and the target image can be expressed as the weighted sum of the pixel loss and the perceptual loss between the two images. The pixel loss between the reconstructed image and the target image can be represented by the mean square error of the pixels between them. The perceptual loss between the reconstructed image and the target image can be represented by the LPIPS between them.

[0096] In other embodiments, the initial image inpainting model can be a Generative Adversarial Network (GAN) model. For a GAN network model, the initial image inpainting model includes a generator and a discriminator. The model's loss function can be expressed as a weighted sum of image reconstruction loss and image discrimination loss. The image reconstruction loss can be a weighted sum of pixel loss and perceptual loss between the reconstructed image and the target image. The image discrimination loss can be represented using the cross-entropy between the true / false labels of the reconstructed image output by the discriminator and the actual true / false labels of multiple reconstructed images. Specifically, the label for a real image output by the discriminator can be 1, and the label for a fake image output by the discriminator can be 0. The actual true / false labels of the reconstructed image can be 0.

[0097] In this embodiment, the first degraded image can be input into the generator corresponding to the current training round, and the generator corresponding to the current training round can be used to generate a reconstructed image corresponding to the first degraded image. Then, the target image and the reconstructed image can be input into the discriminator corresponding to the current training round; and the discriminator corresponding to the current training round can be used to determine the authenticity of the reconstructed image based on the target image and the reconstructed image to obtain the authenticity label of the reconstructed image. In embodiments where there are multiple first degraded images, there are multiple reconstructed images. There is a one-to-one correspondence between the first degraded image and the reconstructed image. The target image and the reconstructed image input into the discriminator are the target image and the reconstructed image corresponding to the same first degraded image.

[0098] Furthermore, the image reconstruction loss can be determined based on the target image and the reconstructed image. The image reconstruction loss is the image reconstruction loss between the target image and the reconstructed image corresponding to the same first degraded image. Specifically, the pixel loss and perceptual loss between the reconstructed image and the target image can be calculated based on the target image and its corresponding reconstructed image. The pixel loss and perceptual loss between the reconstructed image and the target image are the pixel loss and perceptual loss between the target image and the reconstructed image corresponding to the same first degraded image. Correspondingly, the image discrimination loss can also be determined based on the true / false labels of the reconstructed image and the actual true / false labels of the reconstructed image.

[0099] Furthermore, the generator and discriminator corresponding to the current training epoch can be adjusted based on the image reconstruction loss and image discrimination loss. Specifically, the pixel loss, perceptual loss, and image discrimination loss between the reconstructed image and the target image can be weighted and summed to obtain the total loss for the current training epoch. Then, the generator and discriminator corresponding to the current training epoch can be gradient-adjusted based on the total loss. Afterward, the adjusted model can be used to continue processing the multiple first degraded images input into the generator corresponding to the current training epoch and subsequent operations until the set number of training epochs is reached. The generator corresponding to the set number of training epochs is then determined as the target image quality restoration model.

[0100] This embodiment introduces an adversarial training mechanism between the generator and the discriminator. The generator is responsible for generating the restored image, while the discriminator learns to distinguish between the generated image and the real image. During training, the generator continuously attempts to generate more realistic images to deceive the discriminator, while the discriminator continuously improves its discrimination ability. This adversarial process forces the generator to focus on generating details that approximate the distribution of real images in terms of local texture, edge sharpness, and high-frequency information. Therefore, by introducing an adversarial mechanism through a GAN network, the model is prompted to generate more realistic texture details, thereby improving the quality of image restoration.

[0101] The target image restoration model trained in this application embodiment can be applied to various application scenarios. For example, it can be used in image editing software to restore user-submitted images. It can also be applied to shopping software to restore images or videos of goods, or to perform offline restoration of live videos. The following is an exemplary description of the image restoration process performed by the target image restoration model trained in this application embodiment.

[0102] Figure 7 This is a schematic flowchart illustrating the image processing method provided in this application. The method primarily uses the target image inpainting model trained in the foregoing embodiments to inpaint the image. For example... Figure 7 As shown, the method mainly includes the following steps: 701. Obtain the image to be processed.

[0103] 702. The image to be processed is restored using a target image restoration model to obtain the target image. The target image restoration model is trained using the model training methods provided in the foregoing embodiments. The training process of the target image restoration model can be found in the relevant content of the foregoing embodiments and will not be repeated here.

[0104] 703. Output the target image.

[0105] In this embodiment, the training samples for the target image restoration model are obtained by performing degradation processing on the target image based on target degradation parameters optimized to ensure that the texture loss after image degradation is less than or equal to a set texture loss threshold. Therefore, the training samples retain some texture information, providing reliable contextual cues for the model, enabling it to perform texture inference based on real observations. Consequently, the target image restored using this trained target image restoration model has more realistic texture details, rather than producing artifacts.

[0106] The inventors of this application used the initial degradation parameters shown in Table 1 above to degrade multiple target images obtained during the model training phase. The degraded images were used as the input model, and the initial image restoration model was trained using the same number of training epochs as the target image restoration model, resulting in another image restoration model (defined as image restoration model 1). Subsequently, image restoration model 1 and the target image restoration model were used to restore the image quality of the same image, yielding... Figure 8 The left and right images are shown. According to... Figure 8 The left and right images show that the image restoration model 1 trained on the degraded image after initial degradation parameter degradation treatment can introduce artifacts during image quality restoration, such as... Figure 8The image in the left-middle image shows artifacts within the rectangular box. However, the image restored by the target image restoration model 1, trained using the degraded image after degradation processing with the target degradation parameters in this embodiment, shows no such artifacts. Therefore, the target image restoration model trained in this embodiment can effectively suppress artifacts when restoring the image quality of the image to be processed.

[0107] It should be noted that the execution subject of each step of the method provided in the above embodiments can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 101 and 102 can be device A; or the execution subject of step 101 can be device A, and the execution subject of step 102 can be device B; and so on.

[0108] Furthermore, some processes described in the above embodiments and accompanying drawings include multiple operations that appear in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or they may be executed in parallel. The operation numbers, such as 101, 102, etc., are merely used to distinguish different operations and do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel.

[0109] Accordingly, embodiments of this application also provide a computer-readable storage medium storing computer instructions, which, when executed by one or more processors, cause one or more processors to perform the steps in the model training method and / or image processing method provided in the foregoing embodiments.

[0110] Computer-readable storage media include volatile or non-volatile or a combination thereof, and may be removable or non-removable. Examples of computer-readable storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital video disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transfer medium.

[0111] This application also provides a computer program product, including a computer program that, when executed by one or more processors, causes the one or more processors to perform the steps in the model training method and / or image processing method provided in the foregoing embodiments.

[0112] In the embodiments of this application, the specific implementation form of the computer program product is not limited. In some embodiments, the computer program product may be implemented as an application (APP), a mini-program, a computer-side client, a program module, a plug-in, an installation package, a software development kit (SDK), an image file of an optical disc, a plug-in, or software in the form of Software as a Service (SaaS), etc., but is not limited to these.

[0113] The computer program product should understand that each or a combination of the above-described method flow can be implemented by a computer program or instructions. Furthermore, these computer programs or instructions can be applied to the processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device, enabling the processor of the general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to function as an apparatus for implementing the corresponding functions in the above-described method embodiments.

[0114] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 9 As shown, the electronic device includes a memory 90a and a processor 90b. The memory 90a is used to store computer programs and can be configured to store various other data to support operation on a computing platform. Examples of this data include instructions for any application or method operating on the electronic device, data structures, contact data, phone book data, messages, pictures, videos, etc.

[0115] The processor 90b is coupled to the memory 90a and is used to execute computer programs to perform the steps in the model training method and / or image processing method provided in the foregoing embodiments. Specific implementation details of each step can be found in the relevant descriptions of the foregoing embodiments, and will not be repeated here.

[0116] In some alternative implementations, such as Figure 9 As shown, the electronic device may also include optional components such as a communication component 90c, a power supply component 90d, a display component 90e, and an audio component 90f. Figure 9 The diagram only shows some components and does not mean that the electronic device must contain them. Figure 9 The inclusion of all components does not imply that an electronic device can only include... Figure 9 The components shown.

[0117] in addition, Figure 9 The components within the dashed box are optional, not mandatory, and their specific requirements depend on the form factor of the electronic device. The electronic device in this embodiment can be a desktop computer, laptop computer, mobile phone, or IoT device; it can also be a traditional server, cloud server, or server cluster, or other server equipment.

[0118] In this embodiment, the memory is used to store computer programs and can be configured to store various other data to support operation on its host device. The processor can execute the computer programs stored in the memory to implement corresponding control logic. The memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), electrically erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0119] In the embodiments of this application, the processor can be any hardware processing device capable of executing the above-described method logic. Optionally, the processor can be a central processing unit (CPU), a graphics processing unit (GPU), or a microcontroller unit (MCU); it can also be a programmable device such as a field-programmable gate array (FPGA), a programmable array logic (PAL), a general array logic (GAL), or a complex programmable logic device (CPLD); or it can be an advanced RISC machine (ARM) or a system on chip (SoC), etc., but is not limited thereto.

[0120] In this embodiment, the communication component is configured to facilitate wired or wireless communication between its host device and other devices. The device hosting the communication component can access wireless networks based on communication standards, such as 2G or 3G, 4G, 5G, or combinations thereof. In one exemplary embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel.

[0121] In embodiments of this application, the display component may include a liquid crystal display (LCD) and a touch panel (TP). If the display component includes a touch panel, the display component can be implemented as a touchscreen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation.

[0122] In this embodiment, a power supply component is configured to provide power to various components of the device in which it resides. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which the power supply component resides.

[0123] In embodiments of this application, the audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC), which is configured to receive external audio signals when the device containing the audio component is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals can be further stored in memory or transmitted via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals. For example, in devices with voice interaction capabilities, voice interaction with the user can be achieved through the audio component.

[0124] It should be noted that the terms "first" and "second" in this article are used to distinguish different messages, devices, modules, etc., and do not represent a chronological order, nor do they limit "first" and "second" to different types.

[0125] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the aforementioned element.

[0126] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A model training method, characterized in that, include: Acquire the target image and the first degraded image; The first degraded image is obtained by degrading the target image according to the target degradation parameters; The target degradation parameter is obtained by optimizing the initial degradation parameter with the goal of the texture loss after image degradation processing being less than or equal to a set texture loss threshold. Using the first degraded image as the input image and the target image as the reference image, the initial image restoration model is trained to obtain the target image restoration model.

2. The method according to claim 1, characterized in that, Also includes: The reference image is degraded according to the initial degradation parameters to obtain a second degraded image; If the texture loss between the second degraded image and the reference image is greater than the texture loss threshold, then the initial degradation parameters are optimized for at least one round with the goal of the texture loss being less than or equal to the texture loss threshold, until the texture loss is less than or equal to the texture loss threshold. The degradation parameter obtained by optimizing when the texture loss is less than or equal to the texture loss threshold is determined as the target degradation parameter.

3. The method according to claim 2, characterized in that, The step of adjusting the initial degradation parameters for at least one round with the goal of achieving a texture loss less than or equal to the texture loss threshold includes performing at least one round of optimization on the initial degradation parameters according to the following steps: Based on the degradation parameters corresponding to this round, the reference image is degraded to obtain the third degraded image corresponding to this round; Calculate the texture loss between the third degraded image and the reference image in this round; If the texture loss corresponding to the current round is greater than the set texture loss threshold, then the degradation parameter corresponding to the current round is adjusted with the goal of the texture loss being less than or equal to the texture loss threshold. Continue performing degradation processing and subsequent operations on the reference image according to the degradation parameters corresponding to this round, until the texture loss of the degraded image is less than or equal to the texture loss threshold.

4. The method according to claim 3, characterized in that, The degradation parameters include: the probability of scaling operations and the scaling range; The step of adjusting the degradation parameters for the current round with the goal of achieving a texture loss less than or equal to the texture loss threshold includes: The gradient is adjusted according to a set probability to increase the execution probability of the downsampling operation in the scaling operation corresponding to the current round, and the gradient is adjusted according to the probability to decrease the execution probability of the resolution preservation operation in the scaling operation corresponding to the current round; wherein, the sum of the execution probabilities of the scaling operations is 1; And / or, According to the set first adjustment gradient, increase the scaling ratio range corresponding to this round.

5. The method according to claim 3, characterized in that, The degradation parameters include: the range of Gaussian blur standard deviation and / or the range of noise parameters and / or the range of image compression parameters; the adjustment of the degradation parameters corresponding to this round, with the goal of texture loss being less than or equal to the texture loss threshold, includes: According to the set second adjustment gradient, the upper limit of the range of Gaussian blur standard deviation corresponding to this round is reduced; And / or, According to the set third adjustment gradient, the range of noise parameter values ​​corresponding to this round is narrowed; And / or, According to the set fourth adjustment gradient, the lower limit of the range of image compression parameters corresponding to this round is increased.

6. The method according to claim 1, characterized in that, The target degradation parameters include: target degradation parameters corresponding to each of the multi-level degradation processes; and degradation processing of the target image based on the target degradation parameters includes: For the image to be scaled corresponding to this level of degradation processing, the target scaling operation to be performed on the image to be scaled is determined based on the execution probability of the scaling operation included in the target degradation parameters corresponding to this level of degradation processing; the target scaling operation is an upsampling operation or a downsampling operation; a target scaling ratio adapted to the target scaling operation is selected from the scaling ratio range included in the target degradation parameters corresponding to this level of degradation processing; the target scaling operation is performed on the image to be scaled according to the target scaling ratio. And / or, For the image to be blurred corresponding to this level of degradation processing, the target Gaussian blur standard deviation corresponding to the image to be blurred is selected from the range of Gaussian blur standard deviation values ​​included in the target degradation parameters corresponding to this level of degradation processing; Gaussian blur processing is performed on the image to be blurred according to the preset Gaussian kernel and the target Gaussian blur standard deviation. And / or, For the image to be denoised corresponding to this level of degradation processing, the target noise parameter value corresponding to the image to be denoised is selected from the range of noise parameter values ​​included in the target degradation parameters corresponding to this level of degradation processing; and the image to be denoised is then denoised according to the target noise parameter value. And / or, For the image to be compressed corresponding to this level of degradation processing, a target image compression parameter corresponding to the image to be compressed is selected from the range of image compression parameter values ​​included in the target degradation parameter corresponding to this level of degradation processing; and the image to be compressed is compressed according to the target image compression parameter.

7. The method according to claim 6, characterized in that, The noise parameter range includes the range of Gaussian noise standard deviation and the range of Poisson noise amplitude; the target degradation parameter also includes the application probability of Gaussian noise and Poisson noise. The step of selecting the target noise parameter value corresponding to the image to be denoised from the range of noise parameter values ​​included in the target degradation parameters corresponding to the current degradation processing includes: Based on the application probabilities of Gaussian noise and Poisson noise, the target noise type to be applied to the image to be noisy is determined from Gaussian noise and Poisson noise; the target noise type is either Gaussian noise or Poisson noise. Select the target noise parameter value from the range of noise parameter values ​​corresponding to the target noise type.

8. The method according to any one of claims 1-7, characterized in that, The initial image restoration model includes a generator and a discriminator. The step of training the initial image restoration model using the first degraded image as input and the target image as a reference image to obtain the target image restoration model includes training the initial image restoration model for a set number of rounds according to the following steps: The first degraded image is input into the generator corresponding to this training round, and the generator corresponding to this training round is used to generate the reconstructed image corresponding to the first degraded image. The target image and the reconstructed image are input into the discriminator corresponding to the current training round; and the discriminator corresponding to the current training round is used to perform a authenticity judgment on the reconstructed image based on the target image and the reconstructed image, so as to obtain the authenticity label of the reconstructed image; Based on the target image and the reconstructed image, determine the image reconstruction loss; The image discrimination loss is determined based on the true / false labels of the reconstructed image and the actual true / false labels of the reconstructed image; Based on the image reconstruction loss and the image discrimination loss, the generator and the discriminator corresponding to the current training round are adjusted, and the process of inputting the first degraded image into the generator corresponding to the current training round and subsequent operations continues until the training round reaches the set number of rounds. The generator corresponding to the set round is determined as the target image quality restoration model.

9. The method according to any one of claims 1-7, characterized in that, The acquisition of the target image includes: Obtain multiple first candidate images from the image database; From the plurality of first candidate images, obtain a plurality of second candidate images whose pixel mean is within a set pixel value range and whose pixel variance is greater than or equal to a set pixel variance threshold; The plurality of second candidate images are blurred to obtain a plurality of blurred images; Based on the pixel values ​​of the plurality of second candidate images and the pixel values ​​of the plurality of blurred images, calculate the image difference between the plurality of second candidate images and the blurred images corresponding to each of the plurality of second candidate images; From the plurality of second candidate images, the second candidate image whose image difference is greater than or equal to a set difference threshold is selected as the target image.

10. The method according to claim 9, characterized in that, The step of calculating the difference between the multiple second candidate images and their respective corresponding blurred images based on the pixel values ​​of the multiple second candidate images includes: Based on the pixel values ​​of the plurality of second candidate images, the peak signal-to-noise ratio and structural similarity index between the plurality of second candidate images and their respective corresponding blurred images are calculated, which serve as the differences between the plurality of second candidate images and their respective corresponding blurred images; Selecting a second candidate image from the plurality of second candidate images whose difference from the corresponding blurred image is greater than or equal to a set difference threshold includes: From the plurality of second candidate images, the second candidate image with a peak signal-to-noise ratio less than or equal to a set peak signal-to-noise ratio threshold and a structural similarity index less than or equal to a set structural similarity index threshold is selected as the second candidate image whose difference from the corresponding blurred image is greater than or equal to a set difference threshold.

11. The method according to claim 9, characterized in that, The acquisition of the target image also includes: Using a language model, natural language description information is generated for the images and / or videos to be generated; the topics of the descriptions of the images and / or videos to be generated are different from the topics of the image descriptions in the image database. In the natural language description information, a description of the image quality requirements of the image and / or video to be generated is added to obtain target prompt words; The image and / or video to be generated are generated based on the target prompt words using a cross-modal generative model; The target image is obtained from the video frames of the image and / or video to be generated.

12. The method according to claim 11, characterized in that, The step of obtaining the target image from video frames in the image and / or video to be generated includes: Using image quality evaluation model and aesthetic evaluation model, the video frames in the image and / or video to be generated are evaluated for image quality and aesthetics respectively, and the image quality score and aesthetic score of the video frames in the image and / or video to be generated are obtained. From the video frames in the image and / or video to be generated, select images with a quality score greater than or equal to a set quality score threshold and an aesthetic score greater than or equal to a set aesthetic score threshold as the target image.

13. The method according to any one of claims 1-7, characterized in that, The acquisition of the target image includes: Using a language model, natural language descriptions of the images and / or videos to be generated are produced. In the natural language description information, a description of the image quality requirements of the image and / or video to be generated is added to obtain target prompt words; The image and / or video to be generated are generated based on the target prompt words using a cross-modal generative model; The target image is obtained from the video frames of the image and / or video to be generated.

14. An image processing method, characterized in that, include: Obtain the image to be processed; The image to be processed is restored using a target image restoration model to obtain the target image; Output the target image; wherein the target image restoration model is obtained by training the model using the model training method according to any one of claims 1-13.

15. An electronic device, characterized in that, include: A memory and a processor; wherein the memory is used to store computer programs; The processor is coupled to the memory for executing the computer program to perform the steps of the method according to any one of claims 1-14.

16. A computer-readable storage medium storing computer instructions, characterized in that, When the computer instructions are executed by one or more processors, the one or more processors are caused to perform the steps of the method according to any one of claims 1-14.

17. A computer program product, characterized in that, Includes a computer program that, when executed by one or more processors, causes the one or more processors to perform the steps of the method according to any one of claims 1-14.

Images