Model training and image processing method and device, electronic equipment and storage medium
By constructing a bidirectional degradation training dataset, including unclear distorted images and sharpened distorted images, the model is trained, which solves the problem of oversharp pictures in image enhancement technology, and improves the model training effect and image processing quality.
Patent Information
- Application Number
- CN202411896238.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-20
- Publication Date
- 2025-04-15
AI Technical Summary
Existing image enhancement technologies can easily lead to excessive sharpness in the process of improving image clarity, especially on mobile devices, where excessively sharp images will affect user experience and retention.
By constructing a bidirectional degradation training dataset, including unclear distorted images and sharpened distorted images, the model is trained using the model training method and the model parameters are adjusted to avoid oversharp problems.
It effectively alleviates the problem of oversharpness in image enhancement training, improves the effect of model training, and ensures the high quality and user-friendliness of image processing results.
Smart Images
Figure CN119831874A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of artificial intelligence technology. More specifically, the present disclosure relates to a model training and image processing method, device, electronic device and storage medium. Background Art
[0002] Image enhancement technology is used to process low-quality distorted images and improve the subjective image quality of low-quality images. It is a very practical technology and has attracted much attention in academia and industry. In particular, in some video application scenarios, image enhancement algorithms can even be used to improve the quality of videos while reducing the bit rate of video transmission, using image quality to replace bandwidth and achieve the purpose of cost saving.
[0003] The goal of image enhancement technology is to achieve better subjective image quality. However, higher clarity does not mean better subjective image quality. Simply improving image clarity will bring the risk of over-sharp images, especially in most scenarios where users consume images and videos on mobile devices, and the display screens of mobile devices are often small. Therefore, end users are more sensitive to over-sharp images. Once the image is too sharp, even if the image or video content is interesting, it will have a negative impact on user retention on the platform. Summary of the invention
[0004] An exemplary embodiment of the present disclosure is to provide a model training method and device to at least solve the problem of model training in the related art, and may not solve any of the above problems.
[0005] According to an exemplary embodiment of the present disclosure, there is provided a model training method, comprising: acquiring a first image set and a first target image set corresponding to the first image set, wherein the first image set is an image set including unclear distorted images; acquiring a second image set and a second target image set corresponding to the second image set, wherein the second image set is an image set including sharp distorted images; and training a model based on the first image set, the first target image set, the second image set, and the second target image set.
[0006] Optionally, the number of images included in the first image set and the number of images included in the second image set may have a predetermined proportional relationship.
[0007] Optionally, acquiring the first image set may include: performing degradation processing on at least part of the images in the first target image set to obtain the first image set.
[0008] Optionally, acquiring the second image set may include: performing sharpening processing on at least part of the images in the second target image set to obtain the second image set.
[0009] Optionally, obtaining the first target image set may include: assigning values to at least part of the training images in the training image set based on a predetermined sharpening probability; determining the training images in the predetermined training image set whose assigned values are less than or equal to a threshold as the first target images, to obtain the first target image set, wherein the sharpening probability represents the probability that the training images in the training image set will be sharpened.
[0010] Optionally, the acquiring the second target image set may include: determining training images in the predetermined training image set whose assigned values are greater than the threshold as second target images, to obtain the second target image set.
[0011] Optionally, the degradation processing may include at least one of blurring processing, noise adding processing, compression processing, and sampling processing.
[0012] Optionally, the training of the model based on the first image set, the first target image set, the second image set, and the second target image set may include: taking images in the first image set and the second image set as inputs to the model, and using the model to predict output images; comparing target images corresponding to the images in the first target image set and the second target image set with the output images; and adjusting parameters of the model based on the comparison results.
[0013] According to an exemplary embodiment of the present disclosure, there is provided an image processing method, comprising: inputting an image to be processed into a model trained using the model training method described in the present disclosure; and using an image output by the model as a processed image.
[0014] Optionally, the model may include a model related to images and / or videos.
[0015] According to an exemplary embodiment of the present disclosure, a model training device is provided, which may include: a first image acquisition unit, configured to acquire a first image set and a first target image set corresponding to the first image set, wherein the first image set is an image set including unclear distorted images; a second image acquisition unit, configured to acquire a second image set and a second target image set corresponding to the second image set, wherein the second image set is an image set including sharp distorted images; and a model training unit, configured to train a model based on the first image set, the first target image set, the second image set, and the second target image set.
[0016] Optionally, the number of images included in the first image set and the number of images included in the second image set may have a predetermined proportional relationship.
[0017] Optionally, the first image acquisition unit may be configured to: perform degradation processing on at least part of the images in the first target image set to obtain the first image set.
[0018] Optionally, the second image acquisition unit may be configured to: perform sharpening processing on at least part of the images in the second target image set to obtain the second image set.
[0019] Optionally, the first image acquisition unit may be configured to: assign values to at least part of the training images in the training image set based on a predetermined sharpening probability; determine the training images in the predetermined training image set whose assigned values are less than or equal to a threshold as first target images, to obtain a first target image set, wherein the sharpening probability represents the probability that the training images in the training image set will be sharpened.
[0020] Optionally, the second image acquisition unit may be configured to: determine the training images in the predetermined training image set whose values are greater than the threshold as the second target images, to obtain the second target image set.
[0021] Optionally, the degradation processing includes at least one of blurring processing, noise adding processing, compression processing, and sampling processing.
[0022] Optionally, the model training unit can be configured to: use the images in the first image set and the second image set as input of the model, and use the model to predict the output image; compare the target images corresponding to the images in the first target image set and the second target image set with the output image; and adjust the parameters of the model based on the comparison results.
[0023] According to an exemplary embodiment of the present disclosure, there is provided an image processing device, comprising: an image input unit, configured to input an image to be processed into a model trained using the model training method described in the present disclosure; and an image output unit, configured to use the image output by the model as a processed image.
[0024] Optionally, the model may include a model related to images and / or videos.
[0025] According to an exemplary embodiment of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement a model training method according to an exemplary embodiment of the present disclosure.
[0026] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor of an electronic device, the electronic device executes the model training method according to the exemplary embodiment of the present disclosure.
[0027] According to an exemplary embodiment of the present disclosure, a computer program product is provided, including a computer program / instruction. When the computer program / instruction is executed by a processor, the model training method according to the exemplary embodiment of the present disclosure is implemented.
[0028] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:
[0029] By training the model based on the first image set, the first target image set, the second image set including the sharpened distorted images, and the second target image set, the effect of the model training is improved.
[0030] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.
[0032] Figure 1 An exemplary system architecture is shown in which exemplary embodiments of the present disclosure may be applied.
[0033] Figure 2 A flowchart of a model training method according to an exemplary embodiment of the present disclosure is shown.
[0034] Figure 3 A schematic diagram showing processing a training image according to an exemplary embodiment of the present disclosure.
[0035] Figure 4 A flowchart illustrating an image processing method according to an exemplary embodiment of the present disclosure.
[0036] Figure 5 A block diagram of a model training apparatus according to an exemplary embodiment of the present disclosure is shown.
[0037] Figure 6 A block diagram illustrating an image processing apparatus according to an exemplary embodiment of the present disclosure.
[0038] Figure 7 is a block diagram of an electronic device 700 according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION
[0039] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.
[0040] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims.
[0041] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.
[0042] The training of conventional enhancement algorithms often includes three parts: training data set, enhancement network, and loss function. The most common setting of training data set is low-quality-high-quality image pairs, where low-quality images are used as inputs to the enhancement network and high-quality images are used as target labels for training the enhancement network. Since the model's fitting ability is limited and real data is diverse, it is very likely that some test images will be too sharp during the training of the model. The usual solution is to introduce some anti-sharp loss constraints into the loss function, such as using the total variation (TV) loss, that is, calculating the gradient of the output image of the enhancement network to make it as small as possible. This usual solution requires fine-tuning the proportion of each loss term, making the training unstable, time-consuming, and prone to loss of details in the output image.
[0043] For image enhancement scenarios and other related image or video scenarios, the present disclosure proposes an anti-oversharpness technology constructed by optimizing the training data set, which can effectively alleviate the oversharpness problem in image enhancement training and obtain a more robust image enhancement model.
[0044] Next, we will refer to Figures 1 to 7 The model training method and apparatus according to the exemplary embodiments of the present disclosure are described in detail.
[0045] Figure 1 An exemplary system architecture 100 is shown in which exemplary embodiments of the present disclosure may be applied.
[0046] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for a communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc. Users can use terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages (for example, video data upload requests, video data download requests), etc. Various communication client applications may be installed on the terminal devices 101, 102, 103, such as audio and video call software, audio and video recording software, instant messaging software, conference software, email client, social platform software, etc. The terminal devices 101, 102, 103 may be hardware or software. When the terminal devices 101, 102, and 103 are hardware, they can be various electronic devices with display screens and capable of playing, recording, and editing audio and video, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers. When the terminal devices 101, 102, and 103 are software, they can be installed in the electronic devices listed above, and can be implemented as multiple software or software modules (for example, to provide distributed services), or as a single software or software module. No specific limitation is made here.
[0047] The terminal devices 101, 102, and 103 may be equipped with an image acquisition device (e.g., a camera) to collect video data. In practice, the smallest visual unit that constitutes a video is a frame. Each frame is a static image. Synthesizing a temporally continuous sequence of frames together forms a dynamic video. In addition, the terminal devices 101, 102, and 103 may also be equipped with a component for converting an electrical signal into sound (e.g., a speaker) to play sound, and may also be equipped with a device for converting an analog audio signal into a digital audio signal (e.g., a microphone) to collect sound.
[0048] The server 105 may be a server that provides various services, such as a background server that provides support for multimedia applications installed on the terminal devices 101, 102, and 103. The background server may parse, store, and process the received audio and video data upload request and other data, and may also receive the audio and video data download request sent by the terminal devices 101, 102, and 103, and feed back the audio and video data indicated by the audio and video data download request to the terminal devices 101, 102, and 103.
[0049] It should be noted that the server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules (for example, used to provide distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.
[0050] It should be noted that the model training method and image processing method provided in the embodiments of the present disclosure are usually executed by a terminal device, but can also be executed by a server, or can also be executed by a terminal device and a server in collaboration. Accordingly, the model training device and the image processing device can be set in the terminal device, in the server, or in both the terminal device and the server.
[0051] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices, networks and servers, and the present disclosure has no limitation on this.
[0052] Usually when constructing a training data set, the low-quality images used as input are images with some kind of distortion, such as blur, noise, compression, sampling, etc., which means that the image is subtracted from the high-quality image in terms of perception; and the problem of oversharpening is essentially an addition to the high-quality image in terms of perception, that is, it is too clear, with white edges at the edges. In the past, when constructing training data sets, we only focused on the distortion degradation of subtraction, so the problem of oversharpening is easy to occur in the training enhancement model.
[0053] To address this problem, the present invention proposes a bidirectional degradation idea. Adhering to the principle of "too much is as bad as too little", in data construction, in addition to simulating negative degradation (blurring, noise, compression, sampling, etc.), positive degradation is also simulated to sharpen high-quality images. For example, the unsharp masking (USM) technology is used to simulate over-sharp input situations. This enables the model to learn distortion types more comprehensively and forces the model to pull over-sharp input images back to normal high-quality images.
[0054] Figure 2 A flowchart of a model training method according to an exemplary embodiment of the present disclosure is shown. Figure 3 A schematic diagram showing processing a training image according to an exemplary embodiment of the present disclosure.
[0055] Reference Figure 2, in step S201, a first image set and a first target image set corresponding to the first image set are obtained. Here, the first image set is an image set including unclear and distorted images. In the present disclosure, in the case where there is already a first image set including unclear and distorted images, the existing first image set can be directly obtained, and the first target image set corresponding to the first image set can be obtained. An unclear and distorted image is an image whose image clarity is lower than a threshold. For example, an unclear and distorted image can be a blurred image, a noisy image, a compressed image, etc.
[0056] In an exemplary embodiment of the present disclosure, acquiring the first image set may include: performing degradation processing on at least part of the images in the first target image set to obtain the first image set. In the present disclosure, in the absence of a ready-made first image set including unclear and distorted images, unclear and distorted images may be obtained by performing degradation processing on at least part of the images in the first target image set to obtain the first image set.
[0057] In an exemplary embodiment of the present disclosure, the degradation process may include at least one of blurring, adding noise, compression, and sampling. In addition, the degradation process may also be any other degradation process, which is not limited by the present disclosure.
[0058] For example, at least some of the images in the first target image set may be randomly subjected to at least one of blur processing, noise adding processing, compression processing, and sampling processing. When multiple processes of blur processing, noise adding processing, compression processing, and sampling processing are performed on the images, the multiple processes may have different processing orders.
[0059] As an example, one of blurring, noise addition, compression and sampling is performed on the first part of the images in the first target image set, two of blurring, noise addition, compression and sampling are performed on the second part of the images in the first target image set, three of blurring, noise addition, compression and sampling are performed on the third part of the images in the first target image set, and four of blurring, noise addition, compression and sampling are performed on the fourth part of the images in the first target image set.
[0060] As an example, when two of the blurring, noise adding, compression and sampling processes are respectively performed on the second part of the images in the first target image set, the blurring process is first performed on some of the images and then the noise adding process is performed, the blurring process is first performed on some of the images and then the compression process is performed, the blurring process is first performed on some of the images and then the sampling process is performed, the noise adding process is first performed on some of the images and then the blurring process is performed, the compression process is first performed on some of the images and then the blurring process is performed, the sampling process is first performed on some of the images and then the blurring process is performed, and so on.
[0061] As an example, when performing three processes of blurring, noise adding, compression and sampling on the third part of the images in the first target image set, blurring is performed first, noise adding is performed and then compression is performed on some of the third part of the images; noise adding is performed first, compression is performed and then blurring is performed on some of the third part of the images; compression is performed first, blurring is performed and then noise adding is performed on some of the third part of the images; blurring is performed first, noise adding is performed and then sampling is performed on some of the third part of the images; noise adding is performed first, sampling is performed and then blurring is performed on some of the third part of the images; sampling is performed first, blurring is performed and then noise adding is performed on some of the third part of the images, and…
[0062] In an exemplary embodiment of the present disclosure, obtaining a first target image set may include: assigning values to at least part of the training images in the training image set based on a predetermined sharpening probability; determining the training images in the predetermined training image set whose assigned values are less than or equal to a threshold as first target images, and obtaining the first target image set. Here, the sharpening probability represents the probability that the training images in the training image set are sharpened. For example, when the sharpening probability is 8%, it means that the training images in the predetermined training image set whose sharpening probability is greater than or equal to 8% will be determined as the first target images. Here, the assignment can be based on a predetermined rule. For example, the training images in the training image set can be randomly assigned values based on the sharpening probability and the threshold, and the assignment range is related to the threshold. For example, the assignment range can be any range including the threshold, and the threshold can be set arbitrarily, and the present disclosure does not limit this.
[0063] For example, when the sharpening probability is 8% and the threshold is 0.1, the predetermined rule may be to randomly assign a random value greater than 0.1 to 8% of the training images in the predetermined training image set, and to randomly assign a random value less than or equal to 0.1 to 92% of the training images in the predetermined training image set, so that the 92% of the training images in the predetermined training image set with a value less than or equal to 0.1 are determined as the first target image. For example, when the sharpening probability is 10% and the threshold is 0.1, the predetermined rule may be to randomly assign a random value greater than 0.1 to 10% of the training images in the training image set, and to randomly assign a random value less than or equal to 0.1 to 90% of the training images in the training image set, so that the 90% of the training images in the predetermined training image set with a value less than or equal to 0.1 are determined as the first target image. For example, when the sharpening probability is 12% and the threshold is 0.12, the predetermined rule may be to randomly assign a random value greater than 0.12 to 12% of the training images in the training image set, and to randomly assign a random value less than or equal to 0.12 to 88% of the training images in the training image set. Therefore, the 88% of the training images in the predetermined training image set with a value less than or equal to 0.12 are determined as the first target images.
[0064] In step S202, a second image set and a second target image set corresponding to the second image set are obtained. Here, the second image set is an image set including sharp distortion images. In the present disclosure, in the case where there is already a second image set including sharp distortion images, the existing second image set can be directly obtained, and the second target image set corresponding to the second image set can be obtained. The sharp distortion image can be an image with over-sharp white edges. Over-sharp white edges refer to the unnatural white edges or halos that appear on the edges due to excessive sharpening during the image processing process. This phenomenon usually occurs at the edges of the details of the image. The sharpening operation increases the contrast of the edge pixels, making the edges look clearer, but if the sharpening is excessive, unnatural white edges or halos will be generated at the edges, affecting the overall quality of the image. The main reason for over-sharpening white edges is over-sharpening. Sharpening is to focus a blurred image by increasing the contrast of adjacent pixels, but if the sharpening amount is set too large, the contrast of the edge pixels will be too strong, resulting in white edges.
[0065] In an exemplary embodiment of the present disclosure, acquiring the second image set may include: sharpening at least part of the images in the second target image set to obtain the second image set. For example, the images in the second target image set may be processed by the unsharp masking (USM) technology to obtain the sharpened distorted images in the second image set. Here, various sharpening processing methods that can achieve the purpose of sharpening may be used, and the present disclosure does not limit this. In the present disclosure, in the absence of a ready-made second image set including sharpened distorted images, the sharpened distorted images may be obtained by sharpening at least part of the images in the second target image set, thereby achieving the acquisition of the second image set.
[0066] In an exemplary embodiment of the present disclosure, obtaining the second target image set may include: determining the training images in the predetermined training image set whose values are greater than the threshold as the second target images, thereby obtaining the second target image set, thereby facilitating the determination of whether to perform degradation processing or sharpening processing on the images. For example, when the threshold is 8%, the training images in the predetermined training image set whose sharpening probability is less than 8% may be determined as the second target images.
[0067] For example, when the sharpening probability is 8% and the threshold is 0.1, the 8% of training images in the predetermined training image set with a value greater than 0.1 are determined as the second target image. For example, when the sharpening probability is 10% and the threshold is 0.1, the 10% of training images in the predetermined training image set with a value greater than 0.1 are determined as the second target image. For example, when the sharpening probability is 12% and the threshold is 0.12, the 12% of training images in the predetermined training image set with a value greater than 0.12 are determined as the second target image.
[0068] In an exemplary embodiment of the present disclosure, the number of images included in the first image set and the number of images included in the second image set may have a predetermined proportional relationship, thereby controlling the degree of influence of oversharp degradation. For example, the number of images included in the second image set may be, for example, but not limited to, one tenth of the number of images included in the first image set.
[0069] As an example, a probability parameter may be introduced to make the number of images included in the first image set and the number of images included in the second image set have a predetermined proportional relationship, thereby controlling the degree of influence of oversharp degradation. The larger the probability parameter, the higher the possibility of oversharp and low-quality images appearing in the images used as model training data, so the trained model can better restore oversharp images to normal high-quality images (i.e., the trained model is more conservative); conversely, the smaller the probability parameter, the lower the possibility of oversharp and low-quality images appearing in the images used as training data, so the trained model cannot well restore oversharp images to normal high-quality images (i.e., the trained model is more aggressive). In extreme cases, the probability parameter is close to 0, that is, it becomes basically only negative degradation, and the problem of oversharpness is prone to occur. Preferably, when the probability parameter of oversharp distortion is controlled to a small but non-zero degree (for example, but not limited to, for example, about 10%), the effect of model training is better.
[0070] In step S203, the model is trained based on the first image set, the first target image set, the second image set, and the second target image set.
[0071] In an exemplary embodiment of the present disclosure, the model may include a model related to an image and / or a video. For example, the model may be any model or any network structure related to an image and / or a video, and the present disclosure does not limit this. The loss function of the model may be any loss function, and the present disclosure does not limit this. As an example, the model may be an image enhancement model.
[0072] In an exemplary embodiment of the present disclosure, training a model based on a first image set, a first target image set, a second image set, and a second target image set may include: taking images in the first image set and the second image set as inputs to the model, and using the model to predict output images; comparing target images corresponding to the images in the first target image set and the second target image set with the output images; and adjusting parameters of the model based on the comparison results.
[0073] According to the model training method of the exemplary embodiment of the present disclosure, an idea of "bidirectional degradation" is proposed. In addition to simulating the negative degradation of "low definition" in the model training data (for example, images used as model training data), the positive excessive degradation of "high definition" is also simulated. An over-sharp and distorted input image is added to the model training data (for example, images used as model training data) with a small probability. Therefore, when the obtained model training data is used for model training, the model can learn the distortion type more comprehensively, so that the model has the ability to avoid over-sharp images, which can effectively alleviate the problem of over-sharpness in the output results of the model and improve the effect of model training.
[0074] Figure 4 A flowchart illustrating an image processing method according to an exemplary embodiment of the present disclosure.
[0075] Reference Figure 4 In step S401, the image to be processed is input into the model trained using the model training method described in the present disclosure.
[0076] In an exemplary embodiment of the present disclosure, the model may include a model related to an image and / or a video. For example, the model may be any model or any network structure related to an image and / or a video, and the present disclosure does not limit this. The loss function of the model may be any loss function, and the present disclosure does not limit this. As an example, the model may be an image enhancement model.
[0077] In step S402, the image output by the model is used as the processed image. For example, when the model is an image enhancement model, an enhanced image can be obtained by inputting the image to be processed into the model, and the obtained enhanced image has high definition and no oversharpening problem.
[0078] According to the image processing method of the exemplary embodiment of the present disclosure, the model trained by the model training method of the present disclosure is used to perform image processing, so that the image processing effect can be improved by the model's ability to avoid over-sharpening of the image.
[0079] The above has been combined Figures 1 to 4 The model training method and image processing method according to the exemplary embodiment of the present disclosure are described. Figure 5 and Figure 6 A model training device and its units, and an image processing device and its units according to exemplary embodiments of the present disclosure are described.
[0080] Figure 5 A block diagram of a model training apparatus according to an exemplary embodiment of the present disclosure is shown.
[0081] Reference Figure 5 The model training device includes a first image acquisition unit 51, a second image acquisition unit 52 and a model training unit 53.
[0082] The first image acquisition unit 51 is configured to acquire a first image set and a first target image set corresponding to the first image set, wherein the first image set is an image set including unclear and distorted images.
[0083] In an exemplary embodiment of the present disclosure, the number of images included in the first image set and the number of images included in the second image set may have a predetermined proportional relationship.
[0084] In an exemplary embodiment of the present disclosure, the first image acquisition unit 51 may be configured to: perform degradation processing on at least part of the images in the first target image set to obtain the first image set.
[0085] In an exemplary embodiment of the present disclosure, the first image acquisition unit 51 may be configured to: assign values to at least part of the training images in the training image set based on a predetermined sharpening probability; determine the training images in the predetermined training image set whose assigned values are less than or equal to a threshold as first target images, to obtain a first target image set, wherein the sharpening probability represents the probability that the training images in the training image set are sharpened.
[0086] In an exemplary embodiment of the present disclosure, the degradation processing includes at least one of blurring processing, noise adding processing, compression processing, and sampling processing.
[0087] The second image acquisition unit 52 is configured to acquire a second image set and a second target image set corresponding to the second image set, wherein the second image set is an image set including a sharp distortion image.
[0088] In an exemplary embodiment of the present disclosure, the second image acquisition unit 52 may be configured to: perform sharpening processing on at least part of the images in the second target image set to obtain the second image set.
[0089] The model training unit 53 is configured to train the model based on the first image set, the first target image set, the second image set, and the second target image set.
[0090] In an exemplary embodiment of the present disclosure, the second image acquisition unit 52 may be configured to: determine the training images in the predetermined training image set whose values are greater than the threshold as second target images, to obtain a second target image set.
[0091] In an exemplary embodiment of the present disclosure, the model training unit 53 may be configured to: use images in the first image set and the second image set as inputs to the model, and use the model to predict output images; compare target images corresponding to the images in the first target image set and the second target image set with the output image; and adjust parameters of the model based on the comparison results.
[0092] Figure 6 A block diagram illustrating an image processing apparatus according to an exemplary embodiment of the present disclosure.
[0093] Reference Figure 6 , the image processing device includes an image input unit 61 and an image output unit 62.
[0094] The image input unit 61 is configured to input the image to be processed into the model trained using the model training method described in the present disclosure.
[0095] In an exemplary embodiment of the present disclosure, the model may include a model related to an image and / or a video.
[0096] The image output unit 62 is configured to use the image output by the model as a processed image.
[0097] Regarding the device in the above embodiment, the specific manner in which each unit performs the operation has been described in detail in the embodiment of the method, and will not be elaborated here.
[0098] The above has been combined Figure 5 and Figure 6 A model training device according to an exemplary embodiment of the present disclosure is described. Next, Figure 7 An electronic device according to an exemplary embodiment of the present disclosure is described.
[0099] Figure 7 is a block diagram of an electronic device 700 according to an exemplary embodiment of the present disclosure.
[0100] Reference Figure 7 The electronic device 700 includes at least one memory 701 and at least one processor 702, wherein the at least one memory 701 stores a set of computer executable instructions, and when the computer executable instruction set is executed by the at least one processor 702, the method for model training according to the exemplary embodiment of the present disclosure is executed.
[0101] In an exemplary embodiment of the present disclosure, the electronic device 700 may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above-mentioned instruction set. Here, the electronic device 700 is not necessarily a single electronic device, but may also be any device or circuit capable of executing the above-mentioned instructions (or instruction sets) individually or in combination. The electronic device 700 may also be part of an integrated control system or system manager, or may be configured as a portable electronic device interconnected with a local or remote (e.g., via wireless transmission) interface.
[0102] In the electronic device 700, the processor 702 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not limitation, the processor may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.
[0103] The processor 702 may execute instructions or codes stored in the memory 701, which may also store data. Instructions and data may also be sent and received over a network via a network interface device, which may employ any known transmission protocol.
[0104] The memory 701 may be integrated with the processor 702, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 701 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 701 and the processor 702 may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 702 can read files stored in the memory.
[0105] In addition, the electronic device 700 may further include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device 700 may be connected to each other via a bus and / or a network.
[0106] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium including instructions is also provided, such as a memory 701 including instructions, and the above instructions can be executed by a processor 702 of an apparatus 700 to complete the above method. Alternatively, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.
[0107] According to an exemplary embodiment of the present disclosure, a computer program product may also be provided, which includes a computer program / instruction. When the computer program / instruction is executed by a processor, the method of model training according to the exemplary embodiment of the present disclosure is implemented.
[0108] The above has been referred to Figures 1 to 7 The model training method and apparatus according to the exemplary embodiments of the present disclosure are described. However, it should be understood that: Figure 5 and Figure 6 The model training device and its units shown in the figure may be configured as software, hardware, firmware or any combination of the above items to perform specific functions, respectively. Figure 7 The electronic device shown in is not limited to including the components shown above, but some components may be added or deleted as needed, and the above components may also be combined.
[0109] According to the model training method and device disclosed in the present invention, an idea of "bidirectional degradation" is proposed. In addition to simulating the negative degradation of "low definition" in the model training data (for example, images used as model training data), the positive excessive degradation of "high definition" is also simulated. An over-sharp distorted input image is added to the model training data (for example, images used as model training data) with a small probability. A first image set and a first target image set corresponding to the first image set are obtained, wherein the first image set is an image set including unclear distorted images, a second image set and a second target image set corresponding to the second image set are obtained, wherein the second image set is an image set including sharp distorted images. The model is trained based on the first image set, the first target image set, the second image set, and the second target image set, so that the model can learn the distortion type more comprehensively, and the model has the ability to avoid over-sharp images, which can effectively alleviate the problem of over-sharpness in the output results of the trained model and improve the effect of model training.
[0110] According to the image processing method and device disclosed in the present invention, by inputting the image to be processed into the model trained using the model training method described in the present invention, and using the image output by the model as the processed image, the image processing effect can be improved by utilizing the model's ability to avoid oversharpening of the image.
[0111] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.
[0112] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims
1. A model training method, characterized in that: include: Acquire a first image set and a first target image set corresponding to the first image set, wherein the first image set is an image set including unclear and distorted images; Acquire a second image set and a second target image set corresponding to the second image set, wherein the second image set is an image set including sharp distortion images; The model is trained based on the first image set, the first target image set, the second image set, and the second target image set.
2. The model training method according to claim 1, characterized in that: The number of images included in the first image set and the number of images included in the second image set have a predetermined proportional relationship.
3. The model training method according to claim 1, characterized in that: The acquiring of the first image set comprises: Degradation processing is performed on at least part of the images in the first target image set to obtain a first image set.
4. The model training method according to claim 1, characterized in that: The acquiring of the second image set comprises: At least part of the images in the second target image set are sharpened to obtain a second image set.
5. The model training method according to claim 1, characterized in that: The step of acquiring a first target image set comprises: Assigning values to at least a portion of the training images in the training image set based on a predetermined sharpening probability; The training images in the predetermined training image set whose assigned values are less than or equal to a threshold are determined as first target images to obtain a first target image set, wherein the sharpening probability represents the probability that the training images in the training image set are sharpened.
6. The model training method according to claim 5, characterized in that: The acquiring of the second target image set comprises: The training images in the predetermined training image set whose assigned values are greater than the threshold are determined as second target images to obtain a second target image set.
7. The model training method according to claim 3, characterized in that: The degradation processing includes at least one of blurring processing, noise adding processing, compression processing, and sampling processing.
8. The model training method according to claim 1, characterized in that: The training of the model based on the first image set, the first target image set, the second image set, and the second target image set includes: Using images from the first image set and the second image set as inputs to the model, and using the model to predict output images; comparing a target image corresponding to the image in a first target image set and a second target image set with the output image; Parameters of the model are adjusted based on the comparison results.
9. An image processing method, characterized in that: include: Inputting the image to be processed into a model trained using the model training method according to any one of claims 1 to 8; The image output by the model is used as the processed image.
10. The image processing method according to claim 9, characterized in that: The models include models related to images and / or videos.
11. A model training device, characterized in that: include: A first image acquisition unit is configured to acquire a first image set and a first target image set corresponding to the first image set, wherein the first image set is an image set including unclear and distorted images; a second image acquisition unit configured to acquire a second image set and a second target image set corresponding to the second image set, wherein the second image set is an image set including a sharp distortion image; and The model training unit is configured to train the model based on the first image set, the first target image set, the second image set, and the second target image set.
12. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; Wherein, the processor is configured to execute the instructions to implement the model training method as described in any one of claims 1 to 10.
13. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor of an electronic device, the electronic device executes the model training method as described in any one of claims 1 to 10.
14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the model training method described in any one of claims 1 to 10 is implemented.