Image processing and model training method and device

Abstract

The invention discloses an image processing and model training method and device, and the method comprises the steps: determining a target coding mode of a first image; a target super-resolution model corresponding to the target coding mode is determined from multiple super-resolution models, the multiple super-resolution models correspond to different coding type groups, and the target coding mode is matched with the coding type group corresponding to the target super-resolution model; and processing the first image by using the target super-resolution model to generate a second image. The model training method comprises the following steps: determining respective to-be-trained super-resolution models of multiple coding type groups; for each coding mode of the coding type group, obtaining a first sample image coded by the coding mode and a corresponding second sample image; and for each coding type group, taking a first sample image corresponding to the coding type group as training data, taking a second sample image corresponding to the first sample image as a training target, and training the super-resolution model corresponding to the coding type group.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing and model training method and device. Background Art

[0002] Super-resolution reconstruction refers to the technology of restoring a super-resolution image with a higher resolution from a low-resolution image. However, the image quality of the super-resolution image obtained by reconstructing a low-resolution image based on super-resolution reconstruction technology is poor. Summary of the invention

[0003] On the one hand, the present application provides an image processing method, comprising:

[0004] Obtaining a decoded first image, and determining a target encoding mode corresponding to the first image;

[0005] Determining a target super-resolution model corresponding to the target encoding mode from a plurality of super-resolution models, wherein the plurality of super-resolution models correspond to different encoding type groups, the target encoding mode matches the encoding type group corresponding to the target super-resolution model, and encoding modes matched with the same encoding type group have at least one same encoding feature;

[0006] The first image is processed using the target super-resolution model to generate a second image.

[0007] In a possible implementation manner, determining a target super-resolution model corresponding to the target encoding mode from a plurality of super-resolution models includes:

[0008] Based on a mapping list, a target super-resolution model corresponding to the target encoding mode is determined, wherein the mapping list records at least two encoding modes respectively included in a plurality of encoding type groups, and a super-resolution model corresponding to each encoding type group.

[0009] In yet another possible implementation, determining, based on the mapping list, a target super-resolution model corresponding to the target encoding mode includes:

[0010] If the target encoding mode is found from the mapping list, a target super-resolution model corresponding to the target encoding mode is determined from the mapping list;

[0011] If the target encoding method is not found from the mapping list, determine from the mapping list a target encoding type group whose corresponding encoding method meets the requirements of similarity with the target encoding method, and determine the target super-resolution model corresponding to the target encoding type group in the mapping list.

[0012] In yet another possible implementation, determining the target super-resolution model corresponding to the target encoding type group in the mapping list includes:

[0013] If a plurality of target encoding type groups are determined from the mapping list, a plurality of target super-resolution models corresponding to the plurality of target encoding type groups in the mapping list are determined;

[0014] The step of processing the first image by using the target super-resolution model to generate a second image includes:

[0015] Using each target super-resolution model to process the first image respectively, and based on the weights corresponding to each target super-resolution model, fusing the candidate images obtained by processing the first image with each target super-resolution model to obtain a second image;

[0016] or,

[0017] A comprehensive super-resolution model is constructed based on the model parameters in each target super-resolution model, and the first image is processed using the comprehensive super-resolution model to generate a second image.

[0018] In another possible implementation, the super-resolution model recorded in the mapping list is a super-resolution model corresponding to the encoding type group trained by using the first sample image encoded using each encoding method in the encoding type group as training data and the second sample image corresponding to the first sample image as a training target, and the quality of the second sample image is higher than that of the first sample image.

[0019] In another possible implementation, the second sample image corresponding to the first sample image is: an unencoded original image corresponding to the first sample image, or an optimized image obtained by image optimization processing of the original image, wherein the quality of the original image corresponding to the first sample image is higher than the quality of the first sample image;

[0020] The optimized image is obtained by sharpening the detail area of ​​the original image.

[0021] In another possible implementation, in a loss function used for training the super-resolution model, different region weights are assigned to different regions in the second sample image, and different regions in the second sample image have different detail complexities;

[0022] And / or, the loss function used in training the super-resolution model includes a noise suppression term, in which different suppression weights are assigned to different pixel regions in the second sample image, wherein the greater the difference between the pixel region in the second sample image and the corresponding pixel region in the target image, the greater the suppression weight assigned to the pixel region in the second sample image, the pixel region includes at least one pixel, and the target image is an image generated by the super-resolution model by processing the first sample image;

[0023] And / or, the loss function used to train the super-resolution model corresponding to each encoding type group matches the encoding features of the encoding method corresponding to the encoding type group.

[0024] On the other hand, the present application also provides a model training method, comprising:

[0025] Determine super-resolution models to be trained corresponding to at least two encoding type groups, respectively, wherein encoding modes matching the same encoding type group have at least one common encoding feature;

[0026] For each encoding mode corresponding to each encoding type group, a first sample image encoded by the encoding mode corresponding to the encoding type group and a second sample image corresponding to the first sample image are obtained, wherein the quality of the second sample image is higher than the quality of the first sample image;

[0027] For each encoding type group, a first sample image corresponding to the encoding type group is used as training data, and a second sample image corresponding to the first sample image is used as a training target to train a super-resolution model corresponding to the encoding type group to obtain a trained super-resolution model corresponding to the encoding type group.

[0028] In a possible implementation manner, obtaining a first sample image encoded by the encoding method corresponding to the encoding type group, and a second sample image corresponding to the first sample image, includes:

[0029] Obtaining the original image without encoding;

[0030] Encoding the original image using the encoding method corresponding to the encoding type group to obtain a first sample image;

[0031] The original image or an optimized image obtained by optimizing the original image is determined as a second sample image corresponding to the first sample image.

[0032] In another possible implementation, taking a first sample image corresponding to the encoding type group as training data and a second sample image corresponding to the first sample image as a training target, training a super-resolution model corresponding to the encoding type group includes:

[0033] Using a first sample image corresponding to the encoding type group as training data, using a second sample image corresponding to the first sample image as a training target, and training a super-resolution model corresponding to the encoding type group in combination with a loss function;

[0034] Wherein, in the loss function, different area weights are assigned to different areas in the second sample image, and the detail complexity of different areas in the second sample image is different;

[0035] And / or, the loss function includes a noise suppression term, in which different suppression weights are assigned to different pixel regions in the second sample image, wherein the greater the difference between the pixel region in the second sample image and the corresponding pixel region in the target image, the greater the suppression weight assigned to the pixel region in the second sample image, the target image is an image generated by the super-resolution model by processing the first sample image, and the pixel region includes at least one pixel;

[0036] And / or, the loss function used to train the super-resolution model corresponding to each encoding type group matches the encoding features of the encoding method corresponding to the encoding type group.

[0037] In another aspect, the present application further provides an image processing device, comprising:

[0038] a type determination unit, configured to obtain the decoded first image and determine a target encoding mode corresponding to the first image;

[0039] a model determination unit, configured to determine a target super-resolution model corresponding to the target encoding mode from a plurality of super-resolution models, wherein the plurality of super-resolution models correspond to different encoding type groups, the target encoding mode matches the encoding type group corresponding to the target super-resolution model, and encoding modes matched with the same encoding type group have at least one same encoding feature;

[0040] An image processing unit is used to process the first image using the target super-resolution model to generate a second image.

[0041] On the other hand, the present application also provides a model training device, comprising:

[0042] A model determination unit, used to determine super-resolution models to be trained corresponding to at least two encoding type groups, wherein encoding modes matching the same encoding type group have at least one common encoding feature;

[0043] a sample obtaining unit, configured to obtain, for each encoding mode corresponding to each encoding type group, a first sample image encoded by the encoding mode corresponding to the encoding type group, and a second sample image corresponding to the first sample image, wherein the quality of the second sample image is higher than that of the first sample image;

[0044] A model training unit is used to train a super-resolution model corresponding to the encoding type group for each encoding type group, using a first sample image corresponding to the encoding type group as training data and a second sample image corresponding to the first sample image as a training target, so as to obtain a trained super-resolution model corresponding to the encoding type group. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0046] Figure 1 A schematic diagram of a flow chart of the image processing method provided in this application;

[0047] Figure 2 A flowchart of the model training method provided in this application;

[0048] Figure 3 A schematic diagram of another flow chart of the model training method provided in this application;

[0049] Figure 4 This is an example diagram of an implementation framework for training a super-resolution model corresponding to a single encoding method in this application;

[0050] Figure 5 A schematic diagram of another process flow of the image processing method provided by the present application;

[0051] Figure 6 A schematic diagram of the structure of an image processing device provided by the present application;

[0052] Figure 7 A schematic diagram of the structure of a model training device provided in this application;

[0053] Figure 8 A schematic diagram of the composition architecture of an electronic device provided in this application. DETAILED DESCRIPTION

[0054] The embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation mode of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application. It is known to those skilled in the art that with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0055] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0056] like Figure 1 , showing a flow chart of the image processing method provided in an embodiment of the present application. The method of this embodiment is applied to an electronic device, which may be a laptop computer, a desktop computer, a platform computer, etc., or a device node in the cloud or a distributed cluster, without limitation.

[0057] The method of this embodiment may include the following steps S101 to S103:

[0058] S101, obtaining a decoded first image, and determining a target encoding mode corresponding to the first image.

[0059] The first image is an image that needs to be reconstructed using a super-resolution model. Since the first image is decoded from an image that has been compressed, it is affected by the encoder, such as noise introduced during the image compression encoding process by the encoder, so that the image quality of the decoded first image is relatively low. For example, the image resolution of the first image is relatively low and there is more noise.

[0060] The target encoding method is the encoding method used when encoding the first image. It can be understood that, since the types of encoders used to encode the first image are different, the encoding methods used to encode the first image are also different. Therefore, the target encoding method is also the encoding method corresponding to the encoder used to encode the first image.

[0061] In the present application, there may be multiple encoding methods, such as, the encoding method may include but is not limited to Joint Photographic Experts Group (JPEG) encoding, High Efficiency Video Coding (HEVC) or H.264 (Advanced Video Coding of Moving Picture Experts Group-4), etc., among which High Efficiency Video Coding is also called H.265 encoding.

[0062] Wherein, determining the target encoding method corresponding to the first image may be by analyzing the metadata of the first image to determine the target encoding method corresponding to the first image. Wherein, the metadata of the first image may include at least the encoding type of the first image, and may also include other attribute information of the first image, without specific limitation. For example, since the first image is obtained by decoding the decoded image, the metadata of the encoded image can be obtained, and the target encoding method of the encoded image is determined based on the metadata of the encoded image, thereby obtaining the target encoding method corresponding to the first image. Wherein, the metadata of the encoded image is actually the metadata of the first image.

[0063] The target encoding method can also be determined by analyzing the extension of the first image; or, by using a specific encoding detection tool to detect the target encoding method corresponding to the first image; or, by analyzing the compression noise pattern or frequency domain characteristics corresponding to the first image and other characteristics related to image encoding, the target encoding method corresponding to the first image is determined, without specific limitation.

[0064] It should be noted that in the present application, the first image can be an independent single-frame image or an image frame in a video, without any specific limitation.

[0065] S102, determining a target super-resolution model corresponding to the target encoding method from a plurality of super-resolution models.

[0066] Among them, multiple super-resolution models correspond to different encoding type groups, and the target encoding method matches the encoding type group corresponding to the target super-resolution model. For the convenience of distinction, the super-resolution model corresponding to the target encoding method is called the target super-resolution model, and therefore, the target super-resolution model belongs to the multiple super-resolution models.

[0067] In the present application, the encoding methods matched with the same encoding type group have at least one identical encoding feature. For example, the encoding methods matched with the same encoding type group may include at least two encoding methods, and the at least two encoding methods may have one or more identical encoding features. Based on this, the present application actually matches the encoding methods with at least one identical encoding feature into the same encoding type group, and the various encoding methods matched to the same encoding type group can share the same super-resolution model, so there is no need to construct a super-resolution model for each encoding method separately, which naturally reduces the number of super-resolution models that need to be constructed and stored.

[0068] In a possible case, possible encoding type groups can be pre-divided, so the encoding methods included in each encoding type group are predetermined, and of course, the encoding methods included in the same encoding type group have at least one common feature. In this case, after determining the target encoding method corresponding to the first image, a target encoding type group including the target encoding method can be determined, and a target super-resolution model corresponding to the target encoding type group can be determined.

[0069] In another possible case, the present application may only determine that each encoding type group corresponds to at least one encoding feature, but the encoding methods included in the encoding type group are not pre-divided. In this case, after determining the target encoding method corresponding to the first image, the present application may determine the target encoding type group matched to the target encoding method based on the encoding features of the target encoding method, and determine the target super-resolution model corresponding to the target encoding type group.

[0070] The encoding method may have encoding features in various dimensions. For example, the encoding features of the encoding method may include but are not limited to encoding algorithm principles, encoding complexity, and encoding applicable scenarios.

[0071] Take the example of determining different coding type groups based on the coding feature of the coding algorithm principle:

[0072] The coding algorithm principles corresponding to different coding type groups are different. For example, the coding algorithm principles corresponding to different coding type groups may be: transform-based coding (also called transform coding), block prediction-based coding, or entropy-based coding.

[0073] For example, for the coding type group that matches the coding feature of transform-based coding, the coding method that matches it can adopt transform-based coding. Therefore, the coding methods that match this coding type group may include but are not limited to: JPEG coding, AV1 (Open Media Video 1) coding, and intra-frame coding (VCC Intra) in Versatile Video Coding (VCC) and other coding methods.

[0074] For the coding type group that matches the coding feature of block prediction-based coding, the matching coding method can adopt block prediction-based coding. Therefore, the coding methods that match this coding type group may include but are not limited to: H.264 coding, HEVC coding and inter-frame coding (VCC Inter) in VCC.

[0075] Similarly, for the coding type group that matches each coding method based on entropy coding, the coding method may include but is not limited to: Harmanff coding and Shannon coding. In the present application, the target super-resolution model is suitable for super-resolution reconstruction of images encoded by various coding methods matching the coding type group corresponding to the target super-resolution model. Among them, for the convenience of distinction, the coding type group corresponding to the target super-resolution model can be referred to as the target coding type group. For example, the target super-resolution model can perform targeted noise reduction on an image encoded by any coding method corresponding to the target coding type group, reducing the corresponding type of noise introduced in the image due to encoding by the encoder of any coding method corresponding to the target coding type group.

[0076] The target super-resolution model is a trained model, and there may be many possible model types of the target super-resolution model. For example, the target super-resolution model may be a convolutional neural network model, a deep recursive convolutional network, or a generative adversarial network model, etc., without any restriction.

[0077] In the present application, there is no restriction on the specific implementation of the target super-resolution model obtained by training. For example, in one possible implementation, the target super-resolution model is a super-resolution model corresponding to the target encoding type group trained by using a first sample image encoded in an encoding method corresponding to the target encoding type group as training data and a second sample image corresponding to the first sample image as a training target. The quality of the second sample image is higher than the quality of the first sample image.

[0078] The fact that the quality of the second sample image is higher than that of the first sample image can be reflected in multiple aspects. For example, the fact that the quality of the second sample image is higher than that of the first sample image can at least indicate that the resolution of the second sample image is higher than that of the first sample image; it can also indicate that: the noise in the second sample image is less than that in the first sample image, and the edge sharpness and detail richness of the second sample image are higher than those of the first sample image.

[0079] S103: Process the first image using a target super-resolution model to generate a second image.

[0080] The second image is a high-quality image generated by the target super-resolution model based on the first image, and therefore, the quality of the second image is higher than that of the first image.

[0081] For example, the higher quality of the second image than the first image can at least indicate that the resolution of the second image is higher than the resolution of the first image. Of course, the higher quality of the second image than the first image can also indicate that the noise contained in the second image is less than the noise contained in the first image.

[0082] It is understandable that after generating the second image, the present application can also output the second image so that the user can obtain the second image with relatively high image quality. Of course, in actual applications, after generating the second image, the second image can also be saved according to actual needs, which will not be described in detail.

[0083] From the above content, it can be seen that after decoding the first image, the present application will determine the target encoding method corresponding to the first image, and use the target super-resolution model corresponding to the target encoding type group matching the target encoding method to process the first image, thereby being able to select a target super-resolution model suitable for processing the first image encoded by the target encoding method, so that the target super-resolution model can more effectively remove the noise or other influences introduced by the first image during the encoding process, thereby improving the image quality of the generated second image.

[0084] Moreover, in the present application, each encoding method matched to the same encoding type group has at least one identical encoding feature, so that the encoding features of each encoding method matched to the same encoding type group are similar, so that each encoding method matched to the same encoding type group can share the same super-resolution model, which not only improves the breadth of encoding methods applicable to a single super-resolution model, but also can reduce the number of super-resolution models that need to be built and stored.

[0085] In a possible implementation, in order to efficiently determine the super-resolution model corresponding to the coding type group that matches the coding mode and reduce the time required to determine the super-resolution model corresponding to the coding mode, the present application can also construct a mapping list, in which at least two coding modes included in each of the multiple coding type groups and the super-resolution model corresponding to each coding type group are recorded. In this mapping list, different super-resolution models correspond to different coding type groups.

[0086] On this basis, the present application can determine the target super-resolution model corresponding to the target encoding method based on the mapping list.

[0087] For example, the mapping list records the group identifier of each encoding type group, at least two encoding methods corresponding to the group identifier of each encoding type group, and the model identifier of the super-resolution model corresponding to the group identifier of each encoding type group. The group identifier may be a coding feature common to each encoding method in the encoding type group, or a group serial number corresponding to the encoding type group, etc., which is used to uniquely identify the encoding type group, without specific limitation. Similarly, the model identifier of the super-resolution model may be the model name, number, or call address of the super-resolution model, etc., without specific limitation.

[0088] By querying the mapping list to determine the target super-resolution model corresponding to the target encoding method, the time consumed by calculating the target encoding type group that matches the encoding features of the target encoding method can be reduced, and the efficiency of determining the target super-resolution model can naturally be improved, thereby reducing the time consumed by processing the first image to be reconstructed, thereby improving the efficiency of super-resolution image reconstruction of the first image.

[0089] In the present application, there may be multiple possible model structures of the super-resolution model corresponding to each encoding type group. For details, please refer to the related introduction of the possible model structure of the target super-resolution model above, which will not be repeated here.

[0090] In the present application, each super-resolution model can be pre-trained, and the specific training process is not limited. By training each super-resolution model, the same super-resolution model can perform different super-resolution processing effects on images encoded using encoding methods corresponding to different encoding type groups.

[0091] In a possible implementation, in order to make the super-resolution model corresponding to each coding type group pay more attention to the influence of noise and the like introduced by each coding method (or the encoder corresponding to each coding method) in the coding type group on image coding, for the super-resolution model corresponding to any coding type group, the super-resolution model can be a super-resolution model corresponding to the coding type group trained by using the first sample image encoded by each coding method in the coding type group as training data and the second sample image corresponding to the first sample image as training target. As mentioned above, the quality of the second sample image is higher than the quality of the first sample image.

[0092] Among them, for any encoding type group, each first sample image corresponds to an encoding method in the encoding type group, and there can be multiple first sample images for training the super-resolution model, and the encoding methods corresponding to multiple first sample images can include matching at least two encoding methods in the encoding type group.

[0093] For ease of understanding, the following Figure 2The flowchart shown illustrates an implementation method for training super-resolution models corresponding to various coding type groups. Figure 2 A flow chart for implementing the model training method provided in an embodiment of the present application is shown. The model training method may include the following steps S201 to S203:

[0094] S201, determining super-resolution models to be trained corresponding to at least two encoding type groups respectively.

[0095] The encoding modes matching the same encoding type group have at least one common encoding feature. For example, before training the super-resolution model, at least two encoding modes matching the encoding type group can be determined for each encoding type group.

[0096] The model types of the super-resolution models to be trained corresponding to different encoding type groups can be the same or different, and can be selected according to actual needs without limitation. For the model types that can be selected for the super-resolution model, please refer to the previous related introduction, which will not be repeated here.

[0097] S202: For each encoding mode corresponding to each encoding type group, obtain a first sample image encoded by the encoding mode corresponding to the encoding type group, and a second sample image corresponding to the first sample image.

[0098] The first sample image may be an image obtained by encoding an original image with image quality meeting the requirement through an encoder corresponding to an encoding method in the encoding type group, wherein the quality of the original image is higher than the quality of the first sample image.

[0099] In an implementation manner, each encoding type group may correspond to a plurality of first sample images, and the plurality of first sample images include at least one first sample image encoded by each encoding method of the encoding type group.

[0100] For example, in the case where all encoding modes in the encoding type group belong to encoding based on transform compression, assuming that the encoding type group includes two encoding modes, JPEG and AV1, a JPEG encoder can be used to compress and encode at least one original image whose resolution exceeds a set threshold to obtain at least one first sample image corresponding to the JPEG encoding mode; and an AV1 encoder is also required to compress and encode at least one original image whose resolution exceeds a set threshold to obtain at least one first sample image corresponding to the AV1 encoding mode. Each first sample image has the same object content as its corresponding original image, but the resolution of the first sample image is lower than that of the original image.

[0101] The quality of the second sample image is higher than the quality of the first sample image.

[0102] Among them, the goal of training the super-resolution model corresponding to the encoding type group is to expect that the super-resolution model corresponding to the encoding type group can generate the second sample image based on the first sample image. Therefore, the second sample image is the training target for training the super-resolution model corresponding to the encoding type group.

[0103] S203, for each encoding type group, using the first sample image corresponding to the encoding type group as training data, and the second sample image corresponding to the first sample image as a training target, training the super-resolution model corresponding to the encoding type group to obtain a trained super-resolution model corresponding to the encoding type group.

[0104] Among them, the second sample image corresponding to the first sample image is used as the training target, which essentially hopes to minimize the gap between the target image generated by the super-resolution model based on the first sample image and the first sample image, so that the target image generated by the super-resolution model based on the first sample image is as similar as possible to the second sample image.

[0105] From the above content, it can be seen that for each encoding type group, the present application will use the first sample images encoded by each encoding method in the encoding type group as training data, and use the second sample images with relatively high quality corresponding to the first sample images as training targets, and train to obtain the super-resolution model corresponding to the encoding type group, so that the super-resolution model corresponding to each encoding type group can reconstruct the image encoded by any encoding method in the encoding type group into a high-quality image.

[0106] In addition, although the encoders corresponding to each encoding method in each encoding type group will introduce noise when encoding the image, the types of noise introduced by the encoders corresponding to the encoding methods matched to the same encoding type group are similar. Therefore, the same super-resolution model is used to reconstruct the super-resolution images after the images encoded by the encoders corresponding to the encoding methods in each encoding type group. This can not only enable the super-resolution model to effectively remove the specific type of noise introduced by the encoder corresponding to the encoding type group and improve the quality of the reconstructed super-resolution image; it can also effectively reduce the problems of a large number of models and high maintenance costs caused by training a super-resolution model for each encoding method in the encoding type group.

[0107] It is understandable that in the training process of the super-resolution model corresponding to each encoding type group, it is necessary to combine the corresponding loss function to control the training of the super-resolution model. The loss function is used to quantify the difference between the target image generated by the super-resolution model based on the first sample image and the second sample image corresponding to the first sample image. For example, combining the function value of the loss function can determine whether the super-resolution model has achieved the training goal, and adjust the model parameters of the super-resolution model.

[0108] Based on this, in the present application, for each encoding type group, the first sample image corresponding to the encoding type group can be used as training data, and the second sample image corresponding to the first sample image can be used as a training target, and the super-resolution model corresponding to the encoding type can be trained in combination with the loss function.

[0109] The loss function can be selected according to actual needs without any specific restrictions.

[0110] In one possible case of the loss function, in the loss function used for training the super-resolution model, different region weights are assigned to different regions in the second sample image, wherein the detail complexity of different regions in the second sample image is different.

[0111] Each region in the second sample image has a one-to-one correspondence with a corresponding region in the target image generated by the super-resolution model based on the first sample image. Based on this, the function value of the loss function is related to the difference between the target image and the second sample image in different regions and the region weights corresponding to different regions.

[0112] Unlike a conventional loss function whose function value is only related to the overall gap between the target image generated by the super-resolution model and the second sample image, in the present application, different regions of the second sample image are assigned different regional weights in the loss function, thereby enhancing the learning ability of the super-resolution model for regions that meet the requirements for certain detail complexity during the training process of the super-resolution model.

[0113] There may be multiple division bases for different regions in the second sample image. Several possible situations for dividing different regions in the second sample image are described below:

[0114] For example, in a possible case, the second sample image can be divided into detail areas and flat areas. On this basis, different regional weights are assigned to the detail areas and flat areas in the second sample image in the loss function, and the regional weight of the detail area is greater than the regional weight of the flat area, so that in the process of training the super-resolution model, the super-resolution model can learn more detail information in the second sample image, thereby enhancing the sharpness of the detail area of ​​the generated image; and the flat area of ​​the generated image can be kept smooth to reduce new noise introduced into the generated image due to the sharpening process.

[0115] Among them, the detail area of ​​the image refers to the area in the image where the pixel value changes dramatically and contains rich information. The detail area of ​​the image usually contains edges, textures, patterns or complex structures. The flat area of ​​the image refers to the area in the image where the pixel value changes slowly and contains less information. For example, the area in the image where the detail complexity is not less than the set threshold can be the detail area, and the area where the detail complexity is less than the set threshold can be the flat area. The image can include one or more detail areas, and accordingly, the image can also include one or more flat areas.

[0116] In the present application, there is no limitation on the specific implementation of determining the detail area and the flat area in the image. For ease of understanding, two implementations are used for illustration.

[0117] For example, in one implementation, an edge detection algorithm may be used to detect edge areas and non-edge areas in an image. Edge areas are usually detail areas, while non-edge areas are flat areas. The specific implementation process may be: edge detection is performed on an image to obtain a binary edge map, and the area where white pixels in the edge map are located is determined as a detail area, and the area where black pixels are located is determined as a flat area.

[0118] In another implementation, a texture analysis method can be used to determine the detail area and flat area of ​​the image based on the texture complexity of the image, wherein the texture complexity reflects the texture richness of the local area of ​​the image, and the area with a texture complexity not lower than a set complexity threshold is a detail area, and the area with a texture complexity lower than the complexity threshold is a flat area. For example, a texture feature extraction method such as a local binary pattern and a gray level co-occurrence matrix can be used to extract texture feature values ​​of different pixel areas in the image, and the texture feature values ​​can characterize the texture complexity. Accordingly, the detail area and flat area of ​​the image are determined in combination with the set texture feature threshold.

[0119] Of course, there may be other implementations for determining the detail area and the flat area in the image, which will not be described in detail here.

[0120] In yet another possible scenario, based on the object in the second sample image, the second sample image is divided into an object region having the object and a background region having no object, so as to obtain a plurality of different regions in the second sample image.

[0121] The object region and the background region in the second sample image may be determined by using an edge detection algorithm or an object recognition model, etc., without any specific limitation. For example, taking the edge detection algorithm to determine the object region and the background region in the second sample image as an example, the edge of each object in the second sample image may be detected based on the edge detection algorithm to obtain the object region of each object in the second sample image and the background region outside the object region.

[0122] In another possible situation, the target of interest in the second sample image can be set as needed, and the region of interest and the region of non-interest in the second image can be determined to obtain multiple different regions in the second sample image. For example, the region of interest and the region of non-interest in the second sample image can be identified in combination with a recognition model, or the second sample image can be semantically segmented based on a semantic segmentation algorithm to segment the region of interest and the region of non-interest in the second sample image. Of course, there can be other ways to determine the region of interest and the region of non-interest in the second sample image, which are not limited thereto.

[0123] In the present application, the specific form of the loss function may be in many possible forms. For example, the loss function may be a pixel-level loss function, in which at least one of the absolute difference and the square difference of the pixel values ​​at pixel points at different positions of the target image and the second sample image is calculated. For another example, the loss function may be a perceptual loss function, in which the distance between the target image and the second sample image in the feature space may be calculated. For another example, the loss function may be a gradient loss function. Of course, the loss function may also have other forms, which are not limited thereto.

[0124] In order to facilitate the understanding of the specific implementation of assigning different weights to different regions of the second sample image in the loss function, the following takes the loss function as a pixel-level loss function and combines a specific function form of the loss function as an example. The following formula is the loss function An expression of:

[0125]

[0126] in, is the total number of pixels in the second sample image (or target image).

[0127] and are the height and width of the second sample image, that is, the number of pixels in each column and the number of pixels in each row in the second sample image;

[0128] is the second sample image in the Ledi The weights corresponding to the pixels on the row.

[0129] The target image generated by the super-resolution model Ledi The pixel value of the pixel on the row.

[0130] is the first Ledi The pixel value of the pixel on the row.

[0131] In the present application, the weights of different regions in the second sample image or the target image are different. Therefore, in the above formula, the weight of the pixel point is is the area weight corresponding to the area of ​​the pixel point in the second sample image (or target image). It can be seen that the weights corresponding to the pixels in different areas of the second sample image Not the same either.

[0132] In another possible case, in order to reduce the noise that may be amplified by the super-resolution model in the process of reconstructing the super-resolution image, in the present application, the loss function may include a noise suppression term, in which different suppression weights are assigned to different pixel regions in the second sample image, wherein each pixel region includes at least one pixel.

[0133] For any pixel region in the second sample image, the greater the difference between the pixel region and the corresponding pixel region in the target image, the greater the suppression weight assigned to the pixel region. The target image is an image generated by the super-resolution model by processing the first sample image.

[0134] The corresponding pixel region in the target image is a pixel region whose coordinate region is the same as that of the pixel region in the second sample image. Based on this, the suppression weight corresponding to the pixel region in the second sample image is actually the suppression weight corresponding to the corresponding pixel region in the target image.

[0135] The value of the noise suppression term is the weighted sum of the differences between the target image and the second sample image in each pixel area, determined based on the suppression weight of each pixel area in the second sample image. The larger the value of the noise suppression term, the larger the function value of the loss function.

[0136] It is understandable that the greater the difference between the target image and the second sample image in the same pixel area, the more noise there is in the pixel area in the target image. In order to reduce the noise in each pixel area in the target image during the super-resolution model training process, the present application assigns a larger weight (i.e., suppression weight) to the pixel area with a larger difference between the target image and the second sample image, thereby increasing the super-resolution model's learning of the pixel area with a larger difference between the target image and the second sample image, so as to reduce the noise in the pixel area in the target image.

[0137] Among them, as the super-resolution model is trained, the difference between the target image and the second sample image in different pixel areas may change. Therefore, as the difference between the target image and the second sample image in different pixel areas changes, the suppression weights corresponding to different pixel areas will also be adaptively adjusted.

[0138] Still taking the above formula as an example, as the super-resolution model is continuously trained, the gap between the target image and the second sample image in different pixel areas will change, causing the suppression weight corresponding to the pixel area to change, making the weight of each pixel in the pixel area Will also change accordingly.

[0139] It can be understood that, in practical applications, the loss function for training the super-resolution model can include the above two possible situations, that is, while different regional weights are assigned to different regions in the second sample image in the loss function, the loss function also includes the loss suppression term mentioned above. For example, in the loss function, the weight assigned to a certain pixel in the second sample image includes weight a and weight b. Among them, weight a is a fixed weight assigned to the area where the pixel in the second sample image is located. And weight b is a suppression weight determined based on the difference between the pixel in the second sample image and the corresponding pixel in the target image. Therefore, as the super-resolution model is continuously trained, weight b can be a changing weight value.

[0140] In another possible scenario, considering that in the present application, the first sample image can be a static image containing static objects such as scenery or objects, or a dynamic image containing dynamic objects such as animals or people, or a dynamic image derived from a video, etc., in order to more reasonably train the super-resolution model corresponding to each encoding type group, different loss functions can be determined based on the proportion of static images and dynamic images in the first sample image corresponding to the encoding type group.

[0141] In another possible scenario, the loss function used to train the super-resolution model corresponding to each encoding type group matches the encoding features of the encoding method corresponding to the encoding type group. Based on this, for any encoding type group, if the same encoding features of the encoding methods in the encoding type group are different, the loss function selected for training the super-resolution model corresponding to the encoding type group may also be different. Specifically, a suitable loss function can be determined in combination with the edge features shared by the encoding methods in the encoding type group. It can be understood that the loss function for training the super-resolution model can include one or more of the above multiple possible situations, without specific limitation.

[0142] In the present application, there are many possible ways to obtain the second sample image.

[0143] For example, in a possible case, the second sample image may be the original image corresponding to the first sample image mentioned above, that is, the unencoded original image corresponding to the first sample image.

[0144] In this case, for each encoding type group, an unencoded original image can be obtained first. Then, the original image is encoded using an encoder of the encoding method corresponding to the encoding type group to obtain the first sample image. Accordingly, the original image corresponding to the first sample image can be determined as the second sample image.

[0145] It is understandable that by compressing and encoding the original image, not only the resolution of the original image will be reduced, but also compression noise and the like may be introduced into the original image, so that the quality of the generated first sample image is lower than that of the corresponding original image. However, the present application uses the first sample image as training data to train the super-resolution model, and its purpose is to enable the super-resolution model to restore the first sample image to the original image by training the super-resolution model.

[0146] In another possible scenario, the second sample image is an optimized image obtained by performing image optimization processing on the original image corresponding to the first sample image.

[0147] It is understandable that the image quality of the original image can be improved by optimizing the original image, so the quality of the optimized image is not only higher than the quality of the first sample image, but also higher than the quality of the second sample image. Based on this, using the optimized image as the second sample image corresponding to the first sample image to train the super-resolution model is more conducive to improving the quality of the image generated by the super-resolution model.

[0148] The method of optimizing the original image may be any processing method that can improve the quality of the original image, and there is no limitation on this.

[0149] In an optional manner, considering that the information in the detail area of ​​the image is richer, in the present application, the optimized image can be obtained by sharpening the detail area of ​​the original image. In the process of sharpening the detail area of ​​the original image, the flat area in the original image remains unchanged.

[0150] On this basis, since the optimized image is only obtained by sharpening the detail area of ​​the original image, using the optimized image as the second sample image corresponding to the first sample image to train the super-resolution model can enhance the ability of the super-resolution model to learn the detail information in the image, which can not only improve the quality of the image generated by the super-resolution model; it can also avoid excessive processing of flat areas in the image to a certain extent, and reduce the introduction of new noise due to the sharpening and enhancement of flat areas of the image by the super-resolution model during the image generation process.

[0151] For ease of understanding, the model training method of the present application is described below by taking the second sample image as an optimized image corresponding to the first sample image as an example. Figure 3 , shows another flow chart of the model training method provided by the present application, and this embodiment may include:

[0152] S301, determining super-resolution models to be trained corresponding to at least two encoding type groups respectively.

[0153] S302: For each encoding type group, obtain an original image that has not been encoded.

[0154] The original image is a high-quality image that has not been encoded, such as an original image that has not been encoded by any encoding method in the encoding type group.

[0155] For example, for each encoding type group, at least one original image can be obtained.

[0156] S303: Encode the original image using the encoding method corresponding to the encoding type group to obtain the first sample image.

[0157] For example, the original image is encoded using the encoder corresponding to each encoding method in the encoding type group to obtain at least one first sample image corresponding to each encoding method in the encoding type group, and the original images corresponding to the first sample images are the same.

[0158] It can be understood that, after encoding the original image, the encoder corresponding to each encoding method not only reduces the resolution of the original image, but also introduces noise of the noise type corresponding to the encoder. Therefore, the quality of the first sample image obtained by encoding the original image using any encoding method is lower than the quality of the original image.

[0159] S304, performing sharpening processing on the flat area in the original image, and keeping the flat area in the original image unchanged, to obtain an optimized image obtained by processing the original image.

[0160] By sharpening the flat areas of the original image, the edges and details in the original image can be enhanced, thereby improving the image quality. Moreover, only sharpening the flat areas in the original image can introduce as little new noise as possible, so that the optimized image has higher image quality than the original image.

[0161] S305, taking the first sample image corresponding to the encoding type group as training data, taking the optimized image as a training target, and training the super-resolution model corresponding to the encoding type group in combination with the loss function, to obtain a trained super-resolution model corresponding to the encoding type group.

[0162] Since the optimized image has higher image quality than the original image, using the optimized image as the training target for training the super-resolution model can further improve the quality of the images generated by the super-resolution model through continuous training.

[0163] The loss function may be any one or a combination of the aforementioned possible situations, which will not be described in detail.

[0164] In a possible implementation, the present application may also train the super-resolution model corresponding to some commonly used encoding methods separately, thereby obtaining a super-resolution model corresponding to at least one encoding method. On this basis, after determining the target encoding method corresponding to the first image, the present application may first query whether there is a super-resolution model corresponding only to the target encoding method from the super-resolution models corresponding to at least one encoding method. If there is a super-resolution model corresponding only to the target encoding method, the super-resolution model is determined as the target super-resolution model.

[0165] If there is no super-resolution model corresponding only to the target encoding method, the target super-resolution model corresponding to the target encoding method may be determined based on the mapping list.

[0166] The specific implementation of training the super-resolution model corresponding to a single encoding method is similar to the process of training the super-resolution model corresponding to the encoding type group, except that when training the super-resolution model corresponding to a single encoding method, the first sample image used to train the super-resolution model only includes sample images encoded by the encoding method (or encoded by the encoder corresponding to the encoding method). Figure 4The example diagram of the model training implementation framework shown is a brief description of the process of training a super-resolution model corresponding to a single encoding method.

[0167] Depend on Figure 4 It can be seen that after collecting at least one high-quality original image, for each encoding method that needs to train the super-resolution model, the encoder of the encoding method can be used to compress and encode each original image to generate at least one low-quality image corresponding to the encoding method. Among them, part of the low-quality images in the at least one low-quality image is used as training data to form a training data set. The remaining part of the low-quality images in the at least one low-quality image is used as verification data to construct a verification data set.

[0168] In addition, before training the super-resolution model of this encoding method, the key areas (such as detail areas) of the original image are optimized (such as sharpening), and other areas (such as flat areas) are kept unchanged to obtain an optimized image ( Figure 4 golden image in the ).

[0169] On this basis, for each encoding method, the super-resolution model of the encoding method can be trained using low-quality images in the training data set. A noise suppression term can be added to the loss function used to train the super-resolution model. Moreover, in the loss function, different regional weights can be assigned to the key areas (such as detail areas) and non-key areas (such as flat areas) of the optimized image, so as to increase the learning ability and denoising ability of the super-resolution model for key areas such as detail areas.

[0170] Before training the super-resolution model in combination with the function value of the loss function, the model parameters of the super-resolution model may also be initialized.

[0171] During the training process, the low-quality image used as training data is input into the super-resolution model to be trained, so that the super-resolution model outputs the target image (i.e., the forward propagation process). On this basis, the function value of the loss function is calculated based on the target image and the optimized image. Based on the function value of the loss function, the model parameters of the super-resolution model are updated through back propagation. This process is repeated many times to achieve iterative training of the super-resolution model.

[0172] After training a super-resolution model corresponding only to the encoding mode, the super-resolution model can be verified using low-resolution images in the validation set and the model can be optimized. On this basis, the super-resolution model finally trained can be quantized, and the quantized super-resolution model can be stored in a model library, so that the model library can include a super-resolution model applicable to an encoder of at least one encoding mode.

[0173] It is understandable that after determining the target encoding mode corresponding to the first image, if the target encoding mode is queried from the mapping list, the target super-resolution model corresponding to the target encoding mode can be directly determined from the mapping list.

[0174] However, since there are many types of encoding methods, and the encoding methods that can be recorded in the mapping list are limited, after determining the target encoding method corresponding to the first image, it may happen that the target encoding method does not exist in the mapping list. In this case, the present application can also determine from the mapping list a target encoding type group whose similarity with the corresponding encoding method meets the requirements. Accordingly, the target resolution model corresponding to the target encoding type group in the mapping list can be determined, so as to process the first image using the target super-resolution model corresponding to the target encoding type group.

[0175] For example, the coding type group including the coding methods with the highest similarity to the target coding method can be determined as the target coding type group; or, the coding type group including the coding methods with a similarity exceeding a set threshold to the target coding method can be determined as the target coding type group.

[0176] There are multiple implementations for determining the similarity between the encoding method corresponding to the encoding type group and the target encoding method.

[0177] For example, in a possible implementation, for each encoding type group in the mapping list, the similarity between the target encoding method and each encoding method in the encoding type group can be calculated respectively. Accordingly, the encoding type group where the encoding method whose similarity with the target encoding method meets the requirement is located is determined as the target encoding type group.

[0178] Calculating the similarity between the target encoding method and the encoding methods in the encoding type group may be calculating the vector similarity between the vector corresponding to the target encoding method and the vector corresponding to the encoding methods in the encoding type group. The vector corresponding to the target encoding method may be a vector converted from the name of the target encoding method, or the encoding features of the target encoding method are obtained, such as the encoding features of the target encoding method may include encoding characteristics such as encoding principles or encoding applicable scenarios, and the vector corresponding to the target encoding method is determined based on the encoding features of the target encoding method. Determining the vector corresponding to the encoding method in the encoding type group is similar and will not be repeated.

[0179] In another possible implementation, for each encoding type group in the mapping list, the similarity between the target encoding method and each encoding method in the encoding type group is calculated respectively. On this basis, for each encoding type group, the similarity average value or weighted average value of the similarities corresponding to each encoding method in the encoding type group is determined, and the corresponding similarity average value or weighted average value is determined as the similarity between the encoding method corresponding to the encoding type group and the target encoding method, thereby determining the target encoding type group whose similarity between the corresponding encoding method and the target encoding method meets the requirements. For example, the encoding type group with the largest corresponding similarity average value is determined as the target encoding type group.

[0180] It is understandable that there may be multiple coding type groups whose corresponding coding methods meet the requirements of similarity with the target coding method. In this case, the present application can randomly select a coding type group from the determined multiple coding type groups as the target coding type group.

[0181] In an optional manner, when it is determined that there are multiple coding type groups whose similarity between the corresponding coding mode and the target coding mode meets the requirements, the present application can also determine all of the multiple coding type groups as target coding type groups. On this basis, the present application can combine the target super-resolution models corresponding to the multiple target coding type groups to comprehensively process the first image.

[0182] Combine the following Figure 5 For explanation. Figure 5 Another schematic diagram of a process of the image processing method provided by the present application is shown. The method of this embodiment may include:

[0183] S501, obtaining a decoded first image, and determining a target encoding mode corresponding to the first image.

[0184] S502, query whether the target encoding method exists in the mapping list, if yes, execute step S503; if not, execute step S504.

[0185] The mapping list records at least two encoding modes included in each of the multiple encoding type groups, and a super-resolution model corresponding to each encoding type group.

[0186] S503, determining a target super-resolution model corresponding to the target encoding mode from the mapping list, and executing step S506.

[0187] S504: Determine from the mapping list a target coding type group whose similarity between the corresponding coding mode and the target coding mode meets the requirement.

[0188] In this embodiment, if the target encoding method is not queried from the mapping list, a target encoding type group whose similarity between the corresponding encoding method and the target encoding method meets the requirements will be determined based on the encoding methods included in each encoding type group, so as to match the encoding type that contains the encoding method similar to the target encoding method, that is, to determine the target encoding type group that may be matched by the encoding features of the target encoding method.

[0189] For example, a target encoding type group including encoding modes having the highest similarity with the target encoding mode may be determined from the mapping list.

[0190] S505, if a target encoding type group whose similarity between the corresponding encoding method and the target encoding method meets the requirements is determined, determine the target super-resolution model corresponding to the target encoding type group in the mapping list, and execute step S506.

[0191] S506: Input the first image into the target super-resolution model to obtain a second image generated by the target super-resolution model.

[0192] In this embodiment, if the target encoding method is not found in the mapping list, it will be detected whether there is a corresponding encoding method in the mapping list and a target encoding type group whose similarity with the target encoding method meets the requirements. If a target encoding type group is found, the target super-resolution model corresponding to the target encoding type group is directly used to process the first image. In this way, when a new encoding method that does not exist in the mapping list appears, a suitable super-resolution model can be adapted from the mapping list without retraining the corresponding super-resolution model.

[0193] In particular, when there is only one target encoding type group whose similarity between the corresponding encoding method and the target encoding method meets the requirements, it means that the encoding features of the target encoding method are similar to the encoding features of each encoding method in the target encoding type. Therefore, the present application can also add the target encoding method as an encoding method included in the target encoding type group in the mapping list.

[0194] S507: If multiple target encoding type groups whose similarities with the target encoding type meet the requirements are determined, multiple target super-resolution models corresponding to the multiple target encoding type groups are determined from the mapping list.

[0195] S508, constructing a comprehensive super-resolution model based on the model parameters in each target super-resolution model, and using the comprehensive super-resolution model to process the first image to generate a second image.

[0196] Among them, based on the model parameters of each target super-resolution model, constructing a comprehensive super-resolution model can be to fuse model parameters such as weights of multiple target super-resolution models to generate a new super-resolution model.

[0197] For ease of distinction, this application refers to the super-resolution model constructed based on each target super-resolution model as a comprehensive super-resolution model.

[0198] Since the comprehensive super-resolution model combines the image processing characteristics of each target super-resolution model and is applicable to a wider range of encoding methods, a high-quality second image can also be generated by processing the first image using the comprehensive super-resolution model; and there is no need to retrain a new super-resolution model.

[0199] It can be understood that step 508 is explained by taking a method of implementing a first image based on multiple target super-resolution models corresponding to multiple target coding type groups as an example. In practical applications, after determining the multiple target super-resolution models corresponding to the multiple target coding type groups, the present application can also process the first image using each target super-resolution model to obtain a candidate image generated by each target super-resolution model. Based on the weights corresponding to each target super-resolution model, the candidate images obtained by processing the first image with each target super-resolution model are fused to obtain a second image.

[0200] Among them, the weight corresponding to each target super-resolution can be pre-set; or the weight of the target super-resolution model corresponding to each target coding type group can be determined based on the similarity between the coding method corresponding to each target coding type group and the target coding method, without specific limitation.

[0201] Corresponding to the image processing method provided in this application, this application also provides an image processing device.

[0202] like Figure 6 , shows a schematic diagram of a composition structure of an image processing device provided by the present application. The image processing device of this embodiment may include:

[0203] The type determination unit 601 is used to obtain the decoded first image and determine the target encoding mode corresponding to the first image;

[0204] The model determining unit 602 is used to determine a target super-resolution model corresponding to the target encoding mode from a plurality of super-resolution models, wherein the plurality of super-resolution models correspond to different encoding type groups, the target encoding mode matches the encoding type group corresponding to the target super-resolution model, and the encoding modes matched with the same encoding type group have at least one same encoding feature;

[0205] The image processing unit 603 is used to process the first image using the target super-resolution model to generate a second image.

[0206] In a possible implementation, determining a unit from the model includes:

[0207] The model determination subunit is used to determine the target super-resolution model corresponding to the target encoding method based on a mapping list, wherein the mapping list records at least two encoding methods included in each of the multiple encoding type groups, and the super-resolution model corresponding to each encoding type group.

[0208] In yet another possible implementation, the model determination subunit includes:

[0209] A first determination subunit is used to determine a target super-resolution model corresponding to the target encoding mode from the mapping list if the target encoding mode is queried from the mapping list;

[0210] The second determination subunit is used to determine, if the target encoding method is not queried from the mapping list, a target encoding type group whose similarity with the corresponding encoding method meets the requirements from the mapping list, and determine the target super-resolution model corresponding to the target encoding type group in the mapping list.

[0211] In yet another possible implementation, when determining the target super-resolution model corresponding to the target coding type group in the mapping list, the second determining subunit is specifically configured to determine, if multiple target coding type groups are determined from the mapping list, multiple target super-resolution models corresponding to the multiple target coding type groups in the mapping list;

[0212] The image processing unit comprises:

[0213] A first processing subunit is used to process the first image using each target super-resolution model respectively, and fuse the candidate images obtained by processing the first image with each target super-resolution model based on the weights corresponding to each target super-resolution model to obtain a second image;

[0214] or,

[0215] The second processing subunit is used to construct a comprehensive super-resolution model based on the model parameters in each target super-resolution model, and use the comprehensive super-resolution model to process the first image to generate a second image.

[0216] In one possible implementation, the super-resolution model recorded in the mapping list used in the model determination subunit uses the first sample image encoded using each encoding method in the encoding type group as training data, and the second sample image corresponding to the first sample image as a training target. The super-resolution model trained corresponding to the encoding type group has a second sample image with higher quality than the first sample image.

[0217] In a possible implementation, the second sample image corresponding to the first sample image is: an unencoded original image corresponding to the first sample image, or an optimized image obtained by image optimization processing of the original image, wherein the quality of the original image corresponding to the first sample image is higher than the quality of the first sample image.

[0218] In yet another possible implementation, when the second sample image is an optimized image, the optimized image is obtained by sharpening the detail area of ​​the original image.

[0219] In yet another possible implementation, in a loss function used for training the super-resolution model, different region weights are assigned to different regions in the second sample image, and the detail complexity of different regions in the second sample image is different.

[0220] In another possible implementation, the loss function used for training the super-resolution model includes a noise suppression term. In the noise suppression term, different suppression weights are assigned to different pixel point regions in the second sample image, wherein the greater the difference between the pixel point region in the second sample image and the corresponding pixel point region in the target image, the greater the suppression weight assigned to the pixel point region in the second sample image, the pixel point region includes at least one pixel, and the target image is an image generated by the super-resolution model by processing the first sample image.

[0221] In yet another possible implementation, a loss function used to train a super-resolution model corresponding to each encoding type group matches the encoding features of the encoding method corresponding to the encoding type group.

[0222] On the other hand, corresponding to a model training method of the present application, the present application also provides a model training device. Figure 7 , shows a schematic diagram of a composition structure of a model training device provided by the present application. The device of this embodiment may include:

[0223] The model determination unit 701 is used to determine the super-resolution models to be trained corresponding to at least two coding type groups, and the coding modes matching the same coding type group have at least one common coding feature;

[0224] The sample obtaining unit 702 is used to obtain, for each encoding mode corresponding to each encoding type group, a first sample image encoded by the encoding mode corresponding to the encoding type group, and a second sample image corresponding to the first sample image, wherein the quality of the second sample image is higher than that of the first sample image;

[0225] The model training unit 703 is used to train the super-resolution model corresponding to each encoding type group, using the first sample image corresponding to the encoding type group as training data and the second sample image corresponding to the first sample image as a training target, so as to obtain the trained super-resolution model corresponding to the encoding type group.

[0226] In a possible implementation, the sample obtaining unit includes:

[0227] An original image acquisition subunit is used to obtain an original image that has not been encoded;

[0228] The encoding processing subunit is used to encode the original image using the encoding method corresponding to the encoding type group to obtain a first sample image;

[0229] The image determination subunit is used to determine the original image or the optimized image obtained by optimizing the original image as the second sample image corresponding to the first sample image.

[0230] In another possible implementation, the model training unit is specifically configured to use a first sample image corresponding to the encoding type group as training data, a second sample image corresponding to the first sample image as a training target, and train a super-resolution model corresponding to the encoding type group in combination with a loss function;

[0231] Wherein, in the loss function, different area weights are assigned to different areas in the second sample image, and the detail complexity of different areas in the second sample image is different;

[0232] And / or, the loss function includes a noise suppression term, in which different suppression weights are assigned to different pixel regions in the second sample image, wherein the greater the difference between the pixel region in the second sample image and the corresponding pixel region in the target image, the greater the suppression weight assigned to the pixel region in the second sample image, the target image is an image generated by the super-resolution model by processing the first sample image, and the pixel region includes at least one pixel;

[0233] And / or, the loss function used to train the super-resolution model corresponding to each encoding type group matches the encoding features of the encoding method corresponding to the encoding type group.

[0234] The present application also provides an electronic device. Figure 8, which shows a schematic diagram of a composition structure of the electronic device, and the electronic device at least includes a processor 801 and a memory 802;

[0235] The processor 801 is used to execute the image processing method or model training method described in any one of the above embodiments;

[0236] The memory 802 is used to store programs required by the processor to perform operations.

[0237] It is understandable that the electronic device may further include a display unit 803 and an input unit 804 .

[0238] Of course, the electronic device may also have Figure 8 There may be more or fewer components, without limitation.

[0239] A computer program product is also provided in an embodiment of the present application, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any image processing method or model training method provided in the embodiment of the present application.

[0240] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When one or more computer programs are executed by an electronic device, the electronic device can implement any image processing method or model training method provided in an embodiment of the present application.

[0241] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0242] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0243] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0244] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. An image processing method, comprising: Obtaining a decoded first image, and determining a target encoding mode corresponding to the first image; Determining a target super-resolution model corresponding to the target encoding mode from a plurality of super-resolution models, wherein the plurality of super-resolution models correspond to different encoding type groups, the target encoding mode matches the encoding type group corresponding to the target super-resolution model, and encoding modes matched with the same encoding type group have at least one same encoding feature; The first image is processed using the target super-resolution model to generate a second image.

2. The image processing method according to claim 1, wherein determining a target super-resolution model corresponding to the target encoding mode from a plurality of super-resolution models comprises: Based on a mapping list, a target super-resolution model corresponding to the target encoding mode is determined, wherein the mapping list records at least two encoding modes respectively included in a plurality of encoding type groups, and a super-resolution model corresponding to each encoding type group.

3. The image processing method according to claim 2, wherein determining the target super-resolution model corresponding to the target encoding mode based on the mapping list comprises: If the target encoding mode is found from the mapping list, a target super-resolution model corresponding to the target encoding mode is determined from the mapping list; If the target encoding method is not found from the mapping list, determine from the mapping list a target encoding type group whose corresponding encoding method meets the requirements of similarity with the target encoding method, and determine the target super-resolution model corresponding to the target encoding type group in the mapping list.

4. The image processing method according to claim 3, wherein determining the target super-resolution model corresponding to the target encoding type group in the mapping list comprises: If a plurality of target encoding type groups are determined from the mapping list, a plurality of target super-resolution models corresponding to the plurality of target encoding type groups in the mapping list are determined; The step of processing the first image by using the target super-resolution model to generate a second image includes: Using each target super-resolution model to process the first image respectively, and based on the weights corresponding to each target super-resolution model, fusing the candidate images obtained by processing the first image with each target super-resolution model to obtain a second image; or, A comprehensive super-resolution model is constructed based on the model parameters in each target super-resolution model, and the first image is processed using the comprehensive super-resolution model to generate a second image.

5. According to the image processing method of claim 2, the super-resolution model recorded in the mapping list is a super-resolution model corresponding to the encoding type group trained by using the first sample image encoded by each encoding method in the encoding type group as training data and the second sample image corresponding to the first sample image as a training target, and the quality of the second sample image is higher than that of the first sample image.

6. The image processing method according to claim 5, wherein the second sample image corresponding to the first sample image is: an uncoded original image corresponding to the first sample image, or an optimized image obtained by image optimization processing of the original image, wherein: The quality of the original image corresponding to the first sample image is higher than the quality of the first sample image; The optimized image is obtained by sharpening the detail area of ​​the original image.

7. The image processing method according to claim 5, wherein in the loss function used to train the super-resolution model, different area weights are assigned to different areas in the second sample image, and the detail complexity of different areas in the second sample image is different; And / or, the loss function used in training the super-resolution model includes a noise suppression term, in which different suppression weights are assigned to different pixel regions in the second sample image, wherein: The greater the difference between the pixel point region in the second sample image and the corresponding pixel point region in the target image, the greater the suppression weight assigned to the pixel point region in the second sample image, the pixel point region includes at least one pixel point, and the target image is an image generated by the super-resolution model by processing the first sample image; And / or, the loss function used to train the super-resolution model corresponding to each encoding type group matches the encoding features of the encoding method corresponding to the encoding type group.

8. A model training method, comprising: Determine super-resolution models to be trained corresponding to at least two encoding type groups, respectively, wherein encoding modes matching the same encoding type group have at least one common encoding feature; For each encoding mode corresponding to each encoding type group, a first sample image encoded by the encoding mode corresponding to the encoding type group and a second sample image corresponding to the first sample image are obtained, wherein the quality of the second sample image is higher than the quality of the first sample image; For each encoding type group, a first sample image corresponding to the encoding type group is used as training data, and a second sample image corresponding to the first sample image is used as a training target to train a super-resolution model corresponding to the encoding type group to obtain a trained super-resolution model corresponding to the encoding type group.

9. The model training method according to claim 8, wherein obtaining a first sample image encoded by the encoding method corresponding to the encoding type group and a second sample image corresponding to the first sample image comprises: Obtaining the original image without encoding; Encoding the original image using the encoding method corresponding to the encoding type group to obtain a first sample image; The original image or an optimized image obtained by optimizing the original image is determined as a second sample image corresponding to the first sample image.

10. The model training method according to claim 8, taking the first sample image corresponding to the encoding type group as training data, taking the second sample image corresponding to the first sample image as a training target, training the super-resolution model corresponding to the encoding type group, comprising: Using a first sample image corresponding to the encoding type group as training data, using a second sample image corresponding to the first sample image as a training target, and training a super-resolution model corresponding to the encoding type group in combination with a loss function; Wherein, in the loss function, different area weights are assigned to different areas in the second sample image, and the detail complexity of different areas in the second sample image is different; And / or, the loss function includes a noise suppression term, in which different suppression weights are assigned to different pixel regions in the second sample image, wherein the greater the difference between the pixel region in the second sample image and the corresponding pixel region in the target image, the greater the suppression weight assigned to the pixel region in the second sample image, the target image is an image generated by the super-resolution model by processing the first sample image, and the pixel region includes at least one pixel; And / or, the loss function used to train the super-resolution model corresponding to each encoding type group matches the encoding features of the encoding method corresponding to the encoding type group.

11. An image processing device, comprising: a type determination unit, configured to obtain the decoded first image and determine a target encoding mode corresponding to the first image; a model determination unit, configured to determine a target super-resolution model corresponding to the target encoding mode from a plurality of super-resolution models, wherein the plurality of super-resolution models correspond to different encoding type groups, the target encoding mode matches the encoding type group corresponding to the target super-resolution model, and encoding modes matched with the same encoding type group have at least one same encoding feature; An image processing unit is used to process the first image using the target super-resolution model to generate a second image.

12. A model training device, comprising: A model determination unit, used to determine super-resolution models to be trained corresponding to at least two encoding type groups, wherein encoding modes matching the same encoding type group have at least one common encoding feature; a sample obtaining unit, configured to obtain, for each encoding mode corresponding to each encoding type group, a first sample image encoded by the encoding mode corresponding to the encoding type group, and a second sample image corresponding to the first sample image, wherein the quality of the second sample image is higher than that of the first sample image; A model training unit is used to train a super-resolution model corresponding to the encoding type group for each encoding type group, using a first sample image corresponding to the encoding type group as training data and a second sample image corresponding to the first sample image as a training target, so as to obtain a trained super-resolution model corresponding to the encoding type group.

Images