Image processing method, device and electronic equipment

Abstract

The embodiments of the present application provide an image processing method, apparatus, and electronic device. The method includes: first, obtaining a reconstructed image; then, performing image enhancement on the reconstructed image to obtain an intermediate image; then, determining the texture complexity weight corresponding to each region in the intermediate image, where the texture complexity weight is a number between 0 and 1; and then, based on the texture complexity weight corresponding to each region in the intermediate image, attenuating the texture intensity corresponding to each region in the intermediate image to obtain an enhanced image corresponding to the reconstructed image. In this way, the texture of the textured regions in the intermediate image can be preserved while the texture of the non-textured regions can be attenuated. This enhances the texture of the textured regions in the reconstructed image while avoiding the generation of false texture in the non-textured regions, thereby reducing the visual distortion of the reconstructed image, increasing the realism of the image, and improving the subjective quality of the image.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to an image processing method, device, and electronic device. Background Art

[0002] Typically, video is compressed before transmission to improve transmission efficiency. The higher the compression ratio, the smaller the compressed video data volume. However, as the compression ratio increases, visible visual distortion occurs in the video. To reduce this distortion in the reconstructed image, post-processing can be performed to improve quality while maintaining the same bitrate.

[0003] Although post-processing of the reconstructed image can significantly improve the subjective quality of the reconstructed image, it is easy to generate false textures in non-texture areas, affecting the subjective quality of the image. Summary of the Invention

[0004] In order to solve the above technical problems, the present application provides an image processing method, device and electronic device.

[0005] In a first aspect, an embodiment of the present application provides an image processing method, comprising: first, obtaining a reconstructed image; then, performing image enhancement on the reconstructed image to obtain an intermediate image. Next, determining the texture complexity weight corresponding to each region in the intermediate image, where the texture complexity weight is a number between 0 and 1; then, based on the texture complexity weight corresponding to each region in the intermediate image, attenuating the texture intensity corresponding to each region in the intermediate image to obtain an enhanced image corresponding to the reconstructed image. In this way, the texture of the texture region in the intermediate image can be retained while the texture of the non-texture region is attenuated, thereby enhancing the texture of the texture region in the reconstructed image while avoiding the generation of false texture in the non-texture region, thereby reducing the visual distortion of the reconstructed image, increasing the realism of the image, and improving the subjective quality of the image.

[0006] In addition, the present application controls the texture complexity weight in the range of 0 to 1, so that the textures corresponding to adjacent areas are softer, which can avoid the problem of inconsistent texture effects enhanced in adjacent areas.

[0007] For example, the texture complexity weight is proportional to the texture complexity, that is, the greater the texture complexity, the greater the texture complexity weight. In this way, the texture effect of the texture area in the intermediate image can be maintained as much as possible, while the texture effect of the non-texture area is attenuated according to the texture complexity.

[0008] Exemplarily, the intermediate image is a texture enhanced image or a residual image.

[0009] Exemplarily, the resolutions of the intermediate image and the enhanced image are the same as the resolution of the reconstructed image.

[0010] For example, GANEF (Generative Adversarial Network Enhancement Filter) may be used to perform image enhancement on the reconstructed image to obtain an intermediate image.

[0011] Exemplarily, the image output by GANEF may be a texture-enhanced image or a residual image.

[0012] It should be noted that although the enhanced image is the attenuation of texture enhancement in each area of ​​the intermediate image, only the texture intensity of the intermediate image is partially attenuated. Compared with the reconstructed image, the texture intensity of some areas of the enhanced image is still greater than that of the reconstructed image; that is, the enhanced image still has a texture enhancement effect.

[0013] According to the first aspect, the intermediate image is a residual image; based on the texture complexity weights corresponding to the respective regions in the intermediate image, the texture intensities corresponding to the respective regions in the intermediate image are attenuated respectively to obtain an enhanced image corresponding to the reconstructed image, including: multiplying the pixel values ​​of each pixel point in the residual image by the texture complexity weights corresponding to the regions to which each pixel point belongs to obtain a residual update image; and generating an enhanced image based on the residual update image and the reconstructed image.

[0014] Illustratively, the residual image is obtained by subtracting the texture enhanced image from the reconstructed image.

[0015] According to the first aspect, or any implementation of the first aspect above, generating an enhanced image based on the residual updated image and the reconstructed image includes: adding the residual updated image and the reconstructed image to obtain the enhanced image.

[0016] Exemplarily, the residual updated image and the reconstructed image may be added pixel by pixel to obtain an enhanced image.

[0017] According to the first aspect, or any implementation of the first aspect above, the intermediate image is a texture-enhanced image, and the method further includes: performing image fidelity on the reconstructed image to obtain a basic fidelity image.

[0018] Exemplarily, the base fidelity image and the reconstructed image have the same resolution.

[0019] For example, non-GANEF can be used to perform image fidelity on the reconstructed image to obtain a basic fidelity image.

[0020] According to the first aspect, or any implementation method of the first aspect above, the texture complexity weights corresponding to each area in the intermediate image are determined, including: dividing the basic fidelity image and the texture enhanced image into N areas according to a preset partitioning rule, where N is a positive integer; determining the texture complexity of the N areas in the basic fidelity image respectively; and determining the texture complexity weights corresponding to the N areas in the texture enhanced image respectively based on the texture complexity of the N areas in the basic fidelity image.

[0021] According to the first aspect, or any implementation method of the first aspect above, the texture intensity corresponding to each area in the intermediate image is attenuated according to the texture complexity weights corresponding to each area in the intermediate image to obtain an enhanced image corresponding to the reconstructed image, including: determining the weighted calculation weights corresponding to the N areas in the texture-enhanced image and the weighted calculation weights corresponding to the N areas in the basic fidelity image according to the texture complexity weights corresponding to the N areas in the texture-enhanced image; performing weighted calculation on the N areas in the texture-enhanced image and the N areas in the basic fidelity image according to the weighted calculation weights corresponding to the N areas in the texture-enhanced image and the weighted calculation weights corresponding to the N areas in the basic fidelity image to obtain an enhanced image corresponding to the reconstructed image.

[0022] According to the first aspect, or any implementation method of the first aspect above, based on the weighted calculation weights corresponding to the N regions in the texture-enhanced image and the weighted calculation weights corresponding to the N regions in the basic fidelity image, weighted calculation is performed on the N regions in the texture-enhanced image and the N regions in the basic fidelity image to obtain an enhanced image corresponding to the reconstructed image, including: multiplying the weighted calculation weights corresponding to the N regions in the texture-enhanced image by the N regions in the texture-enhanced image to obtain a first product; multiplying the weighted calculation weights corresponding to the N regions in the basic fidelity image by the N regions in the basic fidelity image to obtain a second product; and adding the first product and the second product to obtain an enhanced image corresponding to the reconstructed image.

[0023] According to the first aspect, or any implementation method of the first aspect above, based on the texture complexity weights corresponding to the N regions in the texture-enhanced image, the weighted calculation weights corresponding to the N regions in the texture-enhanced image and the weighted calculation weights corresponding to the N regions in the basic fidelity image are determined, including: determining the texture complexity weights corresponding to the N regions in the texture-enhanced image as the weighted calculation weights corresponding to the N regions in the texture-enhanced image; and determining the difference between 1 and the texture complexity weights corresponding to the N regions in the texture-enhanced image as the weighted calculation weights corresponding to the N regions in the basic fidelity image.

[0024] According to the first aspect, or any implementation of the first aspect above, determining the texture complexity weight corresponding to each region in the intermediate image includes decoding the texture complexity weights corresponding to N regions in the intermediate image from a received bitstream, where N is a positive integer. This improves the accuracy of determining the texture intensity weight corresponding to each region in the intermediate image, thereby more accurately controlling the attenuation of image texture intensity, thereby further improving image quality.

[0025] According to the first aspect, or any implementation method of the first aspect above, the texture complexity weights corresponding to each area in the intermediate image are determined, including: dividing the reconstructed image and the intermediate image into N areas according to a preset partitioning rule, where N is a positive integer; determining the texture complexity of the N areas in the reconstructed image respectively; and determining the texture complexity weights corresponding to the N areas in the intermediate image based on the texture complexity of the N areas in the reconstructed image.

[0026] In a second aspect, an embodiment of the present application provides an image processing device, the device comprising:

[0027] An image acquisition module, used for acquiring a reconstructed image;

[0028] An image enhancement module is used to enhance the reconstructed image to obtain an intermediate image;

[0029] A texture weight determination module is used to determine the texture complexity weight corresponding to each area in the intermediate image, where the texture complexity weight is a number between 0 and 1;

[0030] The texture attenuation module is used to attenuate the texture intensity corresponding to each area in the intermediate image according to the texture complexity weight corresponding to each area in the intermediate image, so as to obtain an enhanced image corresponding to the reconstructed image.

[0031] According to the second aspect, the intermediate image is a residual image; the texture attenuation module comprises:

[0032] The residual update module is used to multiply the pixel value of each pixel in the residual image by the texture complexity weight corresponding to the region to which each pixel belongs, so as to obtain a residual update image;

[0033] The image generation module is used to update the image and reconstruct the image according to the residual to generate an enhanced image.

[0034] According to the second aspect, or any implementation of the second aspect above, the image generation module is specifically configured to add the residual updated image and the reconstructed image to obtain an enhanced image.

[0035] According to the second aspect, or any implementation of the second aspect above, the intermediate image is a texture-enhanced image, and the apparatus further includes: an image fidelity module configured to perform image fidelity on the reconstructed image to obtain a basic fidelity image.

[0036] According to the second aspect, or any implementation method of the second aspect above, the texture weight determination module is specifically used to divide the basic fidelity image and the texture enhanced image into N regions, where N is a positive integer, according to a preset partitioning rule; determine the texture complexity of the N regions in the basic fidelity image respectively; and determine the texture complexity weights corresponding to the N regions in the texture enhanced image based on the texture complexity of the N regions in the basic fidelity image.

[0037] According to the second aspect, or any implementation of the second aspect, the texture attenuation module includes:

[0038] A weighted weight determination module is used to determine weighted calculation weights corresponding to the N regions in the texture-enhanced image and weighted calculation weights corresponding to the N regions in the basic fidelity image according to the texture complexity weights corresponding to the N regions in the texture-enhanced image;

[0039] The weighted calculation module is used to perform weighted calculation on the N areas in the texture-enhanced image and the N areas in the basic fidelity image based on the weighted calculation weights corresponding to the N areas in the texture-enhanced image and the weighted calculation weights corresponding to the N areas in the basic fidelity image, so as to obtain an enhanced image corresponding to the reconstructed image.

[0040] According to the second aspect, or any implementation method of the above second aspect, the weighted calculation module is specifically used to multiply the weighted calculation weights corresponding to the N regions in the texture-enhanced image by the N regions in the texture-enhanced image to obtain a first product; multiply the weighted calculation weights corresponding to the N regions in the basic fidelity image by the N regions in the basic fidelity image to obtain a second product; add the first product and the second product to obtain an enhanced image corresponding to the reconstructed image.

[0041] According to the second aspect, or any implementation method of the above second aspect, the weighted weight determination module is specifically used to determine the texture complexity weights corresponding to the N regions in the texture-enhanced image as the weighted calculation weights corresponding to the N regions in the texture-enhanced image; and determine the difference between 1 and the texture complexity weights corresponding to the N regions in the texture-enhanced image as the weighted calculation weights corresponding to the N regions in the basic fidelity image.

[0042] According to the second aspect, or any implementation of the second aspect above, the texture weight determination module is specifically used to decode the texture complexity weights corresponding to N regions in the intermediate image from the received code stream, where N is a positive integer.

[0043] According to the second aspect, or any implementation of the second aspect above, the texture weight determination module 803 is specifically used to divide the reconstructed image and the intermediate image into N regions, where N is a positive integer, according to a preset partitioning rule; determine the texture complexity of the N regions in the reconstructed image respectively; and determine the texture complexity weights corresponding to the N regions in the intermediate image based on the texture complexity of the N regions in the reconstructed image.

[0044] The second aspect and any implementation of the second aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the second aspect and any implementation of the second aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0045] In a third aspect, an embodiment of the present application provides an electronic device comprising: a memory and a processor, the memory being coupled to the processor; the memory storing program instructions, which, when executed by the processor, enables the electronic device to execute the image processing method in the first aspect or any possible implementation of the first aspect.

[0046] The third aspect and any implementation of the third aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the third aspect and any implementation of the third aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0047] In a fourth aspect, an embodiment of the present application provides a chip comprising one or more interface circuits and one or more processors; the interface circuit is used to receive signals from a memory of an electronic device and send signals to the processor, the signals including computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device executes the image processing method of the first aspect or any possible implementation of the first aspect.

[0048] The fourth aspect and any implementation of the fourth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fourth aspect and any implementation of the fourth aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0049] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a computer or a processor, the computer or the processor executes the image processing method in the first aspect or any possible implementation of the first aspect.

[0050] The fifth aspect and any implementation of the fifth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fifth aspect and any implementation of the fifth aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0051] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a computer or a processor, the computer or the processor executes the image processing method in the first aspect or any possible implementation of the first aspect.

[0052] The sixth aspect and any implementation of the sixth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the sixth aspect and any implementation of the sixth aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1a is a schematic diagram of an exemplary application scenario;

[0054] Figure 1b A schematic diagram of an exemplary processing process is shown;

[0055] Figure 2 A schematic diagram of an exemplary processing process is shown;

[0056] Figure 3a Schematic diagram of an image enhancement process shown as an example;

[0057] Figure 3b A schematic diagram of an exemplary processing process is shown;

[0058] Figure 3c Schematic diagram showing an exemplary image enhancement effect;

[0059] Figure 4 A schematic diagram of an exemplary processing process is shown;

[0060] Figure 5 A schematic diagram of an exemplary processing process is shown;

[0061] Figure 6 Schematic diagram of an image enhancement process shown as an example;

[0062] Figure 7 A schematic diagram of an exemplary processing process is shown;

[0063] Figure 8 is a schematic diagram of an image processing device shown as an example;

[0064] Figure 9 Schematic diagram of the structure of the device shown as an example. DETAILED DESCRIPTION

[0065] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0066] The term "and / or" in this article is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0067] In the description and claims of the embodiments of this application, the terms "first" and "second" are used to distinguish different objects, rather than to describe a specific order of objects. For example, the terms "first target object" and "second target object" are used to distinguish different objects, rather than to describe a specific order of objects.

[0068] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0069] In the description of the embodiments of this application, unless otherwise specified, "multiple" means two or more. For example, "multiple processing units" means two or more processing units; "multiple systems" means two or more systems.

[0070] Figure 1a The figure is a schematic diagram of an exemplary application scenario.

[0071] Reference Figure 1a For example, the encoding end may encode the image to be encoded to obtain a code stream, and then transmit the code stream to the decoding end. After obtaining the code stream, the decoding end may decode the code stream to obtain a reconstructed image.

[0072] Exemplarily, after obtaining the reconstructed image, the decoding end may perform post-processing on the reconstructed image to enhance the reconstructed image to obtain an enhanced image, and then output the enhanced image to reduce visual distortion of the reconstructed image.

[0073] Exemplarily, for the post-processing stage, the present application proposes an image processing method that can enhance the texture of the texture area in the reconstructed image and avoid the generation of false texture in the non-texture area, so as to reduce the visual distortion of the reconstructed image, thereby increasing the realism of the image and improving the subjective quality of the image.

[0074] Figure 1b A schematic diagram illustrating an exemplary processing process.

[0075] S101, obtaining a reconstructed image.

[0076] Exemplarily, after obtaining the code stream, the decoding end may decode the code stream to obtain a reconstructed image.

[0077] S102: Perform image enhancement on the reconstructed image to obtain an intermediate image.

[0078] For example, in order to reduce visual distortion of the reconstructed image, image enhancement may be performed on the reconstructed image to obtain an intermediate image.

[0079] Exemplarily, an image enhancement model can be used to enhance the reconstructed image. Exemplarily, the image enhancement model can be GANEF (Generative Adversarial Network Enhancement Filter), that is, an enhancement filter based on a generative adversarial network. Among them, GAN (Generative Adversarial Network) can include a generator and a discriminator; thus, GANEF also includes a generator and a discriminator. When training GANEF, the generator and the discriminator can be trained alternately and iteratively. Among them, the goal of the discriminator is to distinguish the authenticity of the image generated by the generator and the original image, and the goal of the generator is to generate an image close to the original image to deceive the discriminator. Through effective adversarial training, the generator can generate an image as close to the original image as possible.

[0080] Exemplarily, the GANEF training process can be as follows: multiple sets of training data are generated, where one set of training data includes an image to be encoded and a reconstructed image obtained by decoding the bitstream of the image to be encoded. Exemplarily, the training data can be used to alternately train the generator and the discriminator. When training the generator, the weight parameters of the discriminator can be fixed; when training the discriminator, the weight parameters of the generator can be fixed.

[0081] Exemplarily, the process of training the generator can be as follows: First, the image to be encoded from the training data is input to the discriminator, which performs forward computation on the image to be encoded and outputs a discrimination result. The discriminator's weight parameters are then adjusted to ensure that the probability of the discriminator's discrimination result being "true" approaches a preset probability. The preset probability can be set as needed and is not limited in this application. When the difference between the probability of the discriminator's output of a "true" discrimination result based on the input image to be encoded and the preset probability is less than a probability threshold, the discriminator's weight parameters can be fixed. The probability threshold can be set as needed and is not limited in this application. At this point, the reconstructed image from the training data is input to the generator, which performs forward computation on the reconstructed image and outputs an intermediate image to the discriminator. The discriminator then performs forward computation on the intermediate image and outputs a discrimination result. Furthermore, a loss function value is generated based on the intermediate image and the image to be encoded. Subsequently, the generator's weight parameters are adjusted to maximize the probability of the discriminator's output of a "true" discrimination result and minimize the loss function value.

[0082] Exemplarily, the process of training a discriminator can be as follows: First, the image to be encoded from the training data is input into the discriminator, which performs forward computation on the image to be encoded and outputs a discrimination result. The discriminator's weight parameters are then adjusted to ensure that the probability of the discriminator's discrimination result being "true" approaches a preset probability. Once the difference between the probability of the discriminator's output of a "true" discrimination result based on the input image to be encoded and the preset probability is less than a probability threshold, the reconstructed image from the training data is input into the generator, which performs forward computation on the reconstructed image and outputs an intermediate image to the discriminator. The discriminator then performs forward computation on the intermediate image and outputs a discrimination result. The discriminator's weight parameters are then adjusted to maximize the probability of the discriminator's output of a "false" discrimination result.

[0083] In this way, the generator and discriminator in GANEF are trained alternately and iteratively in the above manner until the discriminator cannot make a good distinction between the intermediate image output by the generator and the image to be encoded.

[0084] Exemplarily, the intermediate image can be a residual image or a texture-enhanced image. During training, when the intermediate image output by the generator is a residual image, a texture-enhanced image can be generated based on the residual image and the reconstructed image. This texture-enhanced image is then input into the discriminator, and the loss function value is calculated based on the texture-enhanced image and the image to be encoded. When the intermediate image output by the generator is a texture-enhanced image, the texture-enhanced image can be directly input into the discriminator.

[0085] Exemplarily, the residual image and the reconstructed image may be added to generate a texture-enhanced image. Exemplarily, the residual image and the reconstructed image may be added pixel by pixel, that is, the pixel values ​​of the pixels at corresponding positions in the residual image and the reconstructed image are added to obtain an enhanced image corresponding to the reconstructed image.

[0086] For example, after GANEF training is completed, only the trained generator in GANEF can be used to perform image enhancement on the reconstructed image to obtain an intermediate image.

[0087] The resolution of the intermediate image is the same as that of the reconstructed image.

[0088] S103: Determine the texture complexity weight corresponding to each region in the intermediate image.

[0089] In one possible approach, after the intermediate image is determined, the texture complexity weight corresponding to each region in the intermediate image may be determined according to the texture complexity of each region in the reconstructed image.

[0090] In one possible approach, after the intermediate image is determined, the texture complexity weights corresponding to the respective regions in the intermediate image may be determined based on the texture complexity of the respective regions in the image to be encoded corresponding to the reconstructed image.

[0091] Exemplarily, the texture complexity weight is proportional to the texture complexity, that is, the higher the texture complexity, the greater the texture complexity weight; the lower the texture complexity, the smaller the texture complexity weight.

[0092] Exemplarily, the texture complexity weight is a decimal between 0 and 1.

[0093] S104 , attenuating the texture intensity corresponding to each region in the intermediate image according to the texture complexity weight corresponding to each region in the intermediate image, to obtain an enhanced image corresponding to the reconstructed image.

[0094] Exemplarily, for each region in the intermediate image, the pixel value of each pixel in the region can be attenuated based on the texture complexity weight corresponding to the region to adjust the texture intensity corresponding to the region. Since the texture complexity weight is proportional to the texture complexity, after the texture intensity is adjusted, the higher the texture complexity of the region in the intermediate image, the smaller the corresponding texture intensity attenuation, and the lower the texture complexity of the region, the greater the corresponding texture enhancement attenuation. In this way, it is possible to enhance the texture of the texture area (higher texture complexity) while avoiding the generation of false stripes in the non-texture area (lower texture complexity). In other words, the present application controls the texture intensity attenuation of the GANEF result based on the texture complexity of the local area of ​​the image, that is, the texture area maintains the original GANEF filtering effect as much as possible, and the non-texture area attenuates the GANEF filtering effect according to the texture complexity. The smaller the texture complexity of the local area, the greater the attenuation, thereby effectively removing the false texture of the non-texture area. In addition, the present application controls the texture complexity weight in the range of 0 to 1, so that the texture corresponding to the adjacent area is softer, which can avoid the problem of inconsistent texture effects enhanced in adjacent areas.

[0095] For example, Figure 1b For the specific implementation method, please refer to Figure 2 、 Figure 4 、 Figure 5 and Figure 7 , and the corresponding description.

[0096] Figure 2 FIG. 1 is a schematic diagram showing an exemplary processing process. Figure 2 In an embodiment, the reconstructed image is first filtered using GANEF to obtain a residual image; then, based on the texture complexity of each region in the reconstructed image, the texture complexity weight corresponding to each region in the residual image is determined; then, the residual image is updated according to the texture complexity weight corresponding to each region, and finally, the updated residual image and the reconstructed image are added to obtain a final enhanced image.

[0097] S201: Perform image enhancement on the reconstructed image to obtain a residual image.

[0098] Exemplarily, after the decoding end obtains the code stream, it can decode the code stream to obtain a reconstructed image; then, it can use an image enhancement model to perform image enhancement on the reconstructed image to obtain an intermediate image.

[0099] Exemplarily, the intermediate image is a residual image.

[0100] Figure 3a FIG. 1 is a schematic diagram illustrating an exemplary image enhancement process.

[0101] In one possible approach, the output of the trained image enhancement model is a residual image. Thus, by inputting the reconstructed image into the image enhancement model, the residual image can be directly obtained, such as Figure 3a (1) shown.

[0102] In one possible approach, the output of the trained image enhancement model is a texture enhanced image. Thus, after the reconstructed image is input into the image enhancement model, the image enhancement model outputs a texture enhanced image; then, based on the texture enhanced image and the reconstructed image, a residual image is determined. Optionally, the texture enhanced image and the reconstructed image can be subtracted pixel by pixel, that is, the pixel values ​​of the corresponding pixels of the texture enhanced image and the reconstructed image are subtracted to obtain a residual image, such as Figure 3a (2) shown.

[0103] Exemplarily, the residual image has the same resolution as the reconstructed image.

[0104] S202 : Determine, based on the reconstructed image, the texture complexity weight corresponding to each region in the residual image.

[0105] Exemplarily, the texture complexity weight corresponding to each region in the residual image may be determined according to the texture complexity corresponding to each region in the reconstructed image. Exemplarily, S202 may include S2021 to S2023:

[0106] S2021: Divide the reconstructed image and the residual image into N regions respectively according to a preset partitioning rule.

[0107] For example, the reconstructed image can be divided into N regions according to a preset partitioning rule; and the residual image can be divided into N regions according to the preset partitioning rule. In other words, the reconstructed image and the residual image are divided into regions in the same manner. Since the reconstructed image and the residual image have the same resolution, the N regions in the reconstructed image correspond one-to-one to the N regions in the residual image.

[0108] Exemplarily, the preset partitioning rule can be set as required, and this application does not impose any restrictions on this; wherein N is a positive integer determined according to the preset partitioning rule. For example, the preset partitioning rule is: divide the image into N regions of size w*h. Where w = W / N1, h = H / N2, N = N1*N2, where W and H are the width and height of the image, and N1 and N2 are positive integers.

[0109] It should be noted that the size of each of the N regions can be the same or different, and this application does not impose any restrictions on this. In addition, the shape of each of the N regions can be the same or different, and this application does not impose any restrictions on this. This application also does not impose any restrictions on the shape of the N regions.

[0110] S2022: Determine the texture complexity of N regions in the reconstructed image respectively.

[0111] The following is an example of determining the texture complexity of the i-th (i is an integer between 1 and N, and the value of i can be 1 or N) region in the reconstructed image.

[0112] Exemplarily, for the i-th region of the reconstructed image, the texture complexity of the i-th region may be determined according to the pixel values ​​of the pixels contained in the i-th region.

[0113] In one possible approach, the texture complexity of the i-th region of the reconstructed image can be determined based on the co-occurrence matrix of the i-th region in the reconstructed image. Exemplarily, the co-occurrence matrix of the i-th region in the reconstructed image can be determined based on the pixel values ​​of the pixels contained in the i-th region in the reconstructed image. Then, the characteristic quantities (such as energy, contrast, entropy, inverse variance, etc.) corresponding to the co-occurrence matrix of the i-th region in the reconstructed image are extracted, and then the image texture complexity of the i-th region in the reconstructed image is determined based on the characteristic quantities of the co-occurrence matrix of the i-th region in the reconstructed image. For example, the characteristic quantities of the co-occurrence matrix of the i-th region in the reconstructed image are used as the texture complexity of the i-th region in the reconstructed image.

[0114] In one possible approach, the texture complexity of the i-th region in the reconstructed image can be determined based on the edge ratio of the i-th region in the reconstructed image. For example, the gradient intensity corresponding to each pixel point in the i-th region in the reconstructed image can be calculated based on the pixel values ​​of the pixel points contained in the i-th region of the reconstructed image. Then, the proportion of pixel points in the i-th region in the reconstructed image whose gradient intensity is greater than the gradient intensity threshold is used as the texture complexity of the i-th region of the reconstructed image. The gradient intensity threshold can be set as required, and this application does not impose any restrictions on this.

[0115] It should be understood that other methods may also be used, such as determining the texture complexity of the i-th region in the reconstructed image based on the grayscale histogram distribution of the i-th region in the reconstructed image, and this application does not impose any limitation on this.

[0116] In this way, according to the above method, the texture complexity of N regions in the reconstructed image can be determined.

[0117] S2023 : Determine texture complexity weights corresponding to the N regions in the residual image based on the texture complexities of the N regions in the reconstructed image.

[0118] The texture complexity weight corresponding to the i-th (i is an integer between 1 and N, i can be equal to 1 and N) region in the residual image is determined below for exemplary purposes.

[0119] In one possible approach, the texture complexity of the i-th region in the reconstructed image may be used as the texture complexity weight corresponding to the i-th region in the residual image.

[0120] In one possible approach, the texture complexity of the i-th region in the reconstructed image can be mapped according to a preset mapping rule to obtain a texture complexity weight corresponding to the i-th region in the residual image. Exemplarily, the preset mapping rule can be set as required, such as normalization, which is not limited in this application. For example, the texture complexity of the i-th region in the reconstructed image can be normalized to obtain a texture complexity weight corresponding to the i-th region in the residual image.

[0121] For example, the texture complexity weight may be a decimal between 0 and 1.

[0122] Exemplarily, the texture complexity weight is proportional to the texture complexity, that is, the higher the texture complexity, the greater the texture complexity weight; the lower the texture complexity, the smaller the texture complexity weight.

[0123] Exemplarily, after obtaining the texture complexity weights corresponding to N regions in the residual image, the texture intensity corresponding to each region in the residual image can be attenuated according to the texture complexity weights corresponding to each region in the residual image to perform image enhancement on the reconstructed image to obtain an enhanced image corresponding to the reconstructed image. S203 to S204 can be referred to:

[0124] S203 , performing residual update on each region in the residual image based on the texture complexity weight corresponding to each region in the residual image to obtain a residual update image.

[0125] Exemplarily, based on the texture complexity weights corresponding to the N regions of the residual image, residual updating may be performed on the N regions in the residual image respectively to obtain a residual updated image.

[0126] In one possible approach, the residual image may be residually updated by multiplying the pixel value of each pixel in the residual image by the texture complexity weight corresponding to the region to which each pixel belongs, thereby obtaining a residual updated image.

[0127] The following is an illustrative description of performing residual update on the i-th (i is an integer between 1 and N, and i can be equal to 1 and N) region in the residual image.

[0128] For example, assuming that the texture enhancement weight corresponding to the i-th region in the residual image is ratio_i, for a pixel point (k, j) in the i-th region, the pixel value corresponding to the pixel point (k, j) can be multiplied by R1(k, j) and ratio_i to update the pixel value of the pixel point (k, j) to obtain a new pixel value R2(k, j) = R1(k, j) * ratio_i. Among them, (k, j) is the integer index of the pixel coordinate of the residual image, k and j represent the horizontal and vertical coordinate indexes respectively, and the pixel index of the upper left corner of the residual image is (0, 0). In other words, after the pixel values ​​of each pixel in the residual image are updated, the residual update image can be obtained, and the pixel value of the pixel in the residual update image is R2(k, j).

[0129] Exemplarily, the resolution of the residual updated image and the reconstructed image are the same.

[0130] Since the texture complexity weight is proportional to the texture complexity, the higher the texture complexity of the residual image, the smaller the corresponding texture intensity attenuation, and the lower the texture complexity, the greater the corresponding texture enhancement attenuation.

[0131] S204: Add the reconstructed image and the residual updated image to obtain an enhanced image.

[0132] Exemplarily, the pixel values ​​of the pixels at corresponding positions in the reconstructed image and the residual updated image may be added to obtain an enhanced image corresponding to the reconstructed image.

[0133] Figure 3b A schematic diagram of the processing process is shown as an example.

[0134] Reference Figure 3b For example, A1 is the reconstructed image. After filtering the reconstructed image using GANEF, a texture-enhanced image is obtained, as shown in A2. Then, the texture-enhanced image A2 can be subtracted from the reconstructed image A1 to obtain a residual image, as shown in A4.

[0135] For example, the gradient of each pixel in the reconstructed image A1 can be calculated, and then the reconstructed image can be binarized based on the gradient of each pixel in the reconstructed image A1 to obtain a binary image, as shown in A3. The binary image A3 is then divided into N regions, and for each region, the texture complexity corresponding to the region is determined based on the proportion of black pixels in the region; thus, the texture complexity corresponding to each of the N regions in the binary image A3 can be obtained. Then, based on the texture complexity corresponding to each of the N regions in the binary image, the texture complexity weights corresponding to each of the N regions in the residual image A4 are determined. Based on the texture complexity weights corresponding to each of the N regions in the residual image A4, the residual image A4 is residually updated to obtain a residual updated image, as shown in A5.

[0136] Exemplarily, the reconstructed image A1 and the residual updated image A5 may be added to obtain the enhanced image A6.

[0137] Figure 3c Schematic diagram showing an exemplary image enhancement effect.

[0138] Reference Figure 3c , exemplary, Figure 3c (1) Yes Figure 3b Schematic diagram of a local area in the texture enhanced image A2, Figure 3c (2) Yes Figure 3b Schematic diagram of the local area in the enhanced image A6.

[0139] For example, Figure 3c (1) and Figure 3c The road surface in (2) is a non-textured area. Figure 3c (1) and Figure 3c (2) Middle ellipse area, and comparison Figure 3c (1) and Figure 3c As can be seen from the rectangular area in (2), there are no false stripes in the non-texture area of ​​the enhanced image obtained in this application.

[0140] In this way, by controlling the texture complexity weight within the range of 0 to 1 based on the texture complexity of the reconstructed image, the texture intensity corresponding to each area in the reconstructed image is attenuated. This can preserve the texture of the textured areas in the intermediate image while attenuating the texture in the non-textured areas. This can enhance the texture of the textured areas (higher texture complexity) in the reconstructed image while avoiding the generation of false stripes in the non-textured areas (lower texture complexity) in the reconstructed image, thereby reducing the visual distortion of the reconstructed image. Furthermore, by controlling the texture complexity weight within the range of 0 to 1, the textures corresponding to adjacent areas are softer, thus avoiding the problem of inconsistent texture enhancement effects in adjacent areas.

[0141] In addition, compared with the prior art which uses two models for post-processing, the present application uses only one model, which reduces the computational complexity.

[0142] Figure 4 FIG. 1 is a schematic diagram showing an exemplary processing process. Figure 4 In an embodiment, the reconstructed image is first filtered using GANEF to obtain a residual image; then, based on the texture complexity of each region in the image to be encoded corresponding to the reconstructed image, the texture complexity weights corresponding to each region of the residual image are determined; then, the residual image is updated according to the texture complexity weights corresponding to each region, and finally, the updated residual image and the reconstructed image are added to obtain a final enhanced image.

[0143] S401 , decoding the code stream to obtain texture complexity weights corresponding to various regions in the reconstructed image and the image to be encoded.

[0144] Exemplarily, the encoder may generate, based on the image to be encoded, a texture complexity weight corresponding to each region in the image to be encoded, which may include S4011 to S4013:

[0145] S4011: Divide the image to be encoded into N areas according to a preset partitioning rule.

[0146] S4012: Determine the texture complexity of N regions in the image to be encoded respectively.

[0147] For example, S4011 to S4012 can refer to the description of S2021 to S2022 above, which will not be repeated here.

[0148] S4013 : Determine texture complexity weights corresponding to the N regions in the image to be encoded based on the texture complexities of the N regions in the image to be encoded.

[0149] The following is an example of determining the texture complexity weight corresponding to the i-th region (i is an integer between 1 and N, and the value of i can be 1 or N) in the image to be encoded.

[0150] In one possible approach, the texture complexity of the i-th region in the image to be coded may be used as the texture complexity weight corresponding to the i-th region in the image to be coded.

[0151] In one possible approach, the texture complexity of the i-th region in the image to be coded may be mapped according to a preset mapping rule to obtain a texture complexity weight corresponding to the i-th region in the image to be coded. For example, the texture complexity of the i-th region in the image to be coded may be normalized to obtain a texture complexity weight corresponding to the i-th region in the image to be coded.

[0152] For example, the texture complexity weight may be a decimal between 0 and 1.

[0153] Exemplarily, the texture complexity weight is proportional to the texture complexity, that is, the higher the texture complexity, the greater the texture complexity weight; the lower the texture complexity, the smaller the texture complexity weight.

[0154] For example, the encoder can encode the image to be encoded to obtain a corresponding bitstream; and can also encode the texture complexity weights corresponding to N regions in the image to be encoded to obtain a corresponding bitstream. The bitstream obtained by encoding the texture complexity weights corresponding to the N regions in the image to be encoded and the bitstream obtained by encoding the image to be encoded can then be sent to the decoder. After receiving the bitstream, the decoder decodes the bitstream to obtain the reconstructed image and the texture complexity weights corresponding to the N regions in the image to be encoded.

[0155] S402: Perform image enhancement on the reconstructed image to obtain a residual image.

[0156] Exemplarily, the decoding end may use an image enhancement model to perform image enhancement on the reconstructed image to obtain an intermediate image. Exemplarily, the intermediate image is a residual image.

[0157] Among them, S402 can refer to the description of S201 above, and will not be repeated here.

[0158] Exemplarily, the resolution of the residual image is the same as the resolution of the reconstructed image and the resolution of the image to be encoded.

[0159] S403 , generating texture complexity weights corresponding to respective regions in the residual image according to the texture complexity weights corresponding to respective regions in the image to be encoded.

[0160] Exemplarily, after obtaining the residual image, the residual image can be divided into N regions according to a pre-set partitioning rule. The region division method for the residual image is the same as the region division method for the image to be encoded. Since the residual image and the image to be encoded have the same resolution, the N regions in the residual image correspond one-to-one with the N regions in the image to be encoded. Furthermore, the texture complexity weight corresponding to the i-th region of the image to be encoded can be used as the texture complexity weight corresponding to the i-th region of the residual image. In this way, the texture complexity weights corresponding to the N regions in the residual image can be determined.

[0161] For example, the texture complexity weight may be a decimal between 0 and 1.

[0162] For example, after obtaining the texture complexity weights corresponding to N regions in the residual image, the texture intensity corresponding to each region in the residual image can be adjusted according to the texture complexity weights corresponding to each region in the residual image to obtain an enhanced image corresponding to the reconstructed image, which can be referred to S404 to S405:

[0163] S404 , performing residual updating on the residual image based on the texture complexity weights corresponding to the respective regions in the residual image to obtain a residual updated image.

[0164] S405: Add the reconstructed image and the residual updated image to obtain an enhanced image.

[0165] For example, S404 to S405 can refer to the description of S203 to S204 above, which will not be repeated here.

[0166] In this way, by controlling the texture complexity weight within the range of 0 to 1 based on the texture complexity of the image to be encoded, the texture enhancement corresponding to each area in the reconstructed image can be adjusted. This can enhance the texture of the texture areas (higher texture complexity) in the reconstructed image while avoiding the generation of false stripes in the non-texture areas (lower texture complexity) of the reconstructed image, thereby reducing the visual distortion of the reconstructed image. In addition, by controlling the texture complexity weight within the range of 0 to 1, the textures corresponding to adjacent areas are softer, which can avoid the problem of inconsistent texture enhancement effects in adjacent areas.

[0167] In addition, compared with the prior art which uses two models for post-processing, the present application uses only one model, which reduces the computational complexity.

[0168] Again, compared with the texture complexity weight determined based on the reconstructed image, the texture complexity weight determined based on the image to be encoded is more accurate, and can more accurately control the attenuation of the image texture intensity, thereby further improving the image quality.

[0169] Figure 5 FIG. 1 is a schematic diagram showing an exemplary processing process. Figure 5 In an embodiment, a non-generative adversarial network (non-GANEF) and a generative adversarial network (GANEF) are first used to filter the reconstructed image to obtain a base-fidelity image and a texture-enhanced image, respectively. Then, based on the texture complexity corresponding to each region in the reconstructed image or the base-fidelity image, the weighted calculation factors corresponding to the base-fidelity image and the texture-enhanced image are determined. Finally, based on the weighted calculation factors, the base-fidelity image and the texture-enhanced image are weightedly fused to obtain the final enhanced image.

[0170] S501: Perform image enhancement on the reconstructed image to obtain a texture enhanced image.

[0171] For example, after receiving the bitstream, the decoder can decode the bitstream to obtain a reconstructed image, and then use the image enhancement model to enhance the reconstructed image to obtain an intermediate image. For example, the intermediate image is a texture-enhanced image.

[0172] Figure 6 FIG. 1 is a schematic diagram illustrating an exemplary image enhancement process.

[0173] In one possible approach, the output of the trained image enhancement model is a residual image. Thus, by inputting the reconstructed image into the image enhancement model, a residual image output by the image enhancement model can be obtained. Then, a texture enhanced image can be determined based on the residual image and the reconstructed image. Optionally, the pixel values ​​of the corresponding pixels in the residual image and the reconstructed image can be added to obtain a texture enhanced image, such as Figure 6 (1) shown.

[0174] In one possible approach, the output of the trained image enhancement model is a texture enhanced image. Thus, after inputting the reconstructed image into the image enhancement model, a texture enhanced image can be directly obtained, such as Figure 6 (2) shown.

[0175] Illustratively, the texture enhanced image has the same resolution as the reconstructed image.

[0176] S502: Perform image fidelity preservation on the reconstructed image to obtain a basic fidelity image.

[0177] Exemplarily, a preset image fidelity model may be used to reconstruct an image for image fidelity to obtain a basic fidelity image.

[0178] Exemplarily, the network used by the image fidelity model can be a non-generative adversarial network, such as a convolutional neural network, etc., and this application does not impose any restrictions on this.

[0179] Exemplarily, the training process of an image fidelity model can be as follows: multiple sets of training data are collected, where one set of training data includes an image to be encoded and a reconstructed image obtained by decoding the code stream of the image to be encoded. For each set of training data, one set of training data is input into the image fidelity model, which performs forward calculations on the reconstructed image and outputs a base fidelity image. A loss function value is calculated based on the base fidelity image output by the image fidelity model and the image to be encoded in the training data. The weight parameters of the image fidelity model are adjusted with the goal of minimizing the loss function value. The image fidelity model can then be trained using the multiple sets of training data collected in the above manner until the number of training times for the image fidelity model equals a preset number of training times, the loss function value of the image fidelity model is less than or equal to a loss function threshold, or the performance of the image fidelity model meets a preset performance condition. Training of the image fidelity model is then terminated, resulting in a trained image fidelity model. Exemplarily, the preset number of training times, loss function threshold, and preset performance condition can all be set as required and are not limited in this application.

[0180] It should be noted that this application does not limit the execution order of S501 and S502.

[0181] Exemplarily, the texture-enhanced image and the base fidelity image have the same resolution as the reconstructed image.

[0182] S503 : Based on the basic fidelity image, generate a texture complexity weight corresponding to each region in the texture enhanced image.

[0183] Exemplarily, S503 may include S5031a to S5034a:

[0184] S5031a: Divide the basic fidelity image and the texture enhanced image into N regions respectively according to a preset partitioning rule.

[0185] S5032a: Determine the texture complexity of N regions in the basic fidelity image respectively.

[0186] S5033a: Determine texture complexity weights corresponding to the N regions in the texture-enhanced image based on the texture complexities of the N regions in the base fidelity image.

[0187] For example, S5031a to S5033a may refer to the description of S2021 to S2023 above, which will not be repeated here.

[0188] In one possible approach, a texture complexity weight corresponding to each region in the texture-enhanced image may be generated based on the reconstructed image, as shown in S5031b to S5033b:

[0189] S5031b: Divide the reconstructed image and the texture-enhanced image into N regions respectively according to a preset partitioning rule.

[0190] S5032b: Determine the texture complexity of N regions in the reconstructed image respectively.

[0191] S5033b: Determine texture complexity weights corresponding to the N regions in the texture-enhanced image based on the texture complexities of the N regions in the reconstructed image.

[0192] For example, S5031b to S5033b may refer to the description of S2021 to S2023 above, which will not be repeated here.

[0193] S504 , performing weighted fusion on the basic fidelity image and the texture enhanced image according to the texture complexity weights corresponding to the respective regions in the texture enhanced image, to obtain an enhanced image.

[0194] For example, after dividing both the texture-enhanced image and the base-fidelity image into N regions, the regions of the texture-enhanced image and the base-fidelity image are in one-to-one correspondence. Thus, each region in the texture-enhanced image can be weightedly fused with the corresponding region in the base-fidelity image to obtain an enhanced image.

[0195] Exemplarily, S504 may include S5041 to S5042:

[0196] S5041 , determining weighted calculation weights corresponding to the N regions in the texture-enhanced image and weighted calculation weights corresponding to the N regions in the basic fidelity image according to the texture complexity weights corresponding to the N regions in the texture-enhanced image.

[0197] For example, the texture complexity weights corresponding to the N regions in the texture-enhanced image can be used as the weighted calculation weights for the N regions in the texture-enhanced image. Furthermore, the difference between 1 and the texture complexity weights corresponding to the N regions in the texture-enhanced image can be used as the weighted calculation weights for the N regions in the base fidelity image.

[0198] For example, for the i-th region, the texture complexity weight corresponding to the i-th region of the texture enhanced image is ratio_i, then the weighted calculation weight of the i-th region of the texture enhanced image can be ratio_i, and the weighted calculation weight of the i-th region of the basic fidelity image is 1-ratio_i.

[0199] S5042: Perform weighted calculation on the N regions in the texture-enhanced image and the N regions in the base fidelity image according to the weighted calculation weights corresponding to the N regions in the texture-enhanced image and the weighted calculation weights corresponding to the N regions in the base fidelity image to obtain an enhanced image.

[0200] Exemplarily, for the i-th region, based on the weighted calculation weight of the i-th region in the texture-enhanced image and the weighted calculation weight of the i-th region in the basic fidelity image, weighted calculation is performed on each pixel point in the i-th region in the texture-enhanced image and each pixel point in the i-th region in the basic fidelity image to obtain an enhanced image of the i-th region.

[0201] Exemplarily, the weighted calculation weights corresponding to the N regions in the texture-enhanced image can be multiplied by the N regions in the texture-enhanced image to obtain a first product; the weighted calculation weights corresponding to the N regions in the basic fidelity image can be multiplied by the N regions in the basic fidelity image to obtain a second product; the first product and the second product are added to obtain an enhanced image corresponding to the reconstructed image.

[0202] For example, the weighted calculation weight of the i-th region in the texture-enhanced image is ratio_i, and the pixel value of a pixel point (j, k) in the i-th region in the texture-enhanced image is E1(j, k); the weighted calculation weight of the i-th region in the basic fidelity image is 1-ratio_i, and the pixel value of a pixel point (j, k) in the i-th region in the basic fidelity image is E2(j, k). Then the pixel value of the pixel point (j, k) in the i-th region of the reconstructed image after image enhancement is R(i, j) = ratio_i*E1(i, j) + (1-ratio_i)*E2(i, j).

[0203] In this way, by controlling the texture complexity weight within the range of 0 to 1 based on the texture complexity of the reconstructed image, the texture enhancement corresponding to each area in the reconstructed image can be adjusted. This can enhance the texture of the textured areas (higher texture complexity) in the reconstructed image while avoiding the generation of false streaks in the non-textured areas (lower texture complexity) in the reconstructed image, thereby reducing the visual distortion of the reconstructed image. In addition, by controlling the texture complexity weight within the range of 0 to 1, the textures corresponding to adjacent areas are softer, which can avoid the problem of inconsistent texture enhancement effects in adjacent areas.

[0204] Figure 7 FIG. 1 is a schematic diagram showing an exemplary processing process. Figure 7 In an embodiment of the present invention, a non-generative adversarial network (non-GANEF) and a generative adversarial network (GANEF) are first used to filter the reconstructed image to obtain a base-fidelity image and a texture-enhanced image, respectively. Then, based on the texture complexity of each region in the image to be encoded or the base-fidelity image corresponding to the reconstructed image, the weighted calculation factors corresponding to the base-fidelity image and the texture-enhanced image are determined. Finally, based on the weighted calculation factors, the base-fidelity image and the texture-enhanced image are weightedly fused to obtain the final enhanced image.

[0205] S701 , decoding a code stream to obtain texture complexity weights corresponding to respective regions in a reconstructed image and an image to be encoded.

[0206] Exemplarily, the encoder may generate a texture complexity weight based on the image to be encoded, which may include S7011 to S7013:

[0207] S7011: Divide the image to be encoded into N areas according to a preset partitioning rule.

[0208] S7012: Determine the texture complexity of N regions in the image to be encoded respectively.

[0209] S7013 : Determine texture complexity weights corresponding to the N regions in the image to be encoded based on the texture complexities of the N regions in the image to be encoded.

[0210] For example, S7011 to S7013 can refer to the description of S2021 to S2023 above, which will not be repeated here.

[0211] S702: Perform image enhancement on the reconstructed image to obtain a texture enhanced image.

[0212] For example, S702 may refer to the description of S502 above, which will not be repeated here.

[0213] S703 , determining a texture enhancement weight corresponding to each region in the texture-enhanced image according to the texture complexity weight corresponding to each region in the image to be encoded.

[0214] Exemplarily, after obtaining a texture-enhanced image, the texture-enhanced image can be divided into N regions according to a pre-set partitioning rule. The region division method for the texture-enhanced image is the same as the region division method for the image to be encoded, and the texture-enhanced image and the image to be encoded have the same resolution. Therefore, the regions in the texture-enhanced image correspond one-to-one with the regions in the image to be encoded. Furthermore, the texture complexity weight corresponding to the i-th region of the image to be encoded can be used as the texture complexity weight corresponding to the i-th region of the texture-enhanced image. In this way, the texture complexity weights corresponding to the N regions in the texture-enhanced image can be determined.

[0215] For example, the texture complexity weight may be a decimal between 0 and 1.

[0216] For example, after obtaining the texture complexity weights corresponding to N regions in the texture-enhanced image, the texture intensity corresponding to each region in the texture-enhanced image may be adjusted according to the texture complexity weights corresponding to each region in the texture-enhanced image to obtain an enhanced image corresponding to the reconstructed image, which may refer to S704 to S705:

[0217] S704: Perform image fidelity preservation on the reconstructed image to obtain a basic fidelity image.

[0218] For example, S704 may refer to the description of S502 above, which will not be repeated here.

[0219] For example, the present application does not limit the execution order of S704 and S702.

[0220] S705 , performing weighted fusion on the basic fidelity image and the texture enhanced image according to the texture complexity weight corresponding to each region in the texture enhanced image to obtain an enhanced image.

[0221] For example, S705 may refer to the description of S504 above, which will not be repeated here.

[0222] In this way, by controlling the texture complexity weight within the range of 0 to 1 based on the texture complexity of the image to be encoded, the texture enhancement corresponding to each area in the reconstructed image can be adjusted. This can enhance the texture of the texture areas (higher texture complexity) in the reconstructed image while avoiding the generation of false stripes in the non-texture areas (lower texture complexity) of the reconstructed image, thereby reducing the visual distortion of the reconstructed image. In addition, by controlling the texture complexity weight within the range of 0 to 1, the textures corresponding to adjacent areas are softer, which can avoid the problem of inconsistent texture enhancement effects in adjacent areas.

[0223] Again, compared with the texture complexity weight determined based on the reconstructed image, the texture complexity weight determined based on the image to be encoded is more accurate, and can more accurately control the attenuation of the image texture intensity, thereby further improving the image quality.

[0224] Figure 8 FIG. 1 is a schematic diagram of an image processing device shown as an example.

[0225] Reference Figure 8 Exemplarily, the image processing apparatus includes: an image acquisition module 801, an image enhancement module 802, a texture weight determination module 803, and a texture attenuation module 804, wherein:

[0226] An image acquisition module 801 is used to acquire a reconstructed image;

[0227] An image enhancement module 802 is used to perform image enhancement on the reconstructed image to obtain an intermediate image;

[0228] The texture weight determination module 803 is used to determine the texture complexity weight corresponding to each region in the intermediate image, where the texture complexity weight is a number between 0 and 1;

[0229] The texture attenuation module 804 is configured to attenuate the texture intensity corresponding to each region in the intermediate image according to the texture complexity weight corresponding to each region in the intermediate image, so as to obtain an enhanced image corresponding to the reconstructed image.

[0230] Exemplarily, the intermediate image is a residual image;

[0231] The texture attenuation module 804 includes:

[0232] The residual update module is used to multiply the pixel value of each pixel in the residual image by the texture complexity weight corresponding to the region to which each pixel belongs, so as to obtain a residual update image;

[0233] The image generation module is used to update the image and reconstruct the image according to the residual to generate an enhanced image.

[0234] Exemplarily, the image generation module is specifically configured to add the residual updated image and the reconstructed image to obtain an enhanced image.

[0235] Exemplarily, the intermediate image is a texture-enhanced image, and the apparatus further includes:

[0236] The image fidelity module is used to perform image fidelity on the reconstructed image to obtain a basic fidelity image.

[0237] Exemplarily, the texture weight determination module 803 is specifically used to divide the basic fidelity image and the texture enhanced image into N regions according to preset partitioning rules, where N is a positive integer; determine the texture complexity of the N regions in the basic fidelity image respectively; and determine the texture complexity weights corresponding to the N regions in the texture enhanced image based on the texture complexity of the N regions in the basic fidelity image.

[0238] Exemplarily, the texture attenuation module 804 includes:

[0239] A weighted weight determination module is used to determine weighted calculation weights corresponding to the N regions in the texture-enhanced image and weighted calculation weights corresponding to the N regions in the basic fidelity image according to the texture complexity weights corresponding to the N regions in the texture-enhanced image;

[0240] The weighted calculation module is used to perform weighted calculation on the N areas in the texture-enhanced image and the N areas in the basic fidelity image based on the weighted calculation weights corresponding to the N areas in the texture-enhanced image and the weighted calculation weights corresponding to the N areas in the basic fidelity image, so as to obtain an enhanced image corresponding to the reconstructed image.

[0241] Exemplarily, the weighted calculation module is specifically used to multiply the weighted calculation weights corresponding to N areas in the texture-enhanced image by the N areas in the texture-enhanced image to obtain a first product; multiply the weighted calculation weights corresponding to N areas in the basic fidelity image by the N areas in the basic fidelity image to obtain a second product; add the first product and the second product to obtain an enhanced image corresponding to the reconstructed image.

[0242] Exemplarily, the weighted weight determination module is specifically used to determine the texture complexity weights corresponding to the N regions in the texture-enhanced image as the weighted calculation weights corresponding to the N regions in the texture-enhanced image; and determine the difference between 1 and the texture complexity weights corresponding to the N regions in the texture-enhanced image as the weighted calculation weights corresponding to the N regions in the basic fidelity image.

[0243] Exemplarily, the texture weight determination module 803 is specifically configured to decode the texture complexity weights corresponding to N regions in the intermediate image from the received bitstream, where N is a positive integer.

[0244] Exemplarily, the texture weight determination module 803 is specifically used to divide the reconstructed image and the intermediate image into N regions, where N is a positive integer, according to a preset partitioning rule; determine the texture complexity of the N regions in the reconstructed image respectively; and determine the texture complexity weights corresponding to the N regions in the intermediate image based on the texture complexity of the N regions in the reconstructed image.

[0245] In one example, Figure 9A schematic block diagram of an apparatus 900 according to an embodiment of the present application is shown. The apparatus 900 may include: a processor 901 and a transceiver / transceiver pin 902 , and optionally, a memory 903 .

[0246] The various components of the device 900 are coupled together via a bus 904, wherein the bus 904 includes, in addition to a data bus, a power bus, a control bus, and a status signal bus. However, for the sake of clarity, all buses are referred to as bus 904 in the figure.

[0247] Optionally, the memory 903 may be used for instructions in the aforementioned method embodiment. The processor 901 may be used to execute instructions in the memory 903 and control the receiving pin to receive a signal and control the transmitting pin to send a signal.

[0248] The apparatus 900 may be the electronic device or a chip of the electronic device in the above method embodiment.

[0249] Among them, all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0250] This embodiment further provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the image processing method in the above-mentioned embodiment.

[0251] This embodiment further provides a computer program product. When the computer program product is run on a computer, it enables the computer to execute the above-mentioned related steps to implement the image processing method in the above-mentioned embodiment.

[0252] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory to enable the chip to execute the image processing method in the above-mentioned method embodiments.

[0253] Among them, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0254] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0255] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0256] Units described as separate components may or may not be physically separate, and components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0257] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0258] Any content of each embodiment of this application, as well as any content of the same embodiment, can be freely combined. Any combination of the above content is within the scope of this application.

[0259] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0260] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

[0261] The steps of the method or algorithm described in conjunction with the disclosure of the embodiments of the present application can be implemented in a hardware manner, or can be implemented by a processor executing a software instruction. The software instruction can be composed of corresponding software modules, and the software module can be stored in a random access memory (Random Access Memory, RAM), a flash memory, a read-only memory (Read Only Memory, ROM), an erasable programmable read-only memory (Erasable Programmable ROM, EPROM), an electrically erasable programmable read-only memory (Electrically EPROM, EEPROM), a register, a hard disk, a mobile hard disk, a read-only compact disc (CD-ROM) or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0262] Those skilled in the art will appreciate that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer-readable storage media and communication media, wherein communication media include any media that facilitates the transmission of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0263] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire a reconstructed image; performing image enhancement on the reconstructed image to obtain an intermediate image; Determining a texture complexity weight corresponding to each region in the intermediate image, wherein the texture complexity weight is a number between 0 and 1; According to the texture complexity weights corresponding to the respective regions in the intermediate image, the texture intensities corresponding to the respective regions in the intermediate image are attenuated respectively to obtain an enhanced image corresponding to the reconstructed image.

2. The method according to claim 1, characterized in that The intermediate image is a residual image; The attenuating the texture intensity corresponding to each region in the intermediate image according to the texture complexity weight corresponding to each region in the intermediate image to obtain an enhanced image corresponding to the reconstructed image includes: Multiplying the pixel value of each pixel in the residual image by the texture complexity weight corresponding to the region to which each pixel belongs to obtain a residual update image; The enhanced image is generated according to the residual updated image and the reconstructed image.

3. The method according to claim 2, characterized in that Generating the enhanced image according to the residual updated image and the reconstructed image includes: The residual updated image and the reconstructed image are added to obtain the enhanced image.

4. The method according to claim 1, wherein The intermediate image is a texture-enhanced image, and the method further comprises: Perform image fidelity on the reconstructed image to obtain a basic fidelity image.

5. The method according to claim 4, characterized in that Determining the texture complexity weight corresponding to each region in the intermediate image includes: According to a preset partitioning rule, the basic fidelity image and the texture enhanced image are divided into N regions respectively, where N is a positive integer; Determining the texture complexity of N regions in the basic fidelity image respectively; Based on the texture complexities of the N regions in the basic fidelity image, texture complexity weights corresponding to the N regions in the texture enhanced image are determined.

6. The method according to claim 5, characterized in that The attenuating the texture intensity corresponding to each region in the intermediate image according to the texture complexity weight corresponding to each region in the intermediate image to obtain an enhanced image corresponding to the reconstructed image includes: Determining weighted calculation weights corresponding to the N regions in the texture-enhanced image and weighted calculation weights corresponding to the N regions in the basic fidelity image, based on texture complexity weights corresponding to the N regions in the texture-enhanced image; According to the weighted calculation weights corresponding to the N regions in the texture-enhanced image and the weighted calculation weights corresponding to the N regions in the basic fidelity image, a weighted calculation is performed on the N regions in the texture-enhanced image and the N regions in the basic fidelity image to obtain an enhanced image corresponding to the reconstructed image.

7. The method according to claim 6, characterized in that The step of performing weighted calculation on the N regions in the texture-enhanced image and the N regions in the base fidelity image based on the weighted calculation weights respectively corresponding to the N regions in the texture-enhanced image and the weighted calculation weights respectively corresponding to the N regions in the base fidelity image to obtain an enhanced image corresponding to the reconstructed image includes: multiplying the weighted calculation weights corresponding to the N regions in the texture-enhanced image by the N regions in the texture-enhanced image to obtain a first product; multiplying the weighted calculation weights corresponding to the N regions in the basic fidelity image by the N regions in the basic fidelity image to obtain a second product; The first product and the second product are added together to obtain an enhanced image corresponding to the reconstructed image.

8. The method according to claim 6, characterized in that The determining, based on the texture complexity weights corresponding to the N regions in the texture-enhanced image, weighted calculation weights corresponding to the N regions in the texture-enhanced image and weighted calculation weights corresponding to the N regions in the basic fidelity image, includes: Determining the texture complexity weights corresponding to the N regions in the texture-enhanced image as weighted calculation weights corresponding to the N regions in the texture-enhanced image; The difference between 1 and the texture complexity weights corresponding to the N regions in the texture enhanced image is determined as the weighted calculation weights corresponding to the N regions in the basic fidelity image.

9. The method according to any one of claims 1 to 8, characterized in that Determining the texture complexity weights corresponding to the respective regions in the intermediate image includes: The texture complexity weights corresponding to the N regions in the intermediate image are decoded from the received code stream, where N is a positive integer.

10. The method according to claim 1 or 2 or 3 or 4 or 6 or 7 or 8 or 9, characterized in that Determining the texture complexity weights corresponding to the respective regions in the intermediate image includes: According to a preset partitioning rule, the reconstructed image and the intermediate image are divided into N regions, where N is a positive integer; Determining the texture complexity of N regions in the reconstructed image respectively; Based on the texture complexities of the N regions in the reconstructed image, texture complexity weights corresponding to the N regions in the intermediate image are determined.

11. An image processing device, characterized in that: The device comprises: An image acquisition module, used for acquiring a reconstructed image; An image enhancement module, configured to perform image enhancement on the reconstructed image to obtain an intermediate image; a texture weight determination module, configured to determine a texture complexity weight corresponding to each region in the intermediate image, wherein the texture complexity weight is a number between 0 and 1; The texture attenuation module is used to attenuate the texture intensity corresponding to each area in the intermediate image according to the texture complexity weight corresponding to each area in the intermediate image, so as to obtain an enhanced image corresponding to the reconstructed image.

12. The device according to claim 11, characterized in that The intermediate image is a residual image; The texture attenuation module includes: A residual updating module is used to multiply the pixel value of each pixel in the residual image by the texture complexity weight corresponding to the region to which each pixel belongs, so as to obtain a residual updated image; An image generation module is configured to generate the enhanced image based on the residual updated image and the reconstructed image.

13. The device according to claim 12, characterized in that The image generation module is specifically configured to add the residual updated image and the reconstructed image to obtain the enhanced image.

14. The device according to claim 11, characterized in that The intermediate image is a texture-enhanced image, and the apparatus further comprises: The image fidelity module is used to perform image fidelity on the reconstructed image to obtain a basic fidelity image.

15. The device according to claim 14, characterized in that The texture weight determination module is specifically configured to divide the base fidelity image and the texture enhanced image into N regions, where N is a positive integer, according to a preset partitioning rule; and determine the texture complexity of the N regions in the base fidelity image respectively; Based on the texture complexities of the N regions in the basic fidelity image, texture complexity weights corresponding to the N regions in the texture enhanced image are determined.

16. The device according to claim 15, characterized in that The texture attenuation module includes: a weighted weight determination module, configured to determine weighted calculation weights corresponding to the N regions in the texture-enhanced image and weighted calculation weights corresponding to the N regions in the base fidelity image, based on the texture complexity weights corresponding to the N regions in the texture-enhanced image; A weighted calculation module is used to perform weighted calculation on the N areas in the texture-enhanced image and the N areas in the basic fidelity image based on the weighted calculation weights corresponding to the N areas in the texture-enhanced image and the weighted calculation weights corresponding to the N areas in the basic fidelity image, so as to obtain an enhanced image corresponding to the reconstructed image.

17. The device according to claim 16, characterized in that The weighted calculation module is specifically used to multiply the weighted calculation weights corresponding to the N regions in the texture-enhanced image by the N regions in the texture-enhanced image to obtain a first product; multiply the weighted calculation weights corresponding to the N regions in the basic fidelity image by the N regions in the basic fidelity image to obtain a second product; and add the first product and the second product to obtain an enhanced image corresponding to the reconstructed image.

18. The device according to claim 16, characterized in that The weighted weight determination module is specifically configured to determine the texture complexity weights corresponding to the N regions in the texture-enhanced image as the weighted calculation weights corresponding to the N regions in the texture-enhanced image; The difference between 1 and the texture complexity weights corresponding to the N regions in the texture enhanced image is determined as the weighted calculation weights corresponding to the N regions in the basic fidelity image.

19. The device according to any one of claims 11 to 18, characterized in that The texture weight determination module is specifically configured to decode the texture complexity weights corresponding to the N regions in the intermediate image from the received code stream, where N is a positive integer.

20. The device according to claim 11 or 12 or 13 or 14 or 16 or 17 or 18 or 19, characterized in that The texture weight determination module is specifically configured to divide the reconstructed image and the intermediate image into N regions, where N is a positive integer, according to a preset partitioning rule; and determine the texture complexity of each of the N regions in the reconstructed image; Based on the texture complexities of the N regions in the reconstructed image, texture complexity weights corresponding to the N regions in the intermediate image are determined.

21. An electronic device, characterized in that: include: a memory and a processor, the memory being coupled to the processor; The memory stores program instructions, and when the processor executes the program instructions, the electronic device executes the image processing method according to any one of claims 1 to 10.

22. A chip, characterized in that: The electronic device comprises one or more interface circuits and one or more processors; the interface circuit is used to receive a signal from a memory of the electronic device and send the signal to the processor, wherein the signal includes a computer instruction stored in the memory; when the processor executes the computer instruction, the electronic device executes the image processing method according to any one of claims 1 to 10.

23. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program runs on a computer or a processor, the computer or the processor executes the image processing method according to any one of claims 1 to 10.

24. A computer program product, characterized in that The computer program product comprises a software program, and when the software program is executed by a computer or a processor, the steps of the method according to any one of claims 1 to 10 are performed.

Images