Image processing method, device, equipment and storage medium

Abstract

The embodiment of the present application discloses an image processing method, apparatus, device and storage medium, wherein the image processing method comprises: performing image processing on an initial image to be processed according to an image processing task to obtain a target image, and determining a reference image whose resolution matches the resolution of the target image based on the initial image; performing structural information extraction processing on the target image to obtain a structural information extraction result; wherein structural information refers to information related to the image quality effect of the image, and the structural information extraction result is used to indicate: pixel points used to describe structural information in the target image; according to the structural information extraction result and the reference image, the intensity of the image quality effect of the target image is controlled. The embodiment of the present application can enhance the visual effect of the final image.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image processing method, apparatus, device and storage medium. Background Art

[0002] At present, with the continuous development of Internet technology, image processing for various image processing tasks has been widely used; for example, image super-resolution tasks, image denoising tasks, and image decompression distortion tasks. In the prior art, any image can be processed according to any image processing task to obtain an image after image processing, but the image quality of the image after image processing may be poor or too distorted, which is difficult to meet the requirements of visual effects, resulting in poor visual effects of the image after image processing. Summary of the invention

[0003] The embodiments of the present application provide an image processing method, apparatus, device and storage medium, which can further control the intensity of the image quality effect of the image obtained by performing the image processing task, thereby improving the visual effect of the final image.

[0004] On the one hand, an embodiment of the present application provides an image processing method, the method comprising:

[0005] Performing image processing on an initial image to be processed according to an image processing task to obtain a target image, and determining a reference image having a resolution matching a resolution of the target image based on the initial image;

[0006] Performing structural information extraction processing on the target image to obtain a structural information extraction result; wherein the structural information refers to information related to the image quality effect, and the structural information extraction result is used to indicate: pixel points used to describe the structural information in the target image;

[0007] The intensity of the image quality effect of the target image is controlled according to the structural information extraction result and the reference image.

[0008] On the other hand, an embodiment of the present application provides an image processing device, the device comprising:

[0009] A processing unit, configured to perform image processing on an initial image to be processed according to an image processing task to obtain a target image, and determine a reference image having a resolution matching a resolution of the target image based on the initial image;

[0010] A structure extraction unit is used to extract structure information from the target image to obtain a structure information extraction result; wherein the structure information refers to information related to the image quality effect, and the structure information extraction result is used to indicate: pixel points used to describe the structure information in the target image;

[0011] An intensity control unit is used to control the intensity of the image quality effect of the target image according to the structural information extraction result and the reference image.

[0012] In another aspect, an embodiment of the present application provides a computer device, the computer device comprising a processor and a memory, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, the following steps are implemented:

[0013] Performing image processing on an initial image to be processed according to an image processing task to obtain a target image, and determining a reference image having a resolution matching a resolution of the target image based on the initial image;

[0014] Performing structural information extraction processing on the target image to obtain a structural information extraction result; wherein the structural information refers to information related to the image quality effect, and the structural information extraction result is used to indicate: pixel points used to describe the structural information in the target image;

[0015] The intensity of the image quality effect of the target image is controlled according to the structural information extraction result and the reference image.

[0016] In another aspect, an embodiment of the present application provides a computer storage medium, wherein the computer storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the following steps:

[0017] Performing image processing on an initial image to be processed according to an image processing task to obtain a target image, and determining a reference image having a resolution matching a resolution of the target image based on the initial image;

[0018] Performing structural information extraction processing on the target image to obtain a structural information extraction result; wherein the structural information refers to information related to the image quality effect, and the structural information extraction result is used to indicate: pixel points used to describe the structural information in the target image;

[0019] The intensity of the image quality effect of the target image is controlled according to the structural information extraction result and the reference image.

[0020] On the other hand, an embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the above-mentioned image processing method is implemented.

[0021] After obtaining the target image and the reference image, the embodiment of the present application can first perform structural information extraction processing on the target image to obtain the structural information extraction result, so that the structural information extraction result can be used to indicate: the pixel points used to describe the structural information in the target image, and the structural information refers to the information related to the image quality effect of the image, so that the structural information extraction result can be used to indicate: the pixel points used to describe the information related to the image quality effect in the target image; then, according to the structural information extraction result and the reference image, the intensity of the image quality effect of the target image is controlled, so that not only the image quality effect of the target image can be optimized, but also the image quality effect of the target image can be weakened, so that the image quality effect of the image is more moderate, thereby meeting the visual effect requirements and effectively improving the visual effect of the final image. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0023] Figure 1a It is a flowchart of an image processing solution provided in an embodiment of the present application;

[0024] Figure 1b is a schematic diagram of interaction between a terminal and a server provided in an embodiment of the present application;

[0025] Figure 2 It is a flowchart of an image processing method provided in an embodiment of the present application;

[0026] Figure 3a is a schematic diagram of an optimized image quality effect provided by an embodiment of the present application;

[0027] Figure 3b is a schematic diagram of a weakened image quality effect provided by an embodiment of the present application;

[0028] Figure 4 is a flowchart of another image processing method provided in an embodiment of the present application;

[0029] Figure 5 It is a flowchart of another image processing method provided in an embodiment of the present application;

[0030] Figure 6a is a schematic diagram of another method for optimizing image quality provided by an embodiment of the present application;

[0031] Figure 6b is a schematic diagram of another image quality weakening effect provided by an embodiment of the present application;

[0032] Figure 7 is a schematic diagram of intensity control for a target area provided in an embodiment of the present application;

[0033] Figure 8 is a schematic diagram of a replaced regional control image provided in an embodiment of the present application;

[0034] Fig. 9 is a structural schematic diagram of an image processing device provided in an embodiment of the present application;

[0035] Fig.10 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0037] In order to control the intensity of the image quality effect of the target image to improve the visual effect of the image, the embodiment of the present application proposes an image processing solution: Figure 1a As shown, the general principle of the image processing scheme proposed in the embodiment of the present application is as follows: First, the initial image to be processed can be processed according to the image processing task to obtain a target image, and a reference image whose resolution matches the resolution of the target image can be determined based on the initial image. After obtaining the target image and the reference image, the intensity of the image quality effect of the target image can be controlled based on the reference image; specifically, the target image can be subjected to structural information extraction processing to obtain a structural information extraction result. Among them, the structural information extraction result is used to indicate: the pixel points used to describe the structural information in the target image, the so-called structural information refers to information related to the image quality effect of the image, such as edge information and detail information; that is, the structural information extraction result can be used to indicate: the pixel points used to describe the edge information and detail information in the target image. Then, the intensity of the image quality effect of the target image can be controlled based on the structural information extraction result and the reference image.

[0038] Practice has shown that the image processing scheme proposed in the embodiments of the present application can have at least the following beneficial effects: ① A structural information extraction result can be obtained, and the structural information extraction result can be used to indicate pixel points used to describe information related to the image quality effect in the target image, so as to facilitate intensity control of the image quality effect of the target image through the structural information extraction result; ② By controlling the intensity of the image quality effect of the target image, not only the image quality effect of the target image can be enhanced, but also the image quality effect of the target image can be weakened, so that the image quality effect of the image after intensity control can better meet the requirements of visual effects, thereby effectively improving the visual effect of the final image.

[0039] In a specific implementation, the above-mentioned image processing scheme can be executed by a computer device, which can be a terminal or a server; wherein the terminal mentioned here can include but is not limited to: smart phones, tablet computers, laptops, desktop computers, smart watches, intelligent voice interaction devices, smart home appliances, vehicle terminals, aircraft, etc.; various clients (applications, APPs) can be run in the terminal, such as video playback clients, social clients, browser clients, information flow clients, education clients, etc. The server mentioned here can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing (cloud computing), cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms, etc.; so-called cloud computing is a computing model that distributes computing tasks on a resource pool composed of a large number of computers, so that various application systems can obtain computing power, storage space and information services as needed. Furthermore, the computer device mentioned in the embodiments of the present application may be located outside or inside the blockchain network, without limitation. The so-called blockchain network is a network composed of a peer-to-peer network (P2P network) and a blockchain, and the blockchain refers to a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm. It is essentially a decentralized database, a string of data blocks (or blocks) generated by cryptographic methods.

[0040] Alternatively, in other embodiments, the above-mentioned image processing scheme may also be jointly executed by the server and the terminal; the terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in this application. For example: the terminal may be responsible for performing image processing on the initial image to be processed according to the image processing task to obtain a target image, and based on the initial image, determine a reference image whose resolution matches the resolution of the target image, and then send the target image to the server; so that the server can extract structural information from the target image, obtain a structural information extraction result, and send the structural information extraction result to the terminal; the terminal then controls the intensity of the image quality effect of the target image based on the structural information extraction result and the reference image, such as Figure 1bAs shown. For another example, the terminal may be responsible for performing image processing on the initial image to be processed according to the image processing task to obtain a target image, and determine a reference image whose resolution matches the resolution of the target image based on the initial image, and then send the target image and the reference image to the server; so that the server can extract the structural information of the target image to obtain the structural information extraction result, and control the intensity of the image quality effect of the target image based on the structural information extraction result and the reference image. It should be understood that only two situations in which the terminal and the server jointly execute the above-mentioned image processing scheme are exemplarily described here, and are not exhaustive.

[0041] Based on the relevant description of the above image processing scheme, the embodiment of the present application proposes an image processing method, which can be executed by the computer device (terminal or server) mentioned above; or, the image processing method can be executed by the terminal and the server together. For the convenience of explanation, the following description will be based on the example of a computer device executing the image processing method; please refer to Figure 2 , the image processing method may include the following steps S201-S203:

[0042] S201 , performing image processing on an initial image to be processed according to an image processing task to obtain a target image, and determining a reference image whose resolution matches that of the target image based on the initial image.

[0043] In the embodiments of the present application, the image processing tasks may include but are not limited to the following: image super-resolution tasks, image denoising tasks, and image decompression distortion tasks, etc. Among them, image super-resolution refers to: restoring a high-resolution image from a low-resolution (LR) image or image sequence. Accordingly, when the image processing task is an image super-resolution task, the computer device may use a super-resolution model to perform image processing on the initial image to be processed to obtain a super-resolution image, thereby using the super-resolution image as the target image; it can be seen that in this case, the resolution of the target image is greater than the resolution of the initial image. Image denoising refers to: the process of reducing noise in a digital image. Accordingly, when the image processing task is an image denoising task, the computer device may use an image denoising model to perform image processing on the initial image to be processed to obtain a denoised image, thereby using the denoised image as the target image. Compression distortion refers to the obvious distortion of an image after it is compressed using destructive data. Image decompression distortion refers to: removing the effects of image compression distortion. Accordingly, when the image processing task is an image decompression distortion task, the computer device can use an image decompression distortion model to process the initial image to be processed, and obtain an image after decompression distortion, thereby using the image after decompression distortion as the target image. It should be noted that when the image processing task is an image denoising task or an image decompression distortion task, the resolution of the target image can be greater than the resolution of the initial image, or less than the resolution of the initial image, or equal to the resolution of the initial image, and this application does not limit this.

[0044] It is worth noting that the computer device may also acquire the initial image before performing image processing on the initial image to be processed according to the image processing task to obtain the target image and determining the reference image whose resolution matches the resolution of the target image based on the initial image. The initial image may be acquired in the following ways, including but not limited to:

[0045] The first acquisition method: the computer device may first obtain the image download link of the initial image, then download the initial image according to the image download link, and use the downloaded image as the initial image. Alternatively, if one or more images are stored in the storage space of the computer device itself, the computer may select at least one image from the one or more stored images, and use the selected image as the initial image.

[0046] The second acquisition method: The image processing method proposed in this application can be applied to various video information flow products. In this case, the computer device can obtain an image in any video in any video information flow product as the above-mentioned initial image; that is, the computer device can extract one or more frames of images from any of the above-mentioned videos, and use the extracted images as the initial images; it should be noted that the video information flow product here is a hot Internet product, and the information flow refers to the use of various methods to achieve information exchange, and with the development of Internet technology, the demand for high-definition video of computer equipment is increasing, so the computer device can extract the initial image from each video of each video information flow product, and then can control the intensity of the picture quality effect of the corresponding target image to improve the video quality.

[0047] In a third acquisition method, the computer device may be configured with a shooting component, and the computer device may collect images in the environment through the shooting component, and use the collected images as initial images.

[0048] S202, extracting structural information from the target image to obtain a structural information extraction result.

[0049] The structural information refers to information related to the image quality effect, and the structural information extraction result is used to indicate: pixel points used to describe the structural information in the target image.

[0050] It should be noted that structural information may include but is not limited to high-frequency information such as edge information and detail information, that is, structural information may include high-frequency information, and high-frequency information refers to: information that the frequency of the image is greater than the frequency threshold, and the frequency of the image refers to an indicator of the degree of grayscale value change, which is the gradient of grayscale in the plane space, then high-frequency information may refer to information with a relatively drastic grayscale value change, that is, high-frequency information may refer to information with a relatively drastic pixel value change. It is understandable that the edge information in the image (that is, the edge between an image and the background in the image) usually has a significant difference, that is, the edge information corresponds to a higher frequency, and the high-frequency information includes the edge information; and the detail information in the image (such as eyebrows, hair, etc.) is also an area where the grayscale value changes sharply, that is, the detail information also corresponds to a higher frequency, and the high-frequency information may also include the detail information.

[0051] In this case, the above structural information extraction result can be used not only to indicate: the pixel points used to describe the edge information in the target image, but also to indicate: the pixel points used to describe the detail information in the target image, and can also be used to indicate: the pixel points used to describe the edge information and detail information in the target image, etc.; this application does not limit this. Since high-frequency information includes but is not limited to edge information and detail information, etc., the structural information extraction result may include a high-frequency information position mask, which is used to indicate: the pixel points used to describe the high-frequency information in the target image.

[0052] S203: Control the intensity of the image quality effect of the target image according to the structural information extraction result and the reference image.

[0053] It should be understood that, in the process of controlling the intensity of the image quality effect of the target image, the computer device can control the intensity of the image quality effect of the target image according to the intensity control direction; based on this, when the intensity control direction is the direction of optimizing the image quality effect, the computer device can optimize the image quality effect of the target image, and when the intensity control direction is the direction of weakening the image quality effect, the computer device can also weaken the image quality effect of the target image. The present application does not limit the specific implementation method of the intensity control of the image quality effect.

[0054] For example, assuming that the above image processing task is an image super-resolution task, in the process of performing image processing (i.e., super-resolution processing) on ​​the initial image to be processed according to the image super-resolution task, unsatisfactory results may occur due to the various possibilities of the image quality of the initial image and the limitation of the generalization ability of the super-resolution model. Figure 3a As shown, assuming that the computer device performs image processing on the initial image to be processed according to the image super-resolution task, and obtains a target image with poor image quality, resulting in the target image not being clear enough, then the above-mentioned intensity control direction can be the direction of optimizing the image quality effect, and the computer device can optimize the image quality effect of the target image to strengthen and improve the clarity of the final image; for example, Figure 3b As shown, assuming that the computer device performs image processing on the initial image to be processed according to the image super-resolution task, and obtains a target image with an overly distorted image quality effect, then the above-mentioned intensity control direction can be a direction of weakening the image quality effect, and the computer device can weaken the image quality effect of the target image. It should be noted that this application uses the depth of lines to represent the image quality effect, that is, if the lines of the image are lighter, the image quality effect is weaker, and if the lines of the image are darker, the image quality effect is stronger, and when the computer device optimizes the image quality effect, the lines of the image will be deepened; when the computer device weakens the image quality effect, the lines of the image will be lighter.

[0055] After obtaining the target image and the reference image, the embodiment of the present application can first perform structural information extraction processing on the target image to obtain the structural information extraction result, so that the structural information extraction result can be used to indicate: the pixel points used to describe the structural information in the target image, and the structural information refers to the information related to the image quality effect of the image, so that the structural information extraction result can be used to indicate: the pixel points used to describe the information related to the image quality effect in the target image; then, according to the structural information extraction result and the reference image, the intensity of the image quality effect of the target image is controlled, so that not only the image quality effect of the target image can be optimized, but also the image quality effect of the target image can be weakened, so that the image quality effect of the image is more moderate, thereby meeting the visual effect requirements and effectively improving the visual effect of the final image.

[0056] See also Figure 4 , is a flowchart of another image processing method provided in an embodiment of the present application. The image processing method can be executed by the computer device (terminal or server) mentioned above; or, the image processing method can be executed by the terminal and the server together. For the convenience of explanation, the following description will be made by taking the computer device executing the image processing method as an example; please refer to Figure 4 , the image processing method may include the following steps S401-S405:

[0057] S401 , performing image processing on an initial image to be processed according to an image processing task to obtain a target image, and determining a reference image whose resolution matches that of the target image based on the initial image.

[0058] In a specific implementation, when the computer device determines a reference image whose resolution matches the resolution of the target image based on the initial image, the computer device may first determine the resolution of the initial image and the resolution of the target image; further, if the resolution of the initial image is equal to the resolution of the target image, the initial image is determined as the reference image; if the resolution of the initial image is greater than or less than the resolution of the target image, the resolution of the initial image is adjusted according to the resolution of the target image to obtain the reference image. It is worth noting that when the resolution of the initial image is greater than the resolution of the target image, the computer device adjusts the resolution of the initial image according to the resolution of the target image, and the specific implementation process of obtaining the reference image may include: down-sampling the initial image according to the resolution of the target image to adjust the resolution of the initial image to obtain the reference image, and the resolution of the reference image is equal to the resolution of the target image; when the resolution of the initial image is less than the resolution of the target image, the computer device adjusts the resolution of the initial image according to the resolution of the target image, and the specific implementation process of obtaining the reference image may include: up-sampling the initial image according to the resolution of the target image to adjust the resolution of the initial image to obtain the reference image, and the resolution of the reference image is equal to the resolution of the target image.

[0059] Among them, when the image processing task includes an image super-resolution task, the resolution of the target image is greater than the resolution of the initial image; in this case, the computer device can upsample the initial image according to the resolution of the target image to obtain a reference image. Specifically, the computer device can first train a super-resolution model, and input the low-resolution initial image into the super-resolution model to obtain a target image (i.e., the super-resolution image), and then upsample the initial image according to the resolution of the target image to obtain a reference image; accordingly, when determining the resolution of the target image, the computer device can calculate the resolution of the target image according to the super-resolution multiple when the image is processed according to the image super-resolution task. That is to say, the computer device can also upsample the initial image according to the super-resolution multiple. It should be noted that the computer device can use images of any size and any format (such as png, bmp, and jpg formats) as the above-mentioned initial image, and this application does not limit this.

[0060] S402, extracting structure information from the target image to obtain a structure information extraction result, where the structure information extraction result includes: a high-frequency information position mask constructed based on the absolute residual value of each pixel in the target image.

[0061] In a specific implementation, when a computer device extracts structural information from a target image and obtains a structural information extraction result, it can first determine the absolute residual value of each pixel point based on the pixel value of each pixel point in the target image and the pixel value of each pixel point in the reference image.

[0062] Accordingly, the specific implementation process of the computer device determining the absolute residual value of each pixel point according to the pixel value of each pixel point in the target image and the pixel value of each pixel point in the reference image may include: weakening the structural information of the reference image to obtain the reference image after weakening; and determining the absolute residual value of each pixel point according to the pixel value of each pixel point in the target image and the pixel value of each pixel point in the reference image after weakening. Wherein, when the computer device performs structural information weakening processing on the reference image to obtain the reference image after weakening, it may first normalize each pixel value in the reference image to obtain the normalized reference image; then, blur the normalized reference image to weaken the structural information of the reference image to obtain the weakened reference image. It is worth noting that the purpose of the computer device blurring the normalized reference image is to fine-tune each pixel value in the normalized reference image, thereby weakening the pixel value of the pixel point used to describe the structural information in the normalized reference image.

[0063] It is understandable that when the computer device normalizes each pixel value in the reference image, each pixel value in the reference image may be normalized to [0, 1], and the reference image may be blurred to obtain a weakened reference image. It is worth noting that the absolute residual value of each pixel determined by the computer device based on the pixel value of each pixel in the target image and the pixel value of each pixel in the weakened reference image may constitute absolute residual information; that is, the computer device may determine the absolute residual information based on the pixel value of each pixel in the target image and the pixel value of each pixel in the weakened reference image (i.e., the blurred reference image), and the absolute residual information may include the absolute residual value of each pixel. Based on this, the computer device may use formula 1.1 to calculate the absolute residual information:

[0064] res=|I sr -I up-blur Formula 1.1

[0065] Among them, |.| means taking the absolute value, I sr Used to represent the target image, that is, I sr Including the pixel value of each pixel in the target image, and I up-blur Used to represent the reference image after weakening, that is, Iup-blur It includes the pixel value of each pixel in the reference image after weakening processing; correspondingly, the absolute residual information res includes the absolute residual value of each pixel point.

[0066] It should be noted that, when the computer device blurs the reference image, it may perform Gaussian blur processing on the reference image to achieve blurring of the reference image, or may perform median filtering on the reference image to achieve blurring of the reference image, or may perform mean filtering on the reference image to achieve blurring of the reference image, or may perform bilateral filtering on the reference image to achieve blurring of the reference image, and so on; that is, the computer device may blur the reference image through Gaussian blur method, median filtering method, mean filtering method, bilateral filtering method, and so on, and the present application does not limit the specific implementation method of blurring.

[0067] Furthermore, the computer device can extract pixel points used to describe high-frequency information from the target image according to the size relationship between the absolute residual value of each pixel point and the high-frequency information threshold; wherein the absolute residual value of each extracted pixel point is greater than the high-frequency information threshold. Then, the computer device can construct a high-frequency information position mask based on the pixel extraction result, and the high-frequency information position mask includes the mask value of each pixel point; wherein the mask value of the extracted pixel point is a first value, and the mask value of the unextracted pixel point is a second value. It should be noted that the high-frequency information threshold can be set according to experience or randomly generated by the computer device, and this application does not limit this.

[0068] Specifically, if the absolute residual value of any pixel in the target image is greater than the high-frequency information threshold, the computer device can extract the pixel from the target image, and then set the mask value of the pixel in the high-frequency information position mask to the first value; if the absolute residual value of the pixel is less than or equal to the high-frequency information threshold, the computer device can set the mask value of the pixel in the high-frequency information position mask to the second value. Based on this, the calculation method of the high-frequency information position mask can be expressed by the following formula 1.2:

[0069] mask=res>thresh? a:b Formula 1.2

[0070] Among them, the high-frequency information position mask mask is a binary matrix, the size of which is consistent with the target image, that is, the high-frequency information position mask includes the mask value of each pixel; and thresh is used to represent the high-frequency information threshold, and a is used for the first value and b is used for the second value. It should be understood that the first value and the second value can be set according to actual needs or empirical values, for example, the first value can be set to 1 and the second value can be set to 0; in this case, when the absolute residual value of any pixel point is greater than the high-frequency information threshold, the position corresponding to the any pixel point in the high-frequency information position mask is marked as 1, that is, the mask value of the any pixel point in the high-frequency information position mask is 1, and when the absolute residual value of any pixel point is less than or equal to the high-frequency information threshold, the position corresponding to the any pixel point in the high-frequency information position mask is marked as 0, that is, the mask value of the any pixel point in the high-frequency information position mask is 0. It is worth noting that the present application does not limit the specific value of the first numerical value, such as the first numerical value can also be 0.9 or 0.8, etc.; and the present application does not limit the specific value of the second numerical value, such as the second numerical value can also be 0.1 or 0.2, etc.

[0071] It should be noted that the computer device can extract the structural information of the target image by calling the structural information retention module in the image processing model to obtain the structural information extraction result. Figure 5 As shown, after obtaining the target image and the reference image, the computer device can input the target image and the reference image into the structure information retention module to extract the structure information. It can be understood that the goal of the structure information retention module is to extract the structure information (i.e. high-frequency information such as edge information and detail information) in the target image; when the image processing task is an image super-resolution task, the structure information is the information generated by the super-resolution model, and retaining it utilizes the generation capability of the super-resolution model. It should be noted that Figure 5 The specific process of image processing is only exemplified and is not limited in this application; for example, the structure information retention module may also include an intensity control module, that is, the computer device may obtain an intensity-controlled image through the structure information retention module; for another example, the computer device may not make a judgment on the specified area, but directly output the intensity-controlled image, and so on.

[0072] S403, optimizing the high-frequency information of the reference image according to the intensity control direction of the target image and the absolute residual value of each pixel point to obtain an optimized reference image.

[0073] Among them, when the intensity control direction is the direction of optimizing the image quality effect, the image quality effect of the high-frequency information in the optimized reference image is better than the image quality effect of the target image; when the intensity control direction is the direction of weakening the image quality effect, the image quality effect of the high-frequency information in the optimized reference image is weaker than the image quality effect of the target image.

[0074] Specifically, the computer device may first determine the intensity control parameter according to the intensity control direction of the target image; wherein, when the intensity control direction is the direction of optimizing the image quality effect, the intensity control parameter is greater than the reference value, and when the intensity control direction is the direction of weakening the image quality effect, the intensity control parameter is less than the reference value. It should be noted that the reference value refers to: a value that makes the image quality effect of the high-frequency information in the optimized reference image equal to the image quality effect of the target image, and the above-mentioned intensity control parameter may also be referred to as a weight parameter.

[0075] For example, assuming that the image processing task is an image super-resolution task, and the above-mentioned benchmark value is 1, if the target image (i.e., the result of super-resolution) is not clear enough and it is desired to enhance the edges, that is, the intensity control direction at this time is to optimize the image quality effect, then the computer device can adjust the intensity control parameter to a parameter greater than 1; correspondingly, if it is desired to weaken the effect of super-resolution, that is, the intensity control direction at this time is to weaken the image quality effect, then the computer device can set the intensity control parameter to a parameter between (0, 1), and the closer the intensity control parameter is to 1, the closer the effect is to the target image, and the closer it is to 0, the closer the effect is to the reference image.

[0076] Further, the computer device may use the intensity control parameter to adjust the absolute residual value of each pixel point to obtain the adjusted absolute residual value of each pixel point; specifically, the computer device may multiply the intensity control parameter and the absolute residual value of each pixel point to adjust the absolute residual value of each pixel point to obtain the adjusted absolute residual value of each pixel point. It should be understood that when the absolute residual value of each pixel point can constitute absolute residual information, the computer device may use the intensity control parameter to adjust the absolute residual information to adjust the absolute residual value of each pixel point to obtain the adjusted absolute residual value of each pixel point; that is, the computer device may multiply the intensity control parameter and the absolute residual information to adjust the absolute residual value of each pixel point.

[0077] After obtaining the adjusted absolute residual value of each pixel point, the computer device can superimpose the adjusted absolute residual value of each pixel point on the pixel value of each pixel point in the reference image to optimize the high-frequency information of the reference image and obtain the optimized reference image. Based on this, the computer device can use formula 1.3 to calculate the pixel value of each pixel point in the optimized reference image, and then obtain the optimized reference image:

[0078] I up-opt =I up +weight*res Formula 1.3

[0079] Among them, I up-optUsed to represent the optimized reference image, that is, I up-opt Including the pixel value of each pixel in the optimized reference image; I up Used to represent the reference image, that is, I up Includes the pixel value of each pixel in the reference image; weight is used to represent the intensity control parameter.

[0080] It should be noted that when the computer device superimposes the adjusted absolute residual value of each pixel point on the pixel value of each pixel point in the reference image to optimize the high-frequency information of the reference image and obtain the optimized reference image, the computer device may superimpose the adjusted absolute residual value of each pixel point on the pixel value of each pixel point in the reference image to optimize the high-frequency information of the reference image and obtain the intermediate image; and clip each pixel value in the intermediate image according to the preset pixel value interval range to obtain the optimized reference image. In this case, if any pixel value in the intermediate image is greater than the maximum value in the preset pixel value interval range, the pixel value is clipped to the maximum value in the preset pixel value interval range, that is, the pixel value is set to the maximum value in the preset pixel value interval range; if the pixel value is less than the minimum value in the preset pixel value interval range, the pixel value is clipped to the minimum value in the preset pixel value interval range, that is, the pixel value is set to the minimum value in the preset pixel value interval range.

[0081] For example, assuming that the above-mentioned preset pixel value interval range is [0, 1], for any pixel point in the intermediate image, if the pixel value of any pixel point in the intermediate image is greater than 1, then the computer device can crop the pixel value of any pixel point in the intermediate image to 1, that is, the pixel value of any pixel point in the optimized reference image is 1; if the pixel value of any pixel point in the intermediate image is any value in the preset pixel value interval range, then the computer device can use the pixel value of any pixel point in the intermediate image as the pixel value of any pixel point in the optimized reference image; if the pixel value of any pixel point in the intermediate image is less than 0, then the computer device can crop the pixel value of any pixel point in the intermediate image to 0, that is, the pixel value of any pixel point in the optimized reference image is 0.

[0082] S404, extracting high-frequency information from the optimized reference image and extracting low-frequency information from the target image using the high-frequency information position mask.

[0083] The low-frequency information of the target image refers to the image information in the target image except the high-frequency information. In other words, the low-frequency information refers to the image information in the target image except the structural information.

[0084] It should be noted that the computer device can perform blur processing on the high-frequency information position mask to obtain a blurred high-frequency information position mask; and use the blurred high-frequency information position mask to extract high-frequency information from the optimized reference image and low-frequency information from the target image. Accordingly, the computer device can use Gaussian blur method, median filtering method, mean filtering method, bilateral filtering method, etc. to blur the high-frequency information position mask. The present application does not limit the blur processing method of the high-frequency information position mask. It can be understood that the purpose of the computer device blurring the high-frequency information position mask is to fine-tune each mask value in the high-frequency information position mask, thereby weakening the mask value of the pixel point used to describe the high-frequency information in the target image in the high-frequency information position mask.

[0085] It is worth noting that the computer device can extract high-frequency information only from the optimized reference image, and extract low-frequency information only from the target image; it can also extract high-frequency information and low-frequency information from the optimized reference image, and extract high-frequency information and low-frequency information from the target image, respectively, and the present application does not limit this. It should be noted that when the computer device extracts high-frequency information and low-frequency information from the optimized reference image, and extracts high-frequency information and low-frequency information from the target image, for any pixel, the computer device can extract the first pixel value of any pixel from the optimized reference image according to the mask value of the any pixel in the high-frequency information position mask, and extract the second pixel value of any pixel from the target image according to the difference between the preset value and the mask value of the any pixel in the high-frequency information position mask; or, the computer device can extract the first pixel value of any pixel from the optimized reference image according to the mask value of the any pixel in the high-frequency information position mask after blurring, and extract the second pixel value of any pixel from the target image according to the difference between the preset value and the mask value of the any pixel in the high-frequency information position mask after blurring, and so on; wherein the preset value can be set according to actual needs or empirical values, for example, the preset value can be set to 1.

[0086] S405, fusing the extracted high-frequency information and the extracted low-frequency information to obtain an intensity-controlled image.

[0087] In one embodiment, the computer device may use the extracted high-frequency information as the high-frequency information in the intensity-controlled image, that is, the pixel value of the pixel point used to describe the high-frequency information in the intensity-controlled image refers to: the corresponding pixel value in the optimized reference image; and the computer device may use the extracted low-frequency information as the low-frequency information in the intensity-controlled image, that is, the pixel value of the pixel point used to describe the low-frequency information in the intensity-controlled image refers to: the corresponding pixel value in the target image. In this case, the high-frequency information in the intensity-controlled image only includes the high-frequency information in the optimized reference image, and the low-frequency information in the intensity-controlled image only includes the low-frequency information in the target image.

[0088] In another embodiment, the computer device may extract high-frequency information from the optimized reference image and the target image according to the high-frequency information position mask, and extract low-frequency information from the optimized reference image and the target image according to the high-frequency information position mask, and then fuse the extracted high-frequency information and the extracted low-frequency information to obtain an intensity-controlled image; or, the computer device may extract high-frequency information from the optimized reference image and the target image according to the high-frequency information position mask after fuzzy processing, and extract low-frequency information from the optimized reference image and the target image according to the high-frequency information position mask after fuzzy processing, and then fuse the extracted high-frequency information and the extracted low-frequency information to obtain an intensity-controlled image, etc. In this case, the computer device may extract the first pixel value of any pixel point in the optimized reference image, and the second pixel value of any pixel point in the target image, and then fuse the first pixel value and the second pixel value to obtain the pixel value of any pixel point in the intensity-controlled image. Based on this, taking the example of a computer device extracting the corresponding high-frequency information and low-frequency information through a high-frequency information position mask after blurring, the computer device can calculate the pixel value of each pixel in the intensity-controlled image according to formula 1.4 to obtain the intensity-controlled image:

[0089] I sc =I mask-blur *I up-opt +(1-I mask-blur )*I sr Formula 1.4

[0090] Among them, I sc Used to represent the image after intensity control, that is, I sc Including the pixel value of each pixel in the image after intensity control; I mask-blurUsed to represent the high-frequency information position mask after blurring, that is, I mask-blur Includes the mask value of each pixel in the high-frequency information position mask after blurring.

[0091] It should be noted that the above intensity-controlled image can be obtained by calling the intensity control module in the image processing model; for example, Figure 5 As shown, the computer device can input the target image and the reference image into the structure information retention module and the intensity control module in sequence, thereby obtaining an intensity-controlled image. It is worth noting that the intensity-controlled image can also be called a full-image intensity-controlled image; when the image processing task is an image super-resolution task, the intensity-controlled image can also be called a global super-resolution intensity control result, and so on.

[0092] It can be understood that when the intensity control direction is to optimize the image quality effect, the image quality effect of the image after intensity control is better than the image quality effect of the target image; when the intensity control direction is to weaken the image quality effect, the image quality effect of the image after intensity control is weaker than the image quality effect of the target image. Figure 6a As shown in FIG. 1 , assuming that the image processing task is an image super-resolution task, and the image quality of the target image is poor, the intensity control direction at this time can be the direction of optimizing the image quality effect. Then, the image after intensity control has an enhanced effect on the edge structure and improves the clarity of the image. For example, Figure 6b As shown, assuming that the image processing task is an image super-resolution task, and the image quality effect of the target image is too distorted, the intensity control direction at this time can be the direction of weakening the image quality effect. Then, the texture distortion intensity on the clothes in the image after intensity control is weakened, thereby improving the naturalness of the picture.

[0093] Furthermore, if the computer device only needs to perform intensity control on the target area (i.e., the specified area), after obtaining the intensity-controlled image, the image block of the reference area (i.e., the specified position) can be cropped from the intensity-controlled image and then replace the specified area of ​​the target image.

[0094] Specifically, after obtaining the image after intensity control, the computer device can determine the target area in the target image that needs to be intensity controlled; and determine the reference area corresponding to the target area from the image after intensity control according to the position of the target area in the target image. Then, the computer device can replace each pixel value in the target area in the target image with each pixel value in the reference area, and obtain the area control image corresponding to the target image. It should be noted that the position of the reference area in the image after intensity control is the same as the position of the target area in the target image, that is, the pixel points included in the target area are the same as the pixel points included in the reference area; based on this, in the same coordinate system, the position coordinates of the reference area are the same as the position coordinates of the target area.

[0095] In this case, since inconsistent effects may occur at the edge, making it look like a frame visually, in order to achieve a good transition, the computer device can expand the pixel values ​​of Q pixels around the specified area, and fuse the pixel values ​​of Q pixels at the corresponding positions of the intensity-controlled image and the target image, and then obtain the replaced area control image.

[0096] Specifically, the computer device can determine the extended area from the regional control image according to the regional margin of the target area, and the extended area refers to the area where Q pixels that need to be transitionally fused are located, and Q is a positive integer; for the i-th pixel in the extended area, the pixel value of the i-th pixel in the target image and the pixel value of the i-th pixel in the image after intensity control are fused to obtain the fused pixel value of the i-th pixel, i∈[1,Q]; then, the computer device can replace the pixel value of each pixel in the extended area with the corresponding fused pixel value in the regional control image to obtain the replaced regional control image. It should be noted that the fusion processing here can refer to weighted summation processing; wherein, if the pixel value of the i-th pixel in the target image and the pixel value of the i-th pixel in the image after intensity control are weighted summation processing according to the same weight, it means that the pixel value of the i-th pixel in the target image and the pixel value of the i-th pixel in the image after intensity control are added and averaged.

[0097] It is understandable that after obtaining the regional control image, the computer device can determine the extended area from the regional control image according to the regional margin of the target area, and replace the pixel value of each pixel in the extended area to obtain the replaced regional control image; it can also determine the target area in the target image, and determine the extended area from the target image according to the regional margin of the target area, and perform corresponding replacement processing on the pixel value of each pixel in the target area and the pixel value of each pixel in the extended area to obtain the replaced regional control image, and the present application does not limit this.

[0098] For example, Figure 7 As shown, it is assumed that region 7011 is the target region in the target image that needs to be intensity controlled, region 7012 (i.e., the shadow region in the target image) is the extended region in the target image, and the computer device can cut out the region at the same position on the image after intensity control (i.e., the image after full image intensity control), that is, cut out region 7021 and region 7022 (i.e., the shadow region in the image after intensity control). Since the pixel values ​​in the extended region (i.e., the marginal position) need to be transitionally fused, the pixel values ​​in region 7012 and the pixel values ​​in region 7022 are fused and used as the final pixel values ​​of the position, that is, the pixel values ​​in region 7032 (i.e., the shadow region in the replaced regional control image) in the replaced regional control image are obtained; and the computer device can replace the pixel values ​​in region 7011 with the pixel values ​​in region 7021, that is, the pixel values ​​in region 7031 in the final result (i.e., the replaced regional control image) are the pixel values ​​in region 7021.

[0099] For example, Figure 8 As shown, assuming that the image processing task is an image super-resolution task, and the image quality effect of the obtained target image is too distorted, the computer device can control the intensity of the image quality effect of the specified area (i.e., the target area) in the target image and the extended area involved in the specified area to obtain a replaced area control image; wherein the area indicated by the dotted box in the replaced area control image represents the area corresponding to the specified area (i.e., the target area) in the replaced area control image, that is, the computer device can only process the face, so that the intensity of the face is weakened, so that the replaced area control image is relatively natural.

[0100] It should be noted that if no area is specified, the computer device can directly output the intensity-controlled image after obtaining the intensity-controlled image; otherwise, the computer device can further obtain the replaced area-controlled image through the intensity-controlled image and output the replaced area-controlled image.

[0101] After obtaining the target image and the reference image, the embodiment of the present application can extract the structural information of the target image to obtain the structural information extraction result, so that the structural information extraction result includes: a high-frequency information position mask constructed based on the absolute residual value of each pixel in the target image, and then optimizing the high-frequency information of the reference image by the absolute residual value of each pixel to obtain the optimized reference image; then using the high-frequency information position mask to extract the high-frequency information from the optimized reference image and the low-frequency information from the target image to obtain the image after intensity control. It can be seen that the embodiment of the present application can extract structural information to perform intensity control of the image quality effect, thereby obtaining the image after intensity control; wherein, when the intensity control direction is the direction of optimizing the image quality effect, the image quality effect of the image after intensity control is better than the image quality effect of the target image, and when the intensity control direction is the direction of weakening the image quality effect, the image quality effect of the image after intensity control is weaker than the image quality effect of the target image; that is, the embodiment of the present application can perform intensity control of the image quality effect based on the target image and in conjunction with the reference image to achieve the goal of improving the image quality effect or weakening the image quality effect. In addition, if the intensity of the image quality effect is only controlled for the target area (i.e., the specified area) of the image, the embodiment of the present application can crop a reference area from the image after intensity control to replace the target area in the target image, obtain a regional control image, and then obtain a replaced regional control image.

[0102] Based on the description of the above-mentioned related embodiments of the image processing method, the present application embodiment also proposes an image processing device, which can be a computer program (including program code) running in a computer device. The image processing device can execute Figure 2 or Figure 4 The image processing method shown; see Fig. 9 , the image processing device can run the following units:

[0103] A processing unit 901 is used to perform image processing on an initial image to be processed according to an image processing task to obtain a target image, and determine a reference image whose resolution matches the resolution of the target image based on the initial image;

[0104] The structure extraction unit 902 is used to extract the structure information of the target image to obtain a structure information extraction result; wherein the structure information refers to information related to the image quality effect, and the structure information extraction result is used to indicate: pixel points used to describe the structure information in the target image;

[0105] The intensity control unit 903 is used to control the intensity of the image quality effect of the target image according to the structural information extraction result and the reference image.

[0106] In one implementation, the structural information of the target image includes high-frequency information of the target image, and the structural information extraction result includes a high-frequency information position mask; when the structural extraction unit 902 performs structural information extraction processing on the target image to obtain the structural information extraction result, it can be specifically used to:

[0107] Determine an absolute residual value of each pixel point according to a pixel value of each pixel point in the target image and a pixel value of each pixel point in the reference image;

[0108] Extracting pixel points for describing the high-frequency information from the target image according to the magnitude relationship between the absolute residual value of each pixel point and the high-frequency information threshold; wherein the absolute residual value of each pixel point extracted is greater than the high-frequency information threshold;

[0109] A high-frequency information position mask is constructed based on the pixel extraction result, and the high-frequency information position mask includes mask values ​​of the respective pixel points; wherein the mask value of the extracted pixel points is a first value, and the mask value of the unextracted pixel points is a second value.

[0110] In another implementation manner, when the structure extraction unit 902 determines the absolute residual value of each pixel point according to the pixel value of each pixel point in the target image and the pixel value of each pixel point in the reference image, it can be specifically used to:

[0111] Performing structural information weakening processing on the reference image to obtain a weakened reference image;

[0112] According to the pixel value of each pixel in the target image and the pixel value of each pixel in the reference image after the weakening process, the absolute residual value of each pixel is determined.

[0113] In another implementation manner, when the structure extraction unit 902 performs a weakening process on the reference image to obtain the weakened reference image, it can be specifically used to:

[0114] Normalizing each pixel value in the reference image to obtain a normalized reference image;

[0115] The normalized reference image is blurred to weaken the structural information of the reference image, thereby obtaining a weakened reference image.

[0116] In another embodiment, the structural information extraction result includes: a high-frequency information position mask constructed based on the absolute residual value of each pixel point in the target image; when the intensity control unit 903 controls the intensity of the image quality effect of the target image according to the structural information extraction result and the reference image, it can be specifically used to:

[0117] According to the intensity control direction of the target image and the absolute residual value of each pixel point, the high-frequency information of the reference image is optimized to obtain an optimized reference image; wherein, when the intensity control direction is a direction for optimizing image quality effect, the image quality effect of the high-frequency information in the optimized reference image is better than the image quality effect of the target image, and when the intensity control direction is a direction for weakening image quality effect, the image quality effect of the high-frequency information in the optimized reference image is weaker than the image quality effect of the target image;

[0118] Extracting high-frequency information from the optimized reference image and extracting low-frequency information from the target image using the high-frequency information position mask; wherein the low-frequency information of the target image refers to image information in the target image other than the high-frequency information;

[0119] The extracted high-frequency information and the extracted low-frequency information are fused to obtain an intensity-controlled image.

[0120] In another implementation manner, the intensity control unit 903 optimizes the high-frequency information of the reference image according to the intensity control direction of the target image and the absolute residual value of each pixel point to obtain the optimized reference image, which can be specifically used to:

[0121] Determining an intensity control parameter according to the intensity control direction of the target image; wherein, when the intensity control direction is a direction for optimizing image quality effects, the intensity control parameter is greater than a reference value, and when the intensity control direction is a direction for weakening image quality effects, the intensity control parameter is less than the reference value;

[0122] The absolute residual value of each pixel point is adjusted by using the intensity control parameter to obtain the adjusted absolute residual value of each pixel point;

[0123] The adjusted absolute residual value of each pixel point is superimposed on the pixel value of each pixel point in the reference image to optimize the high-frequency information of the reference image and obtain an optimized reference image.

[0124] In another implementation manner, when the intensity control unit 903 superimposes the adjusted absolute residual value of each pixel point on the pixel value of each pixel point in the reference image to optimize the high-frequency information of the reference image and obtain the optimized reference image, it can be specifically used to:

[0125] superimposing the adjusted absolute residual value of each pixel point on the pixel value of each pixel point in the reference image to optimize the high-frequency information of the reference image to obtain an intermediate image;

[0126] According to a preset pixel value interval range, each pixel value in the intermediate image is cropped to obtain an optimized reference image.

[0127] In another embodiment, when the intensity control unit 903 uses the high-frequency information position mask to extract the high-frequency information from the optimized reference image and extracts the low-frequency information from the target image, it can be specifically used to:

[0128] Performing fuzzy processing on the high-frequency information position mask to obtain a fuzzy processed high-frequency information position mask;

[0129] The high-frequency information position mask after the fuzzy processing is used to extract high-frequency information from the optimized reference image, and low-frequency information from the target image.

[0130] In another implementation, the intensity control unit 903 may also be used to:

[0131] After obtaining the intensity-controlled image, determining a target area in the target image that needs to be intensity-controlled;

[0132] Determining a reference area corresponding to the target area from the intensity-controlled image according to the position of the target area in the target image;

[0133] Each pixel value in the target area of ​​the target image is replaced by each pixel value in the reference area to obtain a region control image corresponding to the target image.

[0134] In another implementation, the intensity control unit 903 may also be used to:

[0135] Determine an extended area from the area control image according to the area margin of the target area, the extended area refers to the area where Q pixels need to be transitionally fused, where Q is a positive integer;

[0136] For the i-th pixel point in the extended area, a pixel value of the i-th pixel point in the target image and a pixel value of the i-th pixel point in the image after intensity control are fused to obtain a fused pixel value of the i-th pixel point, i∈[1,Q];

[0137] In the area control image, the pixel value of each pixel point in the extended area is replaced with the corresponding fused pixel value to obtain a replaced area control image.

[0138] In another implementation manner, when the processing unit 901 determines, based on the initial image, a reference image having a resolution matching the resolution of the target image, it may be specifically configured to:

[0139] Determining the resolution of the initial image and the resolution of the target image;

[0140] If the resolution of the initial image is equal to the resolution of the target image, determining the initial image as a reference image;

[0141] If the resolution of the initial image is greater than or less than the resolution of the target image, adjusting the resolution of the initial image according to the resolution of the target image to obtain a reference image;

[0142] Wherein, when the image processing task includes an image super-resolution task, the resolution of the target image is greater than the resolution of the initial image.

[0143] According to one embodiment of the present application, Figure 2 or Figure 4 Each step involved in the method shown can be Fig. 9 The image processing device shown in FIG. 1 is executed by each unit in the image processing device shown in FIG. Figure 2 The step S201 shown in FIG. 1 can be performed by Fig. 9 The processing unit 901 shown in FIG. 1 is executed, and step S202 can be performed by Fig. 9 The structure extraction unit 902 shown in FIG. 1 is executed, and step S203 can be performed by Fig. 9 The intensity control unit 903 shown in FIG. Figure 4 The step S401 shown in FIG. 4 can be performed by Fig. 9 The processing unit 901 shown in FIG. 1 is executed, and step S402 may be performed by Fig. 9 The structure extraction unit 902 shown in FIG. 1 is executed, and steps S403-S405 can be performed by Fig. 9 The intensity control unit 903 shown performs, and so on.

[0144] According to another embodiment of the present application, Fig. 9The various units in the image processing device shown can be separately or completely combined into one or several other units to constitute, or one (some) of the units can also be split into multiple smaller units in function to constitute, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In practical applications, the function of one unit can also be implemented by multiple units, or the functions of multiple units can be implemented by one unit. In other embodiments of the present application, the image processing device may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units.

[0145] According to another embodiment of the present application, the program can be executed by running on a general computing device such as a computer including a central processing unit (CPU), a random access memory medium (RAM), a read-only memory medium (ROM), and other processing elements and storage elements. Figure 2 or Figure 4 A computer program (including program code) for each step involved in the corresponding method shown in Fig. 9 The image processing apparatus shown in and the image processing method of the embodiment of the present application are implemented. The computer program can be recorded on, for example, a computer storage medium, and loaded into the above-mentioned computing device through the computer storage medium, and run therein.

[0146] After obtaining the target image and the reference image, the embodiment of the present application can first perform structural information extraction processing on the target image to obtain the structural information extraction result, so that the structural information extraction result can be used to indicate: the pixel points used to describe the structural information in the target image, and the structural information refers to the information related to the image quality effect of the image, so that the structural information extraction result can be used to indicate: the pixel points used to describe the information related to the image quality effect in the target image; then, according to the structural information extraction result and the reference image, the intensity of the image quality effect of the target image is controlled, so that not only the image quality effect of the target image can be optimized, but also the image quality effect of the target image can be weakened, so that the image quality effect of the image is more moderate, thereby meeting the visual effect requirements and effectively improving the visual effect of the final image.

[0147] Based on the description of the above method embodiment and device embodiment, the present application embodiment also provides a computer device. Fig.10 The computer device at least includes a processor 1001, an input interface 1002, an output interface 1003, and a computer storage medium 1004. The processor 1001, the input interface 1002, the output interface 1003, and the computer storage medium 1004 in the computer device may be connected via a bus or other means.

[0148] The computer storage medium 1004 can be stored in the memory of the computer device, and the computer storage medium 1004 is used to store a computer program, and the computer program includes program instructions, and the processor 1001 is used to execute the program instructions stored in the computer storage medium 1004. The processor 1001 (or CPU (Central Processing Unit)) is the computing core and control core of the computer device, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding function; In one embodiment, the processor 1001 described in the embodiment of the present application can be used to perform a series of image processing, specifically including: performing image processing on the initial image to be processed according to the image processing task to obtain a target image, and determining a reference image whose resolution matches the resolution of the target image based on the initial image; extracting the structure information of the target image to obtain a structure information extraction result; wherein the structure information refers to information related to the image quality effect of the image, and the structure information extraction result is used to indicate: the pixel points used to describe the structure information in the target image; according to the structure information extraction result and the reference image, the intensity of the image quality effect of the target image is controlled.

[0149] The embodiment of the present application also provides a computer storage medium (Memory), which is a memory device in a computer device for storing programs and data. It is understandable that the computer storage medium here can include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer storage medium provides a storage space, which stores the operating system of the computer device. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space. These instructions can be one or more computer programs (including program codes). It should be noted that the computer storage medium here can be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as at least one disk storage; optionally, it can also be at least one computer storage medium located away from the aforementioned processor. In one embodiment, the processor can load and execute one or more instructions stored in the computer storage medium to implement the above-mentioned related Figure 2 or Figure 4 The individual method steps in the embodiment of the image processing method are shown.

[0150] After obtaining the target image and the reference image, the embodiment of the present application can first perform structural information extraction processing on the target image to obtain the structural information extraction result, so that the structural information extraction result can be used to indicate: the pixel points used to describe the structural information in the target image, and the structural information refers to the information related to the image quality effect of the image, so that the structural information extraction result can be used to indicate: the pixel points used to describe the information related to the image quality effect in the target image; then, according to the structural information extraction result and the reference image, the intensity of the image quality effect of the target image is controlled, so that not only the image quality effect of the target image can be optimized, but also the image quality effect of the target image can be weakened, so that the image quality effect of the image is more moderate, thereby meeting the visual effect requirements and effectively improving the visual effect of the final image.

[0151] It should be noted that according to one aspect of the present application, a computer program product or a computer program is also provided, the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer storage medium. The processor of the computer device reads the computer instructions from the computer storage medium, and the processor executes the computer instructions, so that the computer device performs the above Figure 2 or Figure 4 The method provided in various optional manners of the image processing method embodiment shown.

[0152] Furthermore, it should be understood that what is disclosed above is only a preferred embodiment of the present application, and certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. An image processing method, It is characterized in that include: Performing image processing on an initial image to be processed according to an image processing task to obtain a target image, and determining a reference image having a resolution matching a resolution of the target image based on the initial image; Performing structural information extraction processing on the target image to obtain a structural information extraction result; wherein the structural information refers to information related to the image quality effect, and the structural information extraction result is used to indicate: pixel points used to describe the structural information in the target image; the structural information of the target image includes high-frequency information of the target image; the structural information extraction result includes: a high-frequency information position mask constructed based on the absolute residual value of each pixel point in the target image; The intensity of the image quality effect of the target image is controlled according to the structural information extraction result and the reference image.

2. The method according to claim 1, It is characterized in that The step of extracting the structure information of the target image to obtain a structure information extraction result includes: Determine an absolute residual value of each pixel point according to a pixel value of each pixel point in the target image and a pixel value of each pixel point in the reference image; Extracting pixel points for describing the high-frequency information from the target image according to the magnitude relationship between the absolute residual value of each pixel point and the high-frequency information threshold; wherein the absolute residual value of each pixel point extracted is greater than the high-frequency information threshold; A high-frequency information position mask is constructed based on the pixel extraction result, and the high-frequency information position mask includes mask values ​​of the respective pixel points; wherein the mask value of the extracted pixel points is a first value, and the mask value of the unextracted pixel points is a second value.

3. The method according to claim 2, It is characterized in that Determining the absolute residual value of each pixel point according to the pixel value of each pixel point in the target image and the pixel value of each pixel point in the reference image includes: Performing structural information weakening processing on the reference image to obtain a weakened reference image; According to the pixel value of each pixel in the target image and the pixel value of each pixel in the reference image after the weakening process, the absolute residual value of each pixel is determined.

4. The method according to claim 3, It is characterized in that The step of performing structural information weakening processing on the reference image to obtain a weakened reference image includes: Normalizing each pixel value in the reference image to obtain a normalized reference image; The normalized reference image is blurred to weaken the structural information of the reference image, thereby obtaining a weakened reference image.

5. The method according to any one of claims 1 to 4, It is characterized in that The controlling the intensity of the image quality effect of the target image according to the structural information extraction result and the reference image includes: According to the intensity control direction of the target image and the absolute residual value of each pixel point, the high-frequency information of the reference image is optimized to obtain an optimized reference image; wherein, when the intensity control direction is a direction for optimizing image quality effect, the image quality effect of the high-frequency information in the optimized reference image is better than the image quality effect of the target image, and when the intensity control direction is a direction for weakening image quality effect, the image quality effect of the high-frequency information in the optimized reference image is weaker than the image quality effect of the target image; Extracting high-frequency information from the optimized reference image and extracting low-frequency information from the target image using the high-frequency information position mask; wherein the low-frequency information of the target image refers to image information in the target image other than the high-frequency information; The extracted high-frequency information and the extracted low-frequency information are fused to obtain an intensity-controlled image.

6. The method according to claim 5, It is characterized in that The step of optimizing the high-frequency information of the reference image according to the intensity control direction of the target image and the absolute residual value of each pixel point to obtain an optimized reference image includes: Determining an intensity control parameter according to the intensity control direction of the target image; wherein, when the intensity control direction is a direction for optimizing image quality effects, the intensity control parameter is greater than a reference value, and when the intensity control direction is a direction for weakening image quality effects, the intensity control parameter is less than the reference value; The absolute residual value of each pixel point is adjusted by using the intensity control parameter to obtain the adjusted absolute residual value of each pixel point; The adjusted absolute residual value of each pixel point is superimposed on the pixel value of each pixel point in the reference image to optimize the high-frequency information of the reference image and obtain an optimized reference image.

7. The method according to claim 6, It is characterized in that The step of superimposing the adjusted absolute residual value of each pixel point on the pixel value of each pixel point in the reference image to optimize the high frequency information of the reference image to obtain an optimized reference image includes: superimposing the adjusted absolute residual value of each pixel point on the pixel value of each pixel point in the reference image to optimize the high-frequency information of the reference image to obtain an intermediate image; According to a preset pixel value interval range, each pixel value in the intermediate image is cropped to obtain an optimized reference image.

8. The method according to claim 5, It is characterized in that The step of extracting high-frequency information from the optimized reference image using the high-frequency information position mask, and extracting low-frequency information from the target image, comprises: Performing fuzzy processing on the high-frequency information position mask to obtain a fuzzy processed high-frequency information position mask; The high-frequency information position mask after the fuzzy processing is used to extract high-frequency information from the optimized reference image, and low-frequency information from the target image.

9. The method according to any one of claims 1 to 4, It is characterized in that The method further comprises: After obtaining the intensity-controlled image, determining a target area in the target image that needs to be intensity-controlled; Determining a reference area corresponding to the target area from the intensity-controlled image according to the position of the target area in the target image; Each pixel value in the target area of ​​the target image is replaced by each pixel value in the reference area to obtain a region control image corresponding to the target image.

10. The method according to claim 9, It is characterized in that The method further comprises: Determine an extended area from the area control image according to the area margin of the target area, the extended area refers to the area where Q pixels need to be transitionally fused, where Q is a positive integer; For the i-th pixel point in the extended area, a pixel value of the i-th pixel point in the target image and a pixel value of the i-th pixel point in the image after intensity control are fused to obtain a fused pixel value of the i-th pixel point, i∈[1,Q]; In the area control image, the pixel value of each pixel point in the extended area is replaced with the corresponding fused pixel value to obtain a replaced area control image.

11. The method according to any one of claims 1 to 4, It is characterized in that The step of determining, based on the initial image, a reference image having a resolution matching that of the target image comprises: Determining the resolution of the initial image and the resolution of the target image; If the resolution of the initial image is equal to the resolution of the target image, determining the initial image as a reference image; If the resolution of the initial image is greater than or less than the resolution of the target image, adjusting the resolution of the initial image according to the resolution of the target image to obtain a reference image; Wherein, when the image processing task includes an image super-resolution task, the resolution of the target image is greater than the resolution of the initial image.

12. An image processing device, It is characterized in that include: A processing unit, configured to perform image processing on an initial image to be processed according to an image processing task to obtain a target image, and determine a reference image having a resolution matching a resolution of the target image based on the initial image; A structure extraction unit is used to extract structure information of the target image to obtain a structure information extraction result; wherein the structure information refers to information related to the image quality effect, and the structure information extraction result is used to indicate: pixel points used to describe the structure information in the target image; the structure information of the target image includes high-frequency information of the target image; the structure information extraction result includes: a high-frequency information position mask constructed based on the absolute residual value of each pixel point in the target image; An intensity control unit is used to control the intensity of the image quality effect of the target image according to the structural information extraction result and the reference image.

13. A computer device, It is characterized in that The method comprises a processor and a memory, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, the method according to any one of claims 1 to 11 is implemented.

14. A computer storage medium, It is characterized in that The computer storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

15. A computer program product, It is characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images