Image optimization methods and apparatus, storage media and electronic devices

Abstract

The present disclosure relates to the field of image processing technology, and specifically to an image optimization method and device, a computer-readable storage medium, and an electronic device, wherein the method comprises: obtaining a current image and a reference image, and performing spatial filtering on the current image to obtain a first filtered image under a first filtering parameter and a second filtered image under a second filtering parameter, wherein the second filtering parameter is greater than the first filtering parameter; performing spatial filtering on the reference image to obtain a reference filtered image of the reference image under the first filtering parameter; performing temporal filtering on the first filtered image according to the reference filtered image to obtain a guide image and a first image to be output; performing contrast enhancement on the current image according to the second filtered image and the guide image to obtain a second image to be output; obtaining an optimized image according to the first image to be output and the second image to be output and outputting it. The technical solution of the embodiment of the present disclosure reduces the amount of calculation and power consumption during image optimization, and improves the image optimization accuracy.

Description

Background Art

[0002] With the development of mobile technology, camera technology has also developed rapidly. However, in the process of video transmission, there are usually various noises mixed in, and the contrast in some videos is weak, which reduces the visual effect of the video. Therefore, it is necessary to optimize the video and image by noise reduction and contrast enhancement.

[0003] When the image noise reduction and contrast enhancement methods in the related art are executed, the image noise reduction and contrast enhancement are performed separately, which results in a large amount of calculation and high power consumption.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The purpose of the present disclosure is to provide an image optimization method, an image optimization device, a computer-readable medium and an electronic device, thereby reducing the amount of calculation and power consumption during image optimization at least to a certain extent.

[0006] According to a first aspect of the present disclosure, there is provided an image optimization method, comprising:

[0007] Acquire a current image and a reference image, and perform spatial domain filtering on the current image to obtain a first filtered image under a first filtering parameter and a second filtered image under a second filtering parameter; wherein the second filtering parameter is greater than the first filtering parameter;

[0008] Performing spatial domain filtering on the reference image to obtain a reference filtered image of the reference image under a first filtering parameter;

[0009] Performing time domain filtering on the first filtered image according to the reference filtered image to obtain a guide image and a first image to be output;

[0010] Performing contrast enhancement on the current image according to the second filtered image and the guide image to obtain a second image to be output;

[0011] An optimized image is obtained according to the first image to be output and the second image to be output and is output.

[0012] According to a second aspect of the present disclosure, there is provided an image optimization device, comprising:

[0013] A first filtering module, used to obtain a current image and a reference image, and perform spatial domain filtering on the current image to obtain a first filtered image under a first filtering parameter and a second filtered image under a second filtering parameter; wherein the second filtering parameter is greater than the first filtering parameter;

[0014] A second filtering module, configured to perform spatial domain filtering on the reference image to obtain a reference filtered image of the reference image under a first filtering parameter;

[0015] A time domain filtering module, configured to perform time domain filtering on the first filtered image according to the reference filtered image to obtain a guide image and a first image to be output;

[0016] an image enhancement module, configured to perform contrast enhancement on the current image according to the second filtered image and the guide image to obtain a second image to be output;

[0017] The image optimization module obtains and outputs an optimized image according to the first image to be output and the second image to be output.

[0018] According to a third aspect of the present disclosure, a computer-readable medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the above method is implemented.

[0019] According to a fourth aspect of the present disclosure, there is provided an electronic device, characterized in that it includes:

[0020] Processor; and

[0021] The memory is used to store one or more programs. When the one or more programs are executed by one or more processors, the one or more processors implement the above method.

[0022] An image optimization method provided by an embodiment of the present disclosure obtains a current image and a reference image, and performs spatial filtering on the current image to obtain a first filtered image under a first filtering parameter and a second filtered image under a second filtering parameter; performs spatial filtering on the reference image to obtain a reference filtered image of the reference image under the first filtering parameter; performs temporal filtering on the first filtered image according to the reference filtered image to obtain a guide image and a first image to be output; performs contrast enhancement on the current image according to the second filtered image and the guide image to obtain a second image to be output; obtains and outputs an optimized image according to the first image to be output and the second image to be output. Compared with the prior art, on the one hand, the second filtered image parameter obtained after denoising the image is directly applied to the contrast enhancement of the image, and the amount of calculation and power consumption are reduced by reusing the calculation results. On the other hand, the result of spatial filtering is used for contrast enhancement at the same time, which solves the problem of noise enhancement when detail enhancement is performed. Furthermore, the first image to be output after temporal filtering is used to enhance the result of contrast enhancement, thereby improving the optimization accuracy of the image.

[0023] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0025] Figure 1 A schematic diagram showing an exemplary system architecture to which embodiments of the present disclosure may be applied;

[0026] Figure 2 A schematic diagram showing an electronic device to which the embodiments of the present disclosure can be applied;

[0027] Figure 3 A flowchart of an image optimization method in an exemplary embodiment of the present disclosure is schematically shown;

[0028] Figure 4 A data flow diagram schematically illustrating an image optimization method in an exemplary embodiment of the present disclosure;

[0029] Figure 5 A flowchart of obtaining a first filtered image and a second filtered image in an exemplary embodiment of the present disclosure is schematically shown;

[0030] Figure 6 Schematically illustrates a data flow diagram for acquiring a first filtered image and a second filtered image in an exemplary embodiment of the present disclosure;

[0031] Figure 7 Schematically showing a flow chart of obtaining a guide image in an exemplary embodiment of the present disclosure;

[0032] Figure 8 Schematically showing a data flow diagram during time domain filtering and contrast enhancement in an exemplary embodiment of the present disclosure;

[0033] Fig. 9 The figure schematically shows the composition of the image optimization device in the exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0034] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0035] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0036] Figure 1 A schematic diagram of a system architecture of an exemplary application environment in which an image optimization method and device according to an embodiment of the present disclosure can be applied is shown.

[0037] like Figure 1 As shown, the system architecture 100 may include one or more of terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc. The terminal devices 101, 102, 103 may be various electronic devices with image processing functions, including but not limited to desktop computers, portable computers, smart phones, tablet computers, etc. It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices, networks and servers. For example, the server 105 may be a server cluster composed of multiple servers.

[0038] The image optimization method provided in the embodiment of the present disclosure is generally executed by the terminal devices 101, 102, and 103, and accordingly, the image optimization device is generally arranged in the terminal devices 101, 102, and 103. However, it is easy for those skilled in the art to understand that the image optimization method provided in the embodiment of the present disclosure can also be executed by the server 105, and accordingly, the image optimization device can also be arranged in the server 105, and this is not particularly limited in the present exemplary embodiment. For example, in an exemplary embodiment, the user can use the terminal devices 101, 102, and 103 to collect the current image and the reference image, and then upload the current image and the reference image to the server 105. After the server generates the optimized image through the image optimization method provided in the embodiment of the present disclosure, the optimized image is transmitted to the terminal devices 101, 102, and 103, etc.

[0039] An exemplary embodiment of the present disclosure provides an electronic device for implementing an image optimization method, which may be Figure 1The terminal device 101, 102, 103 or the server 105 in the embodiment of the present invention comprises at least a processor and a memory, the memory is used to store executable instructions of the processor, and the processor is configured to execute the image optimization method by executing the executable instructions.

[0040] Below Figure 2 Taking the mobile terminal 200 in FIG. 1 as an example, the structure of the electronic device is exemplarily described. It should be understood by those skilled in the art that, in addition to the components specifically used for mobile purposes, Figure 2 The structure in the figure can also be applied to fixed type devices. In other embodiments, the mobile terminal 200 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure can be implemented in hardware, software, or a combination of software and hardware. The interface connection relationship between the components is only shown schematically and does not constitute a structural limitation of the mobile terminal 200. In other embodiments, the mobile terminal 200 may also adopt the same Figure 2 Different interface connection methods, or a combination of multiple interface connection methods.

[0041] like Figure 2 As shown, the mobile terminal 200 may specifically include: a processor 210, an internal memory 221, an external memory interface 222, a Universal Serial Bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 271, a receiver 272, a microphone 273, an earphone interface 274, a sensor module 280, a display screen 290, a camera module 291, an indicator 292, a motor 293, a button 294, and a subscriber identification module (SIM) card interface 295, etc. The sensor module 280 may include a depth sensor 2801, a pressure sensor 2802, a gyroscope sensor 2803, etc.

[0042] The processor 210 may include one or more processing units, for example, the processor 210 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor and / or a neural network processor (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0043] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of the mobile terminal 200 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0044] The processor 210 is provided with a memory. The memory can store instructions for implementing six modular functions: detection instructions, connection instructions, information management instructions, analysis instructions, data transmission instructions, and notification instructions, and the execution is controlled by the processor 210.

[0045] The charging management module 240 is used to receive charging input from the charger. The power management module 241 is used to connect the battery 242, the charging management module 240 and the processor 210. The power management module 241 receives input from the battery 242 and / or the charging management module 240 to power the processor 210, the internal memory 221, the display screen 290, the camera module 291 and the wireless communication module 260.

[0046] The wireless communication function of the mobile terminal 200 can be implemented through antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, modulation and demodulation processor and baseband processor. Among them, antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals; mobile communication module 250 can provide solutions for wireless communication including 2G / 3G / 4G / 5G applied on mobile terminal 200; modulation and demodulation processor can include modulator and demodulator; wireless communication module 260 can provide solutions for wireless communication including wireless local area network (WLAN) (such as wireless fidelity (Wi-Fi) network), Bluetooth (BT) and the like applied on mobile terminal 200. In some embodiments, antenna 1 of mobile terminal 200 is coupled with mobile communication module 250, and antenna 2 is coupled with wireless communication module 260, so that mobile terminal 200 can communicate with network and other devices through wireless communication technology.

[0047] The mobile terminal 200 implements the display function through a GPU, a display screen 290, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 290 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 210 may include one or more GPUs that execute program instructions to generate or change display information.

[0048] The mobile terminal 200 can realize the shooting function through the ISP, the camera module 291, the video codec, the GPU, the display screen 290 and the application processor, etc. Among them, the ISP is used to process the data fed back by the camera module 291; the camera module 291 is used to capture static images or videos; the digital signal processor is used to process digital signals, and can process other digital signals in addition to digital image signals; the video codec is used to compress or decompress digital videos, and the mobile terminal 200 can also support one or more video codecs.

[0049] The external memory interface 222 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the mobile terminal 200. The external memory card communicates with the processor 210 through the external memory interface 222 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0050] The internal memory 221 can be used to store computer executable program codes, and the executable program codes include instructions. The internal memory 221 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the mobile terminal 200 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 221 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash memory (Universal Flash Storage, UFS), etc. The processor 210 executes various functional applications and data processing of the mobile terminal 200 by running instructions stored in the internal memory 221 and / or instructions stored in a memory provided in the processor.

[0051] The mobile terminal 200 can implement audio functions such as music playing and recording through the audio module 270 , the speaker 271 , the receiver 272 , the microphone 273 , the earphone interface 274 and the application processor.

[0052] The depth sensor 2801 is used to obtain the depth information of the scene. In some embodiments, the depth sensor can be arranged in the camera module 291.

[0053] The pressure sensor 2802 is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 2802 can be arranged on the display screen 290. There are many types of pressure sensors 2802, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc.

[0054] The gyro sensor 2803 can be used to determine the motion posture of the mobile terminal 200. In some embodiments, the angular velocity of the mobile terminal 200 around three axes (i.e., x, y, and z axes) can be determined by the gyro sensor 2803. The gyro sensor 2803 can be used for shooting anti-shake, navigation, somatosensory game scenes, etc.

[0055] In addition, sensors with other functions may be set in the sensor module 280 according to actual needs, such as an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc.

[0056] The mobile terminal 200 may also include other devices that provide auxiliary functions. For example, the key 294 includes a power key, a volume key, etc., and the user can generate key signal input related to the user settings and function control of the mobile terminal 200 through key input. For example, the indicator 292, the motor 293, the SIM card interface 295, etc.

[0057] In related technologies, when optimizing images, the main image and the secondary image are mainly registered, matched, and interpolated, and the main image is subjected to bilateral filtering to achieve the effect of contrast enhancement, or low-light images are enhanced by decomposing the brightness, solving the reflection component, and using the retinex model for contrast enhancement and noise suppression. All of these methods have the problems of large computational complexity, high power consumption, and low optimization progress.

[0058] The image optimization method and image optimization device according to the exemplary embodiments of the present disclosure are described in detail below.

[0059] Figure 3 The flowchart of an image optimization method in this exemplary embodiment includes the following steps:

[0060] Step S310, acquiring a current image and a reference image, and performing spatial domain filtering on the current image to obtain a first filtered image under a first filtering parameter and a second filtered image under a second filtering parameter;

[0061] Step S320, performing spatial domain filtering on the reference image to obtain a reference filtered image of the reference image under a first filtering parameter;

[0062] Step S330, performing time domain filtering on the first filtered image according to the reference filtered image to obtain a guide image and a first image to be output;

[0063] Step S340, performing contrast enhancement on the current image according to the second filtered image and the guide image to obtain a second image to be output;

[0064] Step S350: Obtain an optimized image according to the first image to be output and the second image to be output, and output the optimized image.

[0065] Compared with the existing technology, on the one hand, the second filtered image parameters obtained after denoising the image are directly applied to the contrast enhancement of the image. By reusing the calculation results, the amount of calculation and power consumption are reduced. On the other hand, the results of spatial domain filtering are used for contrast enhancement at the same time, which solves the problem of noise enhancement when details are enhanced. Furthermore, the contrast enhancement result of the first image to be output after time domain filtering is strengthened, thereby improving the optimization accuracy of the image.

[0066] The above steps are described in detail below in conjunction with embodiments.

[0067] In step S310, a current image and a reference image are acquired, and spatial domain filtering is performed on the current image to obtain a first filtered image under a first filtering parameter and a second filtered image under a second filtering parameter.

[0068] In an example implementation of the present disclosure, both the reference image and the current image may be images in a video, wherein the image frame being processed may be defined as the current image, wherein the number of reference images may be one, and the previous frame of the current image may be defined as the reference image; the number of reference images may also be multiple, and the previous frames of the current image may be defined as reference images. In this example implementation, the number of reference images is not specifically limited.

[0069] In this example implementation, reference may be made to Figure 4 As shown, the current image and the reference image may first be subjected to spatial filtering 410, and then the results of the spatial filtering may be subjected to temporal filtering 420, and finally contrast enhancement 430 may be performed to obtain an optimized image.

[0070] In this example implementation, the current image may first be subjected to spatial filtering to obtain a first filtered image under a first filtering parameter and a second filtered image under a second filtering parameter. Figure 5 As shown, it may specifically include steps S510 to S550.

[0071] In step S510, the current image is downsampled to obtain two identical sub-images.

[0072] In step S520, mean filtering is performed on the sub-image in a preset window to obtain a first mean value.

[0073] First, the current image can be downsampled to obtain two identical sub-images, that is, the current image can be downsampled by 2 times, and the downsampling module 610 can be used to downsample the current image to obtain two identical sub-images. Then, the sub-images can be mean filtered under a preset window to obtain a first mean value. For details, refer to Figure 6 As shown, a box filter 620 with a window size equal to the preset window is used to perform mean filtering on the sub-image to obtain a first mean.

[0074] In the real-time mode of this example, the size of the above-mentioned preset window is smaller than the size of the above-mentioned sub-image. The size of the preset window can be 0.25 times, 0.2 times, 0.1 times, etc. of the size of the above-mentioned sub-image, and can also be customized according to user needs, which is not specifically limited in this example implementation.

[0075] Step S530, performing mean filtering on the product of the two sub-images under the preset window to obtain a second mean, and obtaining a first variance according to the first mean and the second mean;

[0076] In this exemplary implementation, the product of the two sub-images can be calculated and mean filtered under a preset window, referring to Figure 5 As shown, a box filter with a window size equal to the preset window can still be used to perform mean filtering on the product of the sub-images to obtain the second mean.

[0077] It should be noted that when calculating the product of the above two sub-images, the pixel matrices of the two sub-images may be calculated in a matrix multiplication manner.

[0078] Then, the variance calculation module 630 may be used to calculate a first variance according to the first mean and the second mean, which may be specifically the square of the first mean minus the second mean.

[0079] Step S540: performing guided filtering on the current image according to the first variance and the first filtering parameter to obtain a first filtered image.

[0080] In this example implementation, a first filtering parameter may be determined first, and then the first filtering parameter filtering module 640 may be used to perform guided filtering on the current image according to the first filtering parameter and the first variance to obtain a first filtered image. In this example implementation, the guiding image and the input image of the guided filtering may both be the current image, and the following formula may be used to determine and calculate the first filtered image:

[0081] Q1=a1I+b1

[0082]

[0083]

[0084] Wherein Q1 is the first filtered image, VAR(I) identifies the first variance, I represents the current image, ε1 represents the first filtering parameter, p represents the guide map, which is the same as the current image I in the real-time mode of this example, and cov is the covariance calculation function.

[0085] Step S550: performing guided filtering on the current image according to the first variance and the second filtering parameter to obtain a second filtered image.

[0086] In this example implementation, a second filtering parameter may be first determined, and then the second filtering parameter filtering module 650 may be used to perform guided filtering on the current image according to the second filtering parameter and the first variance to obtain a second filtered image. In this example implementation, the guiding image and the input image of the guided filtering may both be the current image, and the second filtered image may be determined and calculated using the following formula:

[0087] Q2=a2I+b2

[0088]

[0089]

[0090] Wherein Q2 is the second filtered image, VAR(I) identifies the first variance, I represents the current image, ε2 represents the second filtering parameter, p represents the guide map, which is the same as the current image I in the real-time mode of this example, and cov is the covariance calculation function.

[0091] In this exemplary embodiment, the first filtering parameter is greater than the second filtering parameter, so that the second filtered image can retain a larger edge, so that the noise will not be enhanced when used for contrast enhancement, thereby improving the image optimization effect.

[0092] In step S320, spatial domain filtering is performed on the reference image to obtain a reference filtered image of the reference image under a first filtering parameter.

[0093] In this example implementation, the reference image may be spatially filtered to obtain a reference filtered image under a first filtering parameter. When the current image is optimized, the reference image has been optimized. Therefore, the reference filtered image under the first filtering parameter may be directly obtained by scanning the reference image.

[0094] In this example implementation, if the reference filtered image of the reference image cannot be directly obtained, the reference image can be first downsampled to obtain two identical sub-reference images; the sub-reference images can be mean filtered under a preset window to obtain a third mean; the product of the two sub-reference images can be mean filtered under a preset window to obtain a fourth mean, and a second variance can be obtained based on the third mean and the fourth mean; the reference image can be guided filtered based on the second variance and the first filtering parameter to obtain a reference filtered image.

[0095] Specifically, the reference image may be first downsampled to obtain two identical sub-reference images, that is, the reference image may be downsampled by 2 times to obtain two identical sub-reference images. Then, the sub-reference images may be mean filtered under a preset window to obtain a third mean, and the sub-reference images may be mean filtered using a box filter whose window size is the preset window to obtain the third mean.

[0096] In this example implementation, the product of the two sub-reference images may be calculated and mean filtered under a preset window, and a box filter with a window size of the preset window may still be used to mean filter the product of the sub-reference images to obtain the fourth mean.

[0097] Then, the second variance may be calculated according to the third mean and the fourth mean, and specifically may be the square of the third mean minus the fourth mean.

[0098] Then, guided filtering may be performed on the reference image according to the second variance and the first filtering parameter to obtain the reference filtered image. The process of guided filtering has been described in detail above, and therefore will not be described again here.

[0099] Step S330: performing time domain filtering on the first filtered image according to the reference filtered image to obtain a guide image and a first image to be output.

[0100] In this example implementation, the first filtered image may first be subjected to time domain filtering according to the reference filtered image to obtain a guide image; and then the first filtered image and the reference filtered image may be fused according to the guide image to obtain the first image to be output.

[0101] In this example implementation, when the first filtered image is subjected to time domain filtering according to the reference filtered image to obtain the guided image, the reference Figure 7 As shown, it may include steps S710 to S750

[0102] In step S710, a difference image between the first filtered image and the reference filtered image is obtained.

[0103] In this example implementation, refer to Figure 8 As shown, the difference image 810 may be obtained by subtracting the first filtered image obtained above from the reference filtered image.

[0104] In step S720, a salient region image in the first filtered image is obtained.

[0105] In this example implementation, refer to Figure 8As shown, the salient region may be extracted from the first filtered image, and a salient detection algorithm may be used to extract the salient region and perform differential display on the salient region to obtain a salient region image 820 .

[0106] In this example implementation, the saliency detection algorithm may be an LC algorithm, an HC algorithm, an AC algorithm, an FT algorithm, etc., which is not specifically limited in this example implementation.

[0107] In step S730, edge detection is performed on the first filtered image to obtain an edge image.

[0108] In this example implementation, refer to Figure 8 As shown, edge detection may be performed on the first filtered image to obtain an edge image 830, wherein the edge image is an image that differentially displays the edges in the first filtered image.

[0109] In this example implementation, the Sobel operator, the Laplace operator, and the Canny operator may be used to implement edge detection of the image, which is not specifically limited in this example implementation.

[0110] In step S740, the salient region image and the edge image are fused to obtain a fused image.

[0111] In this example implementation, refer to Figure 8 As shown, the image of the salient area and the edge image may be fused 840 to obtain a fused image. Specifically, the image of the salient area and the edge image may be added to obtain the fused image. The fused image includes details that the human eye is more sensitive to, which can improve the accuracy of image optimization.

[0112] In step S750, the guide image is obtained according to the fused image and the difference image.

[0113] In this example implementation, refer to Figure 8 As shown, the fused image and the difference image can be used as guide images.

[0114] In this example implementation, refer to Figure 8As shown, after obtaining the guide image, the first filtered image and the reference filtered image can be guided and fused 850 according to the guide image to obtain the first image to be output. Specifically, the image can be divided into multiple regions. If the pixels in the difference image in the region are less than the preset threshold, the pixel value of each point of the first image to be output in the region is the average value of the pixel values ​​of each point in the region of the first filtered image and the pixel value of each point in the region of the reference filtered image. If the pixels of the difference image in the region are greater than the preset threshold, the fused image of the region is the current image. The specific value of the preset threshold can be customized according to the needs of the user and is not specifically limited in the present example implementation.

[0115] In the real-time mode of this example, the fused image may be used as a guide so that details of the salient area and the edge area are not easily lost, thereby obtaining a better accuracy of the first image to be output.

[0116] In another example implementation, the fusion weight can be determined based on the above-mentioned difference image. The larger the pixel value of the difference image, the larger the fusion weight value of the first filtered image, which can make the first image to be output and the current image more similar.

[0117] Step S340: performing contrast enhancement on the current image according to the second filtered image and the guide image to obtain a second image to be output.

[0118] In an exemplary embodiment of the present disclosure, referring to Figure 8 As shown, a high-pass filtered image can be first obtained based on the current image and the second filtered image. Specifically, the high-pass filtered image is obtained by subtracting the second filtered image from the current image. Then, the high-pass filtered image is contrast enhanced 860 using the guide image as a guide to obtain the second image to be output.

[0119] In this example implementation, the fused image is used as a guide to focus on enhancing the details that are more sensitive to the human eye, and then the difference image is used to enhance and correct the contrast enhancement result, thereby further improving the accuracy of image optimization.

[0120] Step S350: Obtain an optimized image according to the first image to be output and the second image to be output, and output the optimized image.

[0121] In this example implementation, refer to Figure 8As shown, after obtaining the second image to be output, the second image to be output and the first image to be output can be optimized and fused 870 to obtain an optimized image. Specifically, the pixel matrix of the optimized image can be obtained by solving the average value of the pixel matrix of the first image to be output and the pixel matrix of the second image to be output.

[0122] In summary, in this exemplary embodiment, on the one hand, the second filtered image parameters obtained after denoising the image are directly applied to the contrast enhancement of the image, and the amount of calculation and power consumption are reduced by reusing the calculation results. On the other hand, the result of spatial domain filtering is used for contrast enhancement at the same time, which solves the problem of noise enhancement when detail enhancement is performed. Furthermore, the result of contrast enhancement is enhanced by using the first image to be output after time domain filtering, thereby improving the optimization accuracy of the image.

[0123] It should be noted that the above figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0124] For further reference, Fig. 9 As shown, in the implementation of this example, an image optimization device 900 is also provided, including a first filtering module 910, a second filtering module 920, a time domain filtering module 930, an image enhancement module 940 and an image optimization module 950. Among them:

[0125] The first filtering module 910 is used to obtain a current image and a reference image, and perform spatial domain filtering on the current image to obtain a first filtered image under a first filtering parameter and a second filtered image under a second filtering parameter; wherein the second filtering parameter is greater than the first filtering parameter.

[0126] In an exemplary embodiment, the first filtering module 910 can be used to downsample the current image to obtain two identical sub-images; perform mean filtering on the sub-images under a preset window to obtain a first mean; perform mean filtering on the product of the two sub-images under a preset window to obtain a second mean, and obtain a first variance based on the first mean and the second mean; perform guided filtering on the current image based on the first variance and the first filtering parameter to obtain a first filtered image; perform guided filtering on the current image based on the first variance and the second filtering parameter to obtain a second filtered image; wherein the second filtering parameter is greater than the first filtering parameter, and the reference image includes the previous frame image of the current image.

[0127] The second filtering module 920 may be configured to perform spatial domain filtering on the reference image to obtain a reference filtered image of the reference image under the first filtering parameters.

[0128] In an example implementation, the second filtering module 920 may downsample a reference image to obtain two identical sub-reference images; perform mean filtering on the sub-reference images under a preset window to obtain a third mean; perform mean filtering on the product of the two sub-reference images under a preset window to obtain a fourth mean, and obtain a second variance based on the third mean and the fourth mean; perform guided filtering on the reference image based on the second variance and the first filtering parameter to obtain a reference filtered image.

[0129] The time domain filtering module 930 may be configured to perform time domain filtering on the first filtered image according to the reference filtered image to obtain a guide image and a first image to be output.

[0130] In an example implementation, the time domain filtering module 930 may first perform time domain filtering on the first filtered image according to a reference filtered image to obtain a guiding image. Specifically, a difference image between the first filtered image and the reference filtered image may be obtained; a significant area image in the first filtered image may be obtained; edge detection may be performed on the first filtered image to obtain an edge image; the significant area image and the edge image may be fused to obtain a fused image; a guiding image may be obtained based on the fused image and the difference image; and then, based on the guiding image, a first image to be output that is fused with the reference filtered image may be obtained. Specifically, the first filtered image may be fused with the reference filtered image based on the guiding image and the difference image to obtain the first image to be output.

[0131] The image enhancement module 940 may be configured to perform contrast enhancement on the current image according to the second filtered image and the guide image to obtain a second image to be output.

[0132] In an example implementation of the present disclosure, the image enhancement module 940 may obtain a high-pass filtered image according to the second filtered image and the current image; and perform contrast enhancement on the high-pass filtered image using the guide image as a reference to obtain the second image to be output.

[0133] The image optimization module 950 may be configured to obtain and output an optimized image based on the first image to be output and the second image to be output.

[0134] The specific details of each module in the above device have been described in detail in the implementation method of the method part. The undisclosed details can be found in the implementation method of the method part, so they will not be repeated here.

[0135] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0136] The exemplary embodiments of the present disclosure also provide a computer-readable storage medium on which a program product capable of implementing the above-mentioned method of the present specification is stored. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section of the present specification.

[0137] It should be noted that the computer-readable medium shown in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0138] In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0139] In addition, program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0140] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0141] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image optimization method, characterized in that: include: Acquire a current image and a reference image, and perform spatial domain filtering on the current image to obtain a first filtered image under a first filtering parameter and a second filtered image under a second filtering parameter; wherein the second filtering parameter is greater than the first filtering parameter; Performing spatial domain filtering on the reference image to obtain a reference filtered image of the reference image under a first filtering parameter; Performing time domain filtering on the first filtered image according to the reference filtered image to obtain a guide image and a first image to be output, including: performing time domain filtering on the first filtered image according to the reference filtered image to obtain a guide image; fusing the first filtered image and the reference filtered image according to the guide image to obtain the first image to be output; Performing contrast enhancement on the current image according to the second filtered image and the guide image to obtain a second image to be output; Obtaining an optimized image according to the first image to be output and the second image to be output and outputting the optimized image; The step of performing time domain filtering on the first filtered image according to the reference filtered image to obtain the guided image includes: Acquire a difference image between the first filtered image and the reference filtered image; Acquire a salient region image in the first filtered image; Performing edge detection on the first filtered image to obtain an edge image; Fusing the salient region image and the edge image to obtain a fused image; The guide image is obtained according to the fused image and the difference image.

2. The method according to claim 1, characterized in that The performing spatial domain filtering on the current image to obtain a first filtered image under a first filtering parameter and a second filtered image under a second filtering parameter includes: Downsampling the current image to obtain two identical sub-images; Performing mean filtering on the sub-image in a preset window to obtain a first mean value; Performing mean filtering on the product of the two sub-images under the preset window to obtain a second mean, and obtaining a first variance according to the first mean and the second mean; Performing guided filtering on the current image according to the first variance and the first filtering parameter to obtain a first filtered image; The current image is subjected to guided filtering according to the first variance and the second filtering parameter to obtain a second filtered image.

3. The method according to claim 1, characterized in that The first image to be output is obtained by fusing the first filtered image and the reference filtered image according to the guide image, comprising: The first filtered image and the reference filtered image are fused according to the guide image and the difference image to obtain a first image to be output.

4. The method according to claim 1, characterized in that: Performing contrast enhancement on the current image according to the second filtered image and the guide image to obtain a second image to be output includes: Obtaining a high-pass filtered image according to the second filtered image and the current image; The guide image is used as a reference to perform contrast enhancement on the high-pass filtered image to obtain a second image to be output.

5. The method according to claim 1, characterized in that The performing spatial domain filtering on the reference image to obtain a reference filtered image of the reference image under a first filtering parameter includes: Downsampling the reference image to obtain two identical sub-reference images; Performing mean filtering on the sub-reference image in a preset window to obtain a third mean; Performing mean filtering on the product of the two sub-reference images under the preset window to obtain a fourth mean, and obtaining a second variance according to the third mean and the fourth mean; The reference image is subjected to guided filtering according to the second variance and the first filtering parameter to obtain a reference filtered image.

6. The method according to claim 1, characterized in that The current image and the reference image belong to the same video data, and the reference image includes a frame image previous to the current image.

7. An image optimization device, characterized in that: include: A first filtering module, used to obtain a current image and a reference image, and perform spatial domain filtering on the current image to obtain a first filtered image under a first filtering parameter and a second filtered image under a second filtering parameter; A second filtering module, configured to perform spatial domain filtering on the reference image to obtain a reference filtered image of the reference image under a first filtering parameter; a time domain filtering module, configured to perform time domain filtering on the first filtered image according to the reference filtered image to obtain a guide image and a first image to be output, comprising: performing time domain filtering on the first filtered image according to the reference filtered image to obtain a guide image; and fusing the first filtered image and the reference filtered image according to the guide image to obtain the first image to be output; an image enhancement module, configured to perform contrast enhancement on the current image according to the second filtered image and the guide image to obtain a second image to be output; An image optimization module, which obtains and outputs an optimized image based on the first image to be output and the second image to be output; The step of performing time domain filtering on the first filtered image according to the reference filtered image to obtain the guided image includes: Acquire a difference image between the first filtered image and the reference filtered image; Acquire a salient region image in the first filtered image; Performing edge detection on the first filtered image to obtain an edge image; Fusing the salient region image and the edge image to obtain a fused image; The guide image is obtained according to the fused image and the difference image.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the image optimization method according to any one of claims 1 to 6 is implemented.

9. An electronic device, characterized in that: include: processor; as well as A memory for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the image optimization method as described in any one of claims 1 to 6.

Images