Image optimization method and device, storage medium and electronic device

Abstract

The present disclosure relates to the field of image processing technology, and specifically to an image optimization method and device, a computer-readable storage medium, and an electronic device, wherein the method comprises: performing spatial domain filtering on the N-th frame image to obtain a first filtered image; obtaining a second filtered image after performing spatial domain filtering on the N-1-th frame image and a reference optimized image after optimizing the N-1-th frame image, wherein N is a positive integer greater than or equal to 2; determining a differential image according to the second filtered image and the first filtered image, and determining the state information of each pixel in the first filtered image according to the differential image; fusing the first filtered image and the reference optimized image according to the state information of each pixel to obtain a fused image; determining a weight image according to the differential image, and fusing the first filtered image and the fused image according to the weight image to obtain a target optimized image. The technical solution of the embodiment of the present disclosure improves the accuracy of image optimization and reduces the amount of calculation during image optimization.

Description

Background Art

[0002] With the development of mobile technology, camera technology has also developed rapidly. However, during the transmission of videos and images, various noises are usually mixed in, which reduces the visual effect of the video. Therefore, it is necessary to perform optimization processing such as noise reduction on the video and image.

[0003] When optimizing images and videos in the related art, the amount of calculation is large and the accuracy is low.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The purpose of the present disclosure is to provide an image optimization method, an image optimization device, a computer-readable medium and an electronic device, thereby improving the accuracy of image optimization at least to a certain extent and reducing the amount of calculation during image optimization.

[0006] According to a first aspect of the present disclosure, there is provided an image optimization method, comprising:

[0007] Performing spatial domain filtering on the Nth frame image to obtain a first filtered image;

[0008] Acquire a second filtered image after spatial domain filtering of the N-1th frame image and a reference optimized image after optimizing the N-1th frame image, where N is a positive integer greater than or equal to 2;

[0009] determining a differential image according to the second filtered image and the first filtered image, and determining state information of each pixel in the first filtered image according to the differential image;

[0010] fusing the first filtered image and the reference optimized image according to the state information of each pixel point to obtain a fused image;

[0011] A weight image is determined according to the difference image, and the first filtered image and the fused image are fused according to the weight image to obtain a target optimized image.

[0012] According to a second aspect of the present disclosure, there is provided an image optimization device, comprising:

[0013] A filtering module performs spatial domain filtering on the Nth frame image to obtain a first filtered image;

[0014] An acquisition module is provided to acquire a second filtered image after spatial domain filtering of the N-1th frame image and a reference optimized image after optimizing the N-1th frame image, wherein N is a positive integer greater than or equal to 2;

[0015] a determination module, which determines a difference image according to the second filtered image and the first filtered image, and determines state information of each pixel in the first filtered image according to the difference image;

[0016] A fusion module, used for fusing the first filtered image and the reference optimized image according to the state information of each pixel point to obtain a fused image;

[0017] The optimization module is used to determine a weight image according to the difference image, and fuse the first filtered image and the fused image according to the weight image to obtain a target optimized image.

[0018] According to a third aspect of the present disclosure, a computer-readable medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the above method is implemented.

[0019] According to a fourth aspect of the present disclosure, there is provided an electronic device, characterized in that it includes:

[0020] Processor; and

[0021] The memory is used to store one or more programs. When the one or more programs are executed by one or more processors, the one or more processors implement the above method.

[0022] An image optimization method provided by an embodiment of the present disclosure performs spatial domain filtering on the Nth frame image to obtain a first filtered image; obtains a second filtered image after spatial domain filtering on the N-1th frame image and a reference optimized image after optimizing the N-1th frame image, wherein N is a positive integer greater than or equal to 2; determines a differential image based on the second filtered image and the first filtered image, and determines the state information of each pixel in the first filtered image based on the differential image; fuses the first filtered image and the reference optimized image based on the state information of each pixel to obtain a fused image; determines a weight image based on the differential image, and fuses the first filtered image and the fused image based on the weight image to obtain a target optimized image. Compared with the prior art, on the one hand, the state information of each pixel is first determined, and then the first filtered image and the reference optimized image are fused based on the state information to obtain a fused image, and pixels in different states are fused in different ways, which can reduce the amount of calculation compared with the prior art method of first aligning and then weighted fusion. On the other hand, the present invention determines a weight image based on the differential image, and fuses the first filtered image and the fused image based on the weight image to obtain a target optimized image. Compared with the prior art in which only time domain denoising is performed on static areas and spatial domain denoising is used for moving areas, the optimization accuracy of the image is higher.

[0023] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0025] Figure 1 A schematic diagram showing an exemplary system architecture to which embodiments of the present disclosure may be applied;

[0026] Figure 2 A schematic diagram showing an electronic device to which the embodiments of the present disclosure can be applied;

[0027] Figure 3 A flowchart of an image optimization method in an exemplary embodiment of the present disclosure is schematically shown;

[0028] Figure 4 A flowchart of performing spatial domain filtering on an N-th frame image in an exemplary embodiment of the present disclosure is schematically shown;

[0029] Figure 5 A data flow diagram for performing spatial domain filtering on an N-th frame image in an exemplary embodiment of the present disclosure is schematically shown;

[0030] Figure 6 A flowchart for determining pixel status information in an exemplary embodiment of the present disclosure is schematically shown;

[0031] Figure 7 A flowchart of acquiring a fused image in an exemplary embodiment of the present disclosure is schematically shown;

[0032] Figure 8 A data flow diagram schematically illustrating an image optimization method in an exemplary embodiment of the present disclosure;

[0033] Fig. 9 The figure schematically shows the composition of the image optimization device in the exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0034] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0035] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0036] Figure 1 A schematic diagram of a system architecture of an exemplary application environment in which an image optimization method and device according to an embodiment of the present disclosure can be applied is shown.

[0037] like Figure 1 As shown, the system architecture 100 may include one or more of terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc. The terminal devices 101, 102, 103 may be various electronic devices with image processing functions, including but not limited to desktop computers, portable computers, smart phones, tablet computers, etc. It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices, networks and servers. For example, the server 105 may be a server cluster composed of multiple servers.

[0038] The image optimization method provided in the embodiment of the present disclosure is generally executed by the terminal devices 101, 102, and 103, and accordingly, the image optimization device is generally arranged in the terminal devices 101, 102, and 103. However, it is easy for those skilled in the art to understand that the image optimization method provided in the embodiment of the present disclosure can also be executed by the server 105, and accordingly, the image optimization device can also be arranged in the server 105, and this is not particularly limited in the present exemplary embodiment. For example, in an exemplary embodiment, the user can use the terminal devices 101, 102, and 103 to capture the Nth frame image, and then upload the Nth frame image to the server 105. After the server generates the target optimized image through the image optimization method provided in the embodiment of the present disclosure, the target optimized image is transmitted to the terminal devices 101, 102, and 103, etc.

[0039] An exemplary embodiment of the present disclosure provides an electronic device for implementing an image optimization method, which may be Figure 1 The terminal device 101, 102, 103 or the server 105 in the embodiment of the present invention comprises at least a processor and a memory, the memory is used to store executable instructions of the processor, and the processor is configured to execute the image optimization method by executing the executable instructions.

[0040] Below Figure 2 Taking the mobile terminal 200 in FIG. 1 as an example, the structure of the electronic device is exemplarily described. It should be understood by those skilled in the art that, in addition to the components specifically used for mobile purposes, Figure 2 The structure in the figure can also be applied to fixed type devices. In other embodiments, the mobile terminal 200 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure can be implemented in hardware, software, or a combination of software and hardware. The interface connection relationship between the components is only shown schematically and does not constitute a structural limitation of the mobile terminal 200. In other embodiments, the mobile terminal 200 may also adopt the same Figure 2 Different interface connection methods, or a combination of multiple interface connection methods.

[0041] like Figure 2As shown, the mobile terminal 200 may specifically include: a processor 210, an internal memory 221, an external memory interface 222, a Universal Serial Bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 271, a receiver 272, a microphone 273, an earphone interface 274, a sensor module 280, a display screen 290, a camera module 291, an indicator 292, a motor 293, a button 294, and a subscriber identification module (SIM) card interface 295, etc. The sensor module 280 may include a depth sensor 2801, a pressure sensor 2802, a gyroscope sensor 2803, etc.

[0042] The processor 210 may include one or more processing units, for example, the processor 210 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor and / or a neural network processor (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0043] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of the mobile terminal 200 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0044] The processor 210 is provided with a memory. The memory can store instructions for implementing six modular functions: detection instructions, connection instructions, information management instructions, analysis instructions, data transmission instructions, and notification instructions, and the execution is controlled by the processor 210.

[0045] The charging management module 240 is used to receive charging input from the charger. The power management module 241 is used to connect the battery 242, the charging management module 240 and the processor 210. The power management module 241 receives input from the battery 242 and / or the charging management module 240 to power the processor 210, the internal memory 221, the display screen 290, the camera module 291 and the wireless communication module 260.

[0046] The wireless communication function of the mobile terminal 200 can be implemented through antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, modulation and demodulation processor and baseband processor. Among them, antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals; mobile communication module 250 can provide solutions for wireless communication including 2G / 3G / 4G / 5G applied on mobile terminal 200; modulation and demodulation processor can include modulator and demodulator; wireless communication module 260 can provide solutions for wireless communication including wireless local area network (WLAN) (such as wireless fidelity (Wi-Fi) network), Bluetooth (BT) and the like applied on mobile terminal 200. In some embodiments, antenna 1 of mobile terminal 200 is coupled with mobile communication module 250, and antenna 2 is coupled with wireless communication module 260, so that mobile terminal 200 can communicate with network and other devices through wireless communication technology.

[0047] The mobile terminal 200 implements the display function through a GPU, a display screen 290, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 290 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 210 may include one or more GPUs that execute program instructions to generate or change display information.

[0048] The mobile terminal 200 can realize the shooting function through the ISP, the camera module 291, the video codec, the GPU, the display screen 290 and the application processor, etc. Among them, the ISP is used to process the data fed back by the camera module 291; the camera module 291 is used to capture static images or videos; the digital signal processor is used to process digital signals, and can process other digital signals in addition to digital image signals; the video codec is used to compress or decompress digital videos, and the mobile terminal 200 can also support one or more video codecs.

[0049] The external memory interface 222 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the mobile terminal 200. The external memory card communicates with the processor 210 through the external memory interface 222 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0050] The internal memory 221 can be used to store computer executable program codes, and the executable program codes include instructions. The internal memory 221 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the mobile terminal 200 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 221 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash memory (Universal Flash Storage, UFS), etc. The processor 210 executes various functional applications and data processing of the mobile terminal 200 by running instructions stored in the internal memory 221 and / or instructions stored in a memory provided in the processor.

[0051] The mobile terminal 200 can implement audio functions such as music playing and recording through the audio module 270 , the speaker 271 , the receiver 272 , the microphone 273 , the earphone interface 274 and the application processor.

[0052] The depth sensor 2801 is used to obtain the depth information of the scene. In some embodiments, the depth sensor can be arranged in the camera module 291.

[0053] The pressure sensor 2802 is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 2802 can be arranged on the display screen 290. There are many types of pressure sensors 2802, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc.

[0054] The gyro sensor 2803 can be used to determine the motion posture of the mobile terminal 200. In some embodiments, the angular velocity of the mobile terminal 200 around three axes (i.e., x, y, and z axes) can be determined by the gyro sensor 2803. The gyro sensor 2803 can be used for shooting anti-shake, navigation, somatosensory game scenes, etc.

[0055] In addition, sensors with other functions may be set in the sensor module 280 according to actual needs, such as an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc.

[0056] The mobile terminal 200 may also include other devices that provide auxiliary functions. For example, the key 294 includes a power key, a volume key, etc., and the user can generate key signal input related to the user settings and function control of the mobile terminal 200 through key input. For example, the indicator 292, the motor 293, the SIM card interface 295, etc.

[0057] When optimizing an image, some optimization methods use differential images to distinguish between moving and stationary areas in the image, and only fuse the image in the stationary area with the reference frame. The image in the moving area does not participate in the fusion, which is equivalent to only performing time domain noise reduction on the stationary area and using spatial domain noise reduction on the moving area. This results in poor optimization accuracy. Some optimization methods also perform image alignment first, and then perform weighted fusion on the aligned images, which requires a lot of calculations.

[0058] The image optimization method and image optimization device according to the exemplary embodiments of the present disclosure are described in detail below.

[0059] Figure 3 The flowchart of an image optimization method in this exemplary embodiment includes the following steps:

[0060] Step S310, performing spatial domain filtering on the Nth frame image to obtain a first filtered image;

[0061] Step S320, obtaining a second filtered image after spatial domain filtering of the N-1th frame image and a reference optimized image after optimizing the N-1th frame image, where N is a positive integer greater than or equal to 2;

[0062] Step S330, determining a differential image according to the second filtered image and the first filtered image, and determining state information of each pixel in the first filtered image according to the differential image;

[0063] Step S340, fusing the first filtered image and the reference optimized image according to the state information of each pixel to obtain a fused image;

[0064] Step S350, determining a weight image according to the difference image, and fusing the first filtered image and the fused image according to the weight image to obtain a target optimized image.

[0065] Compared with the prior art, on the one hand, the state information of each pixel is first determined, and then the first filtered image and the reference optimized image are fused according to the state information to obtain a fused image. Pixels in different states are fused in different ways, which can reduce the amount of calculation compared to the prior art method of first aligning and then weighted fusion. On the other hand, the present invention determines the weight image according to the differential image, and fuses the first filtered image and the fused image according to the weight image to obtain the target optimized image. Compared with the prior art method of only performing time domain denoising on the static area, the solution of using spatial domain denoising on the moving area has higher optimization accuracy for the image.

[0066] The following is a detailed introduction to each of the above steps.

[0067] In step S310, spatial domain filtering is performed on the Nth frame image to obtain a first filtered image;

[0068] In an exemplary embodiment, in this exemplary implementation, the above-mentioned current image can first be subjected to spatial filtering to obtain a first filtered image under a first filtering parameter and a second filtered image under a second filtering parameter, referring to Figure 4 As shown, it may specifically include steps S410 to S440.

[0069] In step S410, down-sampling the Nth frame image is performed to obtain two identical sub-images;

[0070] In step S420, mean filtering is performed on the sub-image in a preset window to obtain a first mean value;

[0071] Reference Figure 5 As shown, the Nth frame image may be downsampled first to obtain two identical sub-images, that is, the current image may be downsampled by 2 times, and the current image may be downsampled by a downsampling module 510 to obtain two identical sub-images. Then, the sub-images may be mean filtered under a preset window to obtain a first mean value, and the box filter 520 with a window size of the preset window may be used to mean filter the sub-images to obtain the first mean value.

[0072] In the real-time mode of this example, the size of the above-mentioned preset window is smaller than the size of the above-mentioned sub-image. The size of the preset window can be 0.25 times, 0.2 times, 0.1 times, etc. of the size of the above-mentioned sub-image, and can also be customized according to user needs, which is not specifically limited in this example implementation.

[0073] Step S430, performing mean filtering on the product of the two sub-images under a preset window to obtain a second mean, and obtaining a variance based on the first mean and the second mean;

[0074] In this exemplary implementation, the product of the two sub-images can be calculated and mean filtered under a preset window, referring to Figure 5 As shown, a box filter with a window size equal to the preset window can still be used to perform mean filtering on the product of the sub-images to obtain the second mean.

[0075] Then, the variance calculation module 530 may be used to calculate the variance according to the first mean and the second mean, which may be specifically the square of the first mean minus the second mean.

[0076] Step S440, performing guided filtering on the Nth frame image according to the first variance to obtain a first filtered image;

[0077] In this example implementation, a filtering parameter may be determined first, and then the guided filtering module 540 may be used to perform guided filtering on the current image according to the filtering parameter and the variance to obtain a first filtered image. In this example implementation, the guiding map of the guided filtering and the input image may both be the Nth frame image, and the following formula may be used to determine and calculate the first filtered image:

[0078] Q1=a1I+b1

[0079]

[0080]

[0081] Wherein Q1 is the first filtered image, VAR(I) represents variance, I represents the Nth frame image, ε1 represents the filtering parameter, p represents the guide map, which is the same as the Nth frame image I in the real-time mode of this example, and cov is the covariance calculation function.

[0082] In an example implementation of the present disclosure, the above-mentioned spatial domain filtering may also adopt other filtering methods such as "Gaussian blur" and "Laplacian operator", which are not specifically limited in this example implementation.

[0083] In step S320, a second filtered image after spatial domain filtering of the N-1th frame image and a reference optimized image after optimizing the N-1th frame image are obtained, where N is a positive integer greater than or equal to 2.

[0084] In an example implementation of the present disclosure, a second filtered image after spatial filtering of the N-1th frame image may be obtained. The specific process of spatial filtering may refer to the process of obtaining the first filtered image, which will not be repeated here.

[0085] In this example implementation, the reference optimized image is an image obtained by optimizing the N-1th frame image using the optimization method of the present application, wherein N is a positive integer greater than or equal to 2. When N is equal to 2, the reference optimized image is the N-1th frame image, that is, the first frame image, or it may be the empty filtering result of the N-1th frame image, that is, the second filtered image corresponding to the first frame graphic.

[0086] In this example implementation, when optimizing each frame of image, the spatial filtering result of the frame of image and the final optimization result may be saved as a reference for optimizing the next frame of image.

[0087] Step S330, determining a differential image according to the second filtered image and the first filtered image, and determining state information of each pixel of the first filtered image according to the differential image;

[0088] In an example implementation of the present disclosure, a differential image can be determined based on the second filtered image and the first filtered image. Specifically, the differential image can be obtained by subtracting each corresponding pixel point of the first filtered image from that of the second filtered image. In this example implementation, the method for obtaining the differential image is not specifically limited.

[0089] In this example implementation, the state information of each pixel point of the first filtered image is determined according to the differential image, referring to Figure 6 As shown, it may specifically include step S610 and step S620.

[0090] In step S610, a pixel threshold is preconfigured to determine the pixel value of each pixel in the differential image;

[0091] In this example implementation, a pixel threshold can be preset and configured, and then the pixel value of each pixel point in the above-mentioned differential image is determined. The preset threshold can be 0, or 0.5, 0.6, etc., and can also be customized according to user needs. There is no specific limitation in this example implementation.

[0092] In step S620, the state information of each pixel is determined according to the pixel threshold and the pixel value of each pixel.

[0093] In this example implementation, the pixel value of each pixel point can be compared with the pixel threshold. If the pixel value of the pixel point is less than the pixel threshold, the pixel point is determined to be in the second state; if the pixel value of the pixel point is greater than or equal to the pixel threshold, the pixel point is determined to be in the first state.

[0094] Specifically, we can set D i represents the pixel value of the i-th pixel in the differential image, T0 represents the above pixel threshold, then when D iWhen D is less than T0, the state information of the pixel is determined to be the second state. Otherwise, when D i When it is greater than or equal to T0, the state information of the pixel is determined to be the first state.

[0095] For example, if the first filtered image includes a moving area and a stationary area relative to the second filtered image, then the pixels in the first state may be located in the moving area, and the pixels in the second state may be located in the stationary area. That is, the area formed by the pixels in the first state may be used as the moving area, and the area formed by the pixels in the second state may be used as the stationary area.

[0096] Step S340, fusing the first filtered image and the reference optimized image according to the state information of each pixel to obtain a fused image;

[0097] In an exemplary embodiment of the present disclosure, the processor may fuse the first filtered image and the reference optimized image according to the state information of each pixel point to obtain a fused image. Figure 7 As shown, steps S710 to S740 may be performed for each pixel in the first filtered image.

[0098] In step S710, the target pixel point corresponding to the pixel point in the fused image is determined;

[0099] In this example implementation, the first filtered image and the fused image have the same size and the same number of pixels. Therefore, each pixel in the first filtered pixel corresponds one-to-one to each target pixel in the fused image. The target pixel corresponding to the first filtered pixel in the fused image can be determined, wherein the pixel value of the target pixel is unknown and the pixel value of the pixel in the first filtered image is known.

[0100] In step S720, a reference pixel value of a reference pixel point in a reference optimized image corresponding to the pixel point is determined;

[0101] In this example implementation, the first filtered image has the same size and the same number of pixels as the reference optimized image, so each pixel in the first filtered image can correspond one-to-one to each reference pixel in the reference optimized image, and the reference pixel corresponding to the first filtered pixel in the fused image can be determined. The pixel value of each reference pixel is known, and the pixel value of the reference pixel corresponding to the pixel can be determined.

[0102] In step S730, if the state information of the pixel point is the second state, the reference pixel value is used as the pixel value of the target pixel point.

[0103] In this example implementation, if the pixel is determined to be in the second state, the pixel value of the reference pixel corresponding to the pixel may be used as the pixel value of the target pixel corresponding to the pixel.

[0104] In step S740, if the state information of the pixel point is the first state, a reference area including the pixel point is determined, and a reference position of the pixel point in the reference area is determined, and a target area matching the reference area is obtained in the reference optimization map according to the reference area, and the pixel value of the target pixel point is determined according to the target area and the reference position.

[0105] In this example implementation, if the state information of the above-mentioned pixel point is determined to be the first state, a reference area including the pixel point can be determined, and the reference position of the above-mentioned pixel point in the above-mentioned reference area can be determined, wherein the reference position of the above-mentioned pixel point can be at the center point of the above-mentioned reference area, or at the edge of the above-mentioned reference area, and can also be customized according to user needs, which is not specifically limited in this example implementation.

[0106] The reference area may include 10 pixels, or 5, 6, 9, etc. pixels, and may be customized according to user needs, which is not specifically limited in this example implementation. The reference area may be a square or a circle, which may be customized according to user needs, which is not specifically limited in this example implementation.

[0107] After determining the above-mentioned reference area, the above-mentioned reference optimization image can be traversed through the above-mentioned reference area to determine the target area matching the above-mentioned reference area in the reference optimization image. Specifically, a matching degree threshold can be first determined, and then the area in the reference optimization image with the greatest matching degree with the above-mentioned reference area is determined as a candidate area, and the size of the matching degree of the above-mentioned candidate area and the above-mentioned matching degree threshold is determined. When the matching degree of the above-mentioned candidate area is greater than or equal to the matching degree threshold, the above-mentioned candidate area is used as the target area. If the matching degree threshold of the candidate area is less than the above-mentioned matching degree threshold, it is determined that the target area cannot be matched in the reference optimization image.

[0108] When the above-mentioned target area can be matched, the pixel value of the pixel point at the target position corresponding to the reference position in the target area is determined, and it is used as the pixel value of the target pixel point corresponding to the above-mentioned pixel point. If the target area cannot be matched in the reference optimized image, the pixel value of the pixel point is directly used as the pixel value of the target pixel point.

[0109] After executing the above steps for each pixel, a fused image can be obtained.

[0110] In another example implementation of the present disclosure, when the state information of the above-mentioned pixel point is the second state, or the above-mentioned pixel point is the first state and can be matched to the corresponding target area, the pixel value of the above-mentioned pixel point and the pixel value of the reference pixel point corresponding to the pixel point can be fused by weighted fusion to obtain the pixel value of the above-mentioned target pixel point.

[0111] Specifically, the weights of the pixel point and the reference pixel point can be determined according to the pixel value of the differential pixel point corresponding to the pixel point in the differential image, wherein the pixel value of the differential pixel point is negatively correlated with the weight of the reference pixel point, and the specific correlation coefficient can be customized according to user needs and is not specifically limited in this example implementation.

[0112] In this example implementation, when calculating the pixel value of the target pixel, the weight a of the pixel and the weight b of the reference pixel may be determined first, the pixel value A of the pixel and the pixel value B of the reference pixel may be determined, the pixel value of the target pixel may be set to S, and the pixel value of the target pixel may be calculated by the following formula:

[0113] S = (aA + bB) / (a ​​+ b)

[0114] In this example implementation, for a pixel point that cannot be matched to the target area, the pixel value of the pixel point can be directly used as the pixel value of the target pixel point.

[0115] The fused image can be obtained by executing the above steps for each pixel in the first filtered image.

[0116] Step S350, determining a weight image according to the difference image, and fusing the first filtered image and the fused image according to the weight image to obtain a target optimized image.

[0117] In an example implementation of the present disclosure, after obtaining the above-mentioned fused image, a weight image can be determined according to the above-mentioned differential image. Specifically, the mapping relationship between the pixel points in the above-mentioned differential image and the weight of the reference optimized image can be first determined, and then the above-mentioned weight image can be determined according to the above-mentioned mapping relationship, wherein the above-mentioned mapping relationship can be customized according to the user's requirements for image fusion accuracy, and is not specifically limited in this example implementation. The pixel value of the above-mentioned differential image is negatively correlated with the weight of the above-mentioned reference optimized image.

[0118] In this example implementation, the weight value of the determined weight image corresponds to the weight value of the reference optimized image. It should be noted that the weight value in the above-mentioned weight image can also be the weight value of the above-mentioned first filtered image. At this time, the weight value is positively correlated with the pixel value of the above-mentioned differential image, which is not specifically limited in this example implementation.

[0119] In this example implementation, after the weight map image is determined, the first filtered image and the fused image may be fused according to the weights to obtain a target optimized image.

[0120] Specifically, the first filtered image and the fused image may be fused using a pixel fusion formula to obtain a target optimized image, wherein the pixel fusion formula is:

[0121] T i =M i *W i +I i *(1-W i )

[0122] Among them, T i represents the target optimization image, M i Fusion image, I i represents the first filtered image, W i Represents the weight image, i represents the pixel coordinate, where i is a positive integer.

[0123] In an exemplary embodiment of the present disclosure, the image optimization method may include post-processing the target and optimized images using the Nth frame image to update the target optimized image.

[0124] Specifically, in the post-processing process, the target optimized image may be subjected to operations such as removing ghosts, restoring textures, and enhancing details without using the N-th frame image as a reference, so as to improve the optimization effect of the N-th frame image.

[0125] In this example implementation, refer to Figure 8 As shown, the image optimization scheme in this scheme is generally described. First, the N-th frame image 801 is subjected to spatial domain filtering 802 to obtain the first filtered image 803, and then the second filtered image 804 and the reference optimized image 808 corresponding to the N-1-th frame image can be obtained. The acquisition of the second filtered image 804 and the reference optimized image 808 has been described in detail above, so it is not repeated here. Then, the first filtered image 803 and the second filtered image 804 are subjected to differential processing 805 to obtain a differential image 806, and then the first filtered image 803 and the reference optimized image 808 are subjected to fusion processing 809 based on the differential image 806 to obtain a fused image 810, and then the weight image 807 can be determined using the differential image 806, and the first filtered image 803 and the fused image 810 are subjected to weighted fusion 811 based on the weight image 807, and the N-th frame image 801 is used as a reference for post-processing 812 to obtain a target optimized image 813. The specific details of each of the above steps have been described in detail above, so they will not be repeated here.

[0126] In summary, in this exemplary embodiment, the state information of each pixel is first determined. If the state information of the pixel is the second state, the reference pixel value is used as the pixel value of the target pixel. If the state information of the pixel is the first state, a reference area is determined with the pixel as the center, and a target area matching the reference area is obtained in the reference optimization map according to the reference area. The pixel value of the center point of the target area is used as the pixel value of the target pixel. Pixels in different states are fused in different ways, which can reduce the amount of calculation compared to the prior art method of aligning first and then weighted fusion. On the other hand, the present disclosure determines the weight image based on the differential image, and fuses the first filtered image and the fused image again according to the weight image to obtain the target optimized image. Compared with the prior art method of only performing time domain denoising on the static area, the solution of using spatial domain denoising on the moving area has higher optimization accuracy for the image.

[0127] It should be noted that the above figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0128] For further reference, Fig. 9 As shown, in the implementation of this example, an image optimization device 900 is also provided, including a filtering module 910, an acquisition module 920, a determination module 930, a fusion module 940 and an optimization module 950. Among them:

[0129] The filtering module 910 can be used to perform spatial domain filtering on the Nth frame image to obtain a first filtered image. Specifically, the Nth frame image can be downsampled to obtain two identical sub-images; the sub-images can be mean filtered under a preset window to obtain a first mean; the product of the two sub-images can be mean filtered under a preset window to obtain a second mean, and the variance can be obtained based on the first mean and the second mean; and the Nth frame image can be guided filtered based on the first variance to obtain the first filtered image.

[0130] The acquisition module 920 can be used to acquire a second filtered image after spatial domain filtering of the N-1th frame image and a reference optimized image after optimizing the N-1th frame image, where N is a positive integer greater than or equal to 2.

[0131] The determination module 930 may be configured to determine a differential image according to the second filtered image and the first filtered image, and determine state information of each pixel of the first filtered image according to the differential image.

[0132] In this example implementation, the determination module 930 may pre-configure a pixel threshold to determine the pixel value of each pixel in the differential image; determine the state information of each pixel according to the pixel threshold and the pixel value of each pixel. If the pixel value of the pixel is less than the pixel threshold, the pixel is determined to be in the second state; if the pixel value of the pixel is greater than or equal to the pixel threshold, the pixel is determined to be in the first state.

[0133] The fusion module 940 may be configured to fuse the first filtered image and the reference optimized image according to the status information of each pixel point to obtain a fused image.

[0134] In an example implementation, the fusion module 940 may perform the following steps for each pixel in the first filtered image: determine the target pixel corresponding to the pixel in the fused image; determine the reference pixel value of the reference pixel in the reference optimized image corresponding to the pixel; if the state information of the pixel is the second state, use the reference pixel value as the pixel value of the target pixel; if the state information of the pixel is the first state, determine a reference area including the pixel, and determine the reference position of the pixel in the reference area, obtain a target area matching the reference area in the reference optimized image according to the reference area, and determine the pixel value of the target pixel according to the target area and the reference position. When obtaining a target area matching the reference area in the reference optimized image according to the reference area, a candidate area in the reference optimized image with the greatest matching degree with the above-mentioned reference area may be determined; configure a matching degree threshold, and when the matching degree of the candidate area is greater than or equal to the matching degree threshold, determine the candidate area as the target area.

[0135] The optimization module 950 can be used to determine the weight image according to the difference image, and fuse the first filtered image and the fused image according to the weight image to obtain the target optimized image. Specifically, the mapping relationship between the pixel value of the difference image and the weight of the reference optimized image can be first determined; the weight image can be obtained according to the mapping relationship and the difference image.

[0136] In an exemplary embodiment, the optimization module 950 may fuse the first filtered image and the fused image using a pixel fusion formula, wherein the pixel fusion formula is:

[0137] T i =M i *W i +I i *(1-W i )

[0138] Among them, T i represents the target optimization image, M i Fusion image, I i represents the first filtered image, W iRepresents the weight image, i represents the pixel coordinate, where i is a positive integer.

[0139] The specific details of each module in the above device have been described in detail in the implementation method of the method part. The undisclosed details can be found in the implementation method of the method part, so they will not be repeated here.

[0140] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0141] The exemplary embodiments of the present disclosure also provide a computer-readable storage medium on which a program product capable of implementing the above-mentioned method of the present specification is stored. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section of the present specification.

[0142] It should be noted that the computer-readable medium shown in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0143] In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0144] In addition, program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0145] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0146] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image optimization method, characterized in that: include: Performing spatial domain filtering on the Nth frame image to obtain a first filtered image; Acquire a second filtered image after spatial domain filtering of the N-1th frame image and a reference optimized image after optimizing the N-1th frame image, where N is a positive integer greater than or equal to 2; Determine a differential image according to the second filtered image and the first filtered image, and preconfigure a pixel threshold to determine a pixel value of each pixel in the differential image; If the pixel value of the pixel point is less than the pixel threshold, determining that the pixel point is in the second state; If the pixel value of the pixel point is greater than or equal to the pixel threshold, the pixel point is determined to be in the first state; For each pixel in the first filtered image, perform the following steps: Determine the target pixel point corresponding to the pixel point in the fused image; Determine a reference pixel value of a reference pixel point in a reference optimized image corresponding to the pixel point; If the state information of the pixel point is the second state, using the reference pixel value as the pixel value of the target pixel point; If the state information of the pixel is the first state, a reference area including the pixel is determined, and a reference position of the pixel in the reference area is determined, a target area matching the reference area is obtained in the reference optimization map according to the reference area, and a pixel value of the target pixel is determined according to the target area and the reference position; A weight image is determined according to the difference image, and the first filtered image and the fused image are fused according to the weight image to obtain a target optimized image.

2. The method according to claim 1, characterized in that The acquiring, in the reference optimization map, a target region matching the reference region according to the reference region comprises: Determine a candidate region in the reference optimized image that has the greatest matching degree with the reference region; A matching degree threshold is configured, and when the matching degree of the candidate area is greater than or equal to the matching degree threshold, the candidate area is determined as the target area.

3. The method according to claim 2, characterized in that The method further comprises: When the matching degree of the candidate area is less than the matching degree threshold, the pixel value of the pixel point is used as the pixel value of the target pixel point.

4. The method according to claim 1, characterized in that: The step of determining a weight image according to the difference image comprises: Determining a mapping relationship between the differential image pixel value and the reference optimized image weight; Acquire the weight image according to the mapping relationship and the difference image; The pixel value of the differential image is negatively correlated with the weight of the reference optimized image.

5. The method according to claim 1, characterized in that Determining a weight image according to the difference image, and fusing the first filtered image and the fused image according to the weight image to obtain a target optimized image, comprising: The first filtered image and the fused image are fused using a pixel fusion formula to obtain the target optimized image, wherein the pixel fusion formula is: ； in, represents the target optimization image, Fuse images, represents the first filtered image, Represents the weight image, i represents the pixel coordinate, where i is a positive integer.

6. The method according to claim 1, characterized in that The method further comprises: The target optimized image is post-processed according to the Nth frame image to update the target optimized image.

7. The method according to claim 1, characterized in that Performing spatial domain filtering on the Nth frame image to obtain a first filtered image includes: Downsampling the Nth frame image to obtain two identical sub-images; Performing mean filtering on the sub-image in a preset window to obtain a first mean value; Performing mean filtering on the product of the two sub-images under the preset window to obtain a second mean, and obtaining a variance according to the first mean and the second mean; A first filtered image is obtained by performing guided filtering on the Nth frame image according to the variance and the image.

8. The method according to claim 1, characterized in that When N is equal to 2, the reference optimized image is the same as the N-1th frame image.

9. An image optimization device, characterized in that: include: A filtering module performs spatial domain filtering on the Nth frame image to obtain a first filtered image; An acquisition module is provided to acquire a second filtered image after spatial domain filtering of the N-1th frame image and a reference optimized image after optimizing the N-1th frame image, wherein N is a positive integer greater than or equal to 2; a determination module, which determines a differential image according to the second filtered image and the first filtered image, and preconfigures a pixel threshold to determine a pixel value of each pixel in the differential image; If the pixel value of the pixel point is less than the pixel threshold, the pixel point is determined to be in the second state; if the pixel value of the pixel point is greater than or equal to the pixel threshold, the pixel point is determined to be in the first state; A fusion module is used to perform the following steps for each pixel point in the first filtered image: determine the target pixel point corresponding to the pixel point in the fused image; determine the reference pixel value of the reference pixel point in the reference optimized image corresponding to the pixel point; if the state information of the pixel point is the second state, use the reference pixel value as the pixel value of the target pixel point; If the state information of the pixel is the first state, a reference area including the pixel is determined, and a reference position of the pixel in the reference area is determined, a target area matching the reference area is obtained in the reference optimization map according to the reference area, and a pixel value of the target pixel is determined according to the target area and the reference position; The optimization module is used to determine a weight image according to the difference image, and fuse the first filtered image and the fused image according to the weight image to obtain a target optimized image.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the image optimization method according to any one of claims 1 to 8 is implemented.

11. An electronic device, characterized in that: include: processor; as well as A memory for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the image optimization method as described in any one of claims 1 to 8.

Images