Image processing apparatus, method, storage medium, and program product

Abstract

The embodiment of the invention belongs to the image processing technology, and provides an image processing device and method, a storage medium and a program product, the image processing device comprises an image sensor and a controller connected with the image sensor, and the controller is configured to be connected with the image sensor; obtaining a target image, wherein the target image is an image obtained by fusing a plurality of frames of images collected by the image sensor under different exposure parameters; processing the target image, and determining a fusion proportion matrix of the target image; obtaining at least one noise reduction compensation parameter of the target image according to the fusion proportion matrix and a preset noise reduction parameter; and performing noise reduction processing on the target image after global noise reduction based on the at least one noise reduction compensation parameter. In the method, the noise reduction compensation parameter used for performing secondary noise reduction on the fused image is obtained based on the preset noise reduction parameter corresponding to the reference region, and secondary noise reduction processing is performed on the fused image based on the noise reduction compensation parameter, so that the image quality can be effectively improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology. More specifically, it relates to an image processing apparatus, method, storage medium, and program product. Background Technology

[0002] With the development of technology, users have higher and higher requirements for image quality. As multiple exposure fusion technology can enhance the dynamic range of images and provide higher quality images, it has been widely used.

[0003] Multiple exposure fusion technology typically involves capturing multiple images of the same object with different exposure times and then fusing them to obtain a single high dynamic range image. Since images inherently contain noise, global noise reduction is usually applied to the resulting high dynamic range image to improve image quality.

[0004] However, images with different exposure times exhibit different noise characteristics, resulting in poor noise reduction performance when global noise reduction is applied to the fused image, leading to poor image quality after noise reduction. Summary of the Invention

[0005] Exemplary embodiments of this application provide an image processing apparatus, method, storage medium, and program product that can improve the noise reduction effect on images fused with different exposure times, thereby improving image quality.

[0006] In a first aspect, embodiments of this application provide an image processing device, the image processing device comprising: an image sensor and a controller connected to the image sensor, the controller being configured to;

[0007] Acquire a target image, which is an image obtained by fusing multiple frames of images acquired by the image sensor under different exposure parameters; the multiple frames of images are images acquired from the same scene;

[0008] The target image is processed to determine the fusion ratio matrix of the target image; the fusion ratio matrix is ​​used to fuse pixels at the same position in multiple frames of images acquired under different exposure parameters;

[0009] Based on the fusion ratio matrix and preset noise reduction parameters, at least one noise reduction compensation parameter of the target image is obtained;

[0010] The target image after global denoising is denoised based on at least one of the denoising compensation parameters.

[0011] In some embodiments, the controller is configured to:

[0012] Based on the exposure parameters, content recognition is performed on the target image to obtain the fusion weights of pixels at the same position in the multiple frames of images under different exposure parameters.

[0013] The values ​​of the pixels are weighted according to the fusion weights to obtain the fusion ratio matrix.

[0014] In some embodiments, the controller is configured to:

[0015] The fusion ratio matrix is ​​split to obtain reference matrices corresponding to different exposure parameters;

[0016] Obtain the preset noise reduction parameters corresponding to the exposure modes of different exposure parameters; different exposure modes correspond to one or more preset noise reduction parameters;

[0017] The noise reduction compensation parameters are obtained by multiplying the reference matrix and the preset noise reduction parameters.

[0018] In some embodiments, the exposure parameters include exposure duration, and the controller is configured to:

[0019] Determine the exposure duration range to which the exposure duration belongs;

[0020] Based on the exposure duration range and the mapping relationship between the exposure duration range and the preset noise reduction parameters, the preset noise reduction parameters are obtained.

[0021] In some embodiments, the controller is configured to:

[0022] Obtain the target region in the target image that needs to be denoised, and the denoising compensation parameters corresponding to the target region;

[0023] The target area is subjected to noise reduction processing based on the noise reduction compensation parameters.

[0024] In some embodiments, the controller is configured to:

[0025] Based on the reference matrix of the target region, the reference image frame of the target region in the fusion process is determined;

[0026] The noise reduction compensation parameters corresponding to the reference image frame are used as the noise reduction compensation parameters corresponding to the target region.

[0027] In some embodiments, the controller is configured to:

[0028] Perform global noise reduction on the target image;

[0029] Based on the noise distribution of the target image before global denoising and the noise distribution of the target image after global denoising, the target image is partitioned for identification. If it is determined that there is a target region that needs to be denoised and compensated, the step of obtaining the fusion ratio matrix is ​​executed.

[0030] Secondly, embodiments of this application provide an image processing method, including:

[0031] Acquire a target image, which is an image obtained by fusing multiple frames of images captured under different exposure parameters; the multiple frames of images are images captured from the same scene;

[0032] The target image is processed to determine the fusion ratio matrix of the target image; the fusion ratio matrix is ​​used to fuse pixels at the same position in multiple frames of images acquired under different exposure parameters;

[0033] Based on the fusion ratio matrix and preset noise reduction parameters, at least one noise reduction compensation parameter of the target image is obtained;

[0034] The target image after global denoising is denoised based on at least one of the denoising compensation parameters.

[0035] Thirdly, embodiments of this application provide an image processing apparatus, including:

[0036] The acquisition module is used to acquire a target image, which is an image obtained by fusing multiple frames of images acquired under different exposure parameters; the multiple frames of images are images acquired from the same scene.

[0037] The first processing module is used to process the target image and determine the fusion ratio matrix of the target image; the fusion ratio matrix is ​​used to fuse pixels at the same position in multiple frames of images acquired under different exposure parameters.

[0038] The second processing module is used to obtain at least one noise reduction compensation parameter of the target image based on the fusion ratio matrix and preset noise reduction parameters.

[0039] A noise reduction module is used to perform noise reduction processing on the globally denoised target image based on at least one of the noise reduction compensation parameters.

[0040] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method described in the second aspect.

[0041] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the method described in the second aspect.

[0042] In a sixth aspect, embodiments of this application provide a chip, the chip including a processor, the processor being configured to invoke a computer program in memory to perform the method described in the second aspect.

[0043] The image processing apparatus, method, storage medium, and program products provided in this application embodiment include: an image processing apparatus comprising: an image sensor and a controller connected to the image sensor; the controller being configured to: acquire a target image, wherein the target image is an image obtained by fusing multiple frames acquired by the image sensor under different exposure parameters; the multiple frames are images acquired from the same scene; process the target image to determine a fusion ratio matrix of the target image; the fusion ratio matrix is ​​used to fuse pixels at the same position in the multiple frames acquired under different exposure parameters; obtain at least one noise reduction compensation parameter of the target image based on the fusion ratio matrix and preset noise reduction parameters; and perform noise reduction processing on the globally denoised target image based on at least one of the noise reduction compensation parameters. In the above method, based on the preset noise reduction parameters corresponding to the reference region, noise reduction compensation parameters are obtained for secondary noise reduction of the fused image. Performing secondary noise reduction processing on the fused image based on the noise reduction compensation parameters can effectively improve image quality. Attached Figure Description

[0044] To more clearly illustrate the implementation methods in the embodiments of this application or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0045] Figure 1 A scenario diagram provided for an embodiment of this application;

[0046] Figure 2A A schematic diagram of an image processing device provided in an embodiment of this application. Figure 1 ;

[0047] Figure 2B Schematic diagram 2 of an image processing device provided in an embodiment of this application;

[0048] Figure 2C A schematic diagram of an image processing device provided in an embodiment of this application. Figure 3 ;

[0049] Figure 2D A schematic diagram of an image processing device provided in an embodiment of this application. Figure 4 ;

[0050] Figure 3This is a schematic diagram of the structure of a display device provided in an embodiment of this application;

[0051] Figure 4 A flowchart illustrating an image processing method provided in this application embodiment. Figure 1 ;

[0052] Figure 5 A schematic diagram illustrating an image fusion method provided in an embodiment of this application;

[0053] Figure 6 A schematic flowchart of an image processing method provided in an embodiment of this application is shown below;

[0054] Figure 7 This is a schematic diagram of the structure of an image processing device provided in an embodiment of this application. Detailed Implementation

[0055] To make the objectives, implementation methods and advantages of this application clearer, the exemplary implementation methods of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the described exemplary embodiments are only some embodiments of this application, and not all embodiments.

[0056] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0057] Furthermore, the terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclusively include, for example, a product or device that includes a series of components is not necessarily limited to those that are explicitly listed, but may include other components that are not explicitly listed or that are inherent to such product or device.

[0058] In the exemplary embodiments of this application, the terms "first" and "second" are used to distinguish identical or similar items with substantially the same function and effect, without limiting their order. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that the terms "first" and "second" are not necessarily different.

[0059] It should be noted that in the exemplary embodiments of this application, the words "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0060] With the development of technology, users have higher and higher requirements for image quality. As multiple exposure fusion technology can enhance the dynamic range of images and provide higher quality images, it has been widely used.

[0061] Multiple exposure fusion technology typically involves capturing multiple images of the same object with different exposure times and then fusing them to obtain a single high dynamic range (HDR) image. For example, an image processing device (such as a camera) continuously captures images of long and short exposure frames, or long, medium, and short exposure frames, using an image sensor. Then, different reference frames are selected based on the different exposure intensities for image fusion, ultimately resulting in a fused HDR image. The long exposure frame can be an image captured with a long exposure time, the short exposure frame can be an image captured with a short exposure time, and so on. The long and short exposure times can be predefined.

[0062] Since images from different exposure frames contain a certain amount of noise, global noise reduction is usually performed on the resulting high dynamic range image to improve image quality. However, the noise characteristics of images from different exposure frames vary, resulting in poor noise reduction performance when performing global noise reduction on the fused image, leading to poor image quality after noise reduction.

[0063] In view of this, embodiments of this application provide an image processing device, method, storage medium, and program product. The image processing device analyzes and processes the fused image to determine the fusion ratio matrix of the fused image during the fusion process. Based on the fusion ratio matrix, it determines the reference regions of the images of different exposure frames referenced during the fusion process. Based on the preset noise reduction parameters corresponding to the reference regions, it obtains noise reduction compensation parameters for secondary noise reduction of the fused image. Secondary noise reduction processing is performed on the fused image based on the noise reduction compensation parameters, which can effectively improve image quality.

[0064] The technical solutions of this application will be described in detail below with reference to specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0065] Figure 1 This is a schematic diagram of a scenario provided for an embodiment of this application, such as... Figure 1 As shown, it includes: an image processing device 1 and a target area 2.

[0066] Image processing device 1 may include controller 101 and image sensor 102. Image sensor 102 can capture images of target area 2 at different exposure times, obtaining multiple frames of images with different exposure times, and then fuse these multiple frames of images with different exposure times to obtain a fused image of target area 2. Image sensor 102 can send the obtained fused image to controller 101.

[0067] The controller 101 can perform global noise reduction on the received fused image. When it is determined that the noise reduction effect is not good, it can analyze and process the fused image to obtain noise reduction compensation parameters for secondary noise reduction. Based on the noise reduction compensation parameters, it can perform secondary noise reduction on the fused image after global noise reduction to improve image quality.

[0068] In some embodiments, the image processing device 1 may further include a memory 103, in which the image sensor 102 can store the fused image of the acquired target area. The controller 101 can retrieve the fused image of the acquired target area from the memory, perform global noise reduction processing on the fused image of the target area, and determine whether to perform secondary noise reduction processing.

[0069] In some embodiments, the memory 103 may be deployed inside the image processing device 1 or outside the image processing device 1, and may be connected to the image processing device 1 via a wired or wireless data interface.

[0070] The following is about Figure 1 The possible implementations of the image processing device shown are described below:

[0071] In some embodiments, the image processing device includes a controller and an image sensor integrated together, such as Figure 2A The image processing device shown includes a controller and sensors. This device can independently adjust white balance parameters, for example, in a camera with data processing capabilities in the field of surveillance.

[0072] In some embodiments, the image processing device includes an image sensor deployed at the front end and a controller deployed at the back end, with the image sensor and controller connected via wired or wireless means, for example... Figure 2B The image processing device shown has an image sensor deployed in the target area and a controller deployed on a backend server. This server can be a local server or a cloud server.

[0073] In some embodiments, the image processing device may include an image sensor and a controller that are integrated into another device, for example... Figure 2CThe display device shown can have an image sensor and a controller. For example, in a conference setting, the image sensor could be a camera integrated into the conference screen, and the controller could be the central controller of the conference screen.

[0074] In some embodiments, the image processing device may include an image sensor and a controller that are deployed in different devices, such as... Figure 2D As shown, the controller included in the image processing device can be deployed within the display device, and the image sensor included in the image processing device can be a standalone device, or the image sensor can be deployed within a standalone device. The image sensor can be connected to the display device via wired or wireless means.

[0075] It should be understood that the above are merely schematic diagrams of possible forms of image processing devices. The forms of image processing devices may also include other forms, and the embodiments of this application do not limit them.

[0076] The following example illustrates the integration of image processing equipment into a display device (e.g., Figure 2C Taking the example shown, we will introduce a possible hardware configuration for a display device.

[0077] Figure 3 This is a schematic diagram illustrating a possible hardware configuration of a display device 200 provided in this application. (See diagram below.) Figure 3 As shown, in some embodiments, the display device 200 may include at least one of the following: a tuner 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a power supply 280, a memory 290, and a user interface 2100.

[0078] In some embodiments, the controller includes a processor, a video processor, an audio processor, a graphics processor, RAM, ROM, and a first interface to an nth interface for input / output.

[0079] The display 260 includes a display screen assembly for presenting images, a driving assembly for driving image display, a component for receiving image signals from the controller output, and a user control UI interface for displaying video content, image content, menu control interface, and user control UI interface.

[0080] The display 260 may be a liquid crystal display, an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Miniled LED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), and a projection display, etc., and may also be a projection device and a projection screen.

[0081] The communicator 220 is a component used to communicate with external devices or servers according to various communication protocol types. For example, the communicator may include at least one of the following: a WiFi module, a Bluetooth module, a wired Ethernet module, other network communication protocol chips or near-field communication protocol chips, and an infrared receiver. The display device 200 can establish the transmission and reception of control signals and data signals with the external control device 100 or the server 400 through the communicator 220.

[0082] User interface 2100 can be used to receive control signals from control device 100 (such as infrared remote control).

[0083] Detector 230 is used to collect signals from the external environment or to interact with the external environment. For example, detector 230 includes a light receiver, a sensor for collecting ambient light intensity; or, detector 230 includes an image acquisition device, such as a camera, which can be used to collect external environmental scenes, user attributes, or user interaction gestures; or, detector 230 includes a sound acquisition device, such as a microphone, for receiving external sounds.

[0084] The external device interface 240 may include, but is not limited to, one or more of the following: High Definition Multimedia Interface (HDMI), analog or high-definition component input interface (component), composite video input interface (CVBS), USB input interface (USB), RGB port, etc. It may also be a composite input / output interface formed by multiple interfaces mentioned above.

[0085] The tuner / demodulator 210 receives broadcast television signals via wired or wireless means, and demodulates audio and video signals, such as EPG data signals, from multiple wireless or wired broadcast television signals.

[0086] In some embodiments, the controller 250 and the tuner 210 may be located in different separate devices, that is, the tuner 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0087] The controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in the memory 290. The controller 250 controls the overall operation of the display device 200. For example, in response to receiving a user command to select a UI object to display on the monitor 260, the controller 250 can execute operations related to the object selected by the user command.

[0088] In some embodiments, the controller includes at least one of a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), RAM (random access memory), ROM (read-only memory), a first to an nth interface for input / output, a communication bus, etc.

[0089] Users can input commands through a graphical user interface (GUI) displayed on the monitor 260, and the user input interface receives the user input commands through the GUI. Alternatively, users can input commands by entering specific sounds or gestures, and the user input interface receives the user input commands by recognizing the sounds or gestures through sensors.

[0090] A "user interface" is the medium through which an application or operating system interacts and exchanges information with the user. It converts information from its internal form to a form that the user can accept. A common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed graphically. It can be an icon, window, control, or other interface element displayed on the screen of an electronic device. Controls can include visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets.

[0091] Below Figure 1-3 Based on the image processing device shown, the technical solutions provided in the embodiments of this application are described with the controller of the image processing device as the execution subject.

[0092] Figure 4 A flowchart illustrating the image processing method provided in the embodiments of this application. Figure 1 ,like Figure 4 As shown, it includes the following steps:

[0093] S401. Acquire a target image, wherein the target image is an image obtained by fusing multiple frames of images acquired by the image sensor under different exposure parameters; the multiple frames of images are images acquired from the same scene.

[0094] In some embodiments, exposure parameters may refer to parameters associated with exposure, such as exposure duration, exposure gain, etc.

[0095] In some embodiments, the controller can control the exposure of the image sensor, setting different exposure parameters and image acquisition modes. For example, image acquisition modes may include multiple exposure modes and normal exposure modes.

[0096] In multiple exposure mode, the image sensor can acquire multiple frames of images based on different exposure parameters, fuse these multiple frames to obtain the target image, and transmit the target image to the controller. For example, taking exposure time as the exposure parameter and acquiring two frames, acquiring multiple frames with different exposure parameters can result in long-exposure frames acquired with long exposure times and short-exposure frames acquired with short exposure times. The process of fusing multiple frames to obtain the target image can refer to existing image fusion methods, and will not be elaborated further in this embodiment.

[0097] In some embodiments, the image sensor can store the acquired target image in a memory, and the controller can retrieve the target image from the memory.

[0098] In some embodiments, the memory may also store images captured by the image sensor in normal mode. When the controller retrieves an image from the memory, it can determine, based on the image's attribute information, that the image is a target image obtained by fusing multiple frames acquired under different exposure parameters before executing subsequent steps. The image's attribute information includes the image's acquisition mode and acquisition parameters. For example, when the image is a target image, the attribute information may include information such as the acquisition mode being a multiple exposure mode, the image being obtained by fusing N frames, and the exposure parameters of each frame.

[0099] S402. Process the target image to determine the fusion ratio matrix of the target image; the fusion ratio matrix is ​​used to fuse pixels at the same position in multiple frames of images acquired under different exposure parameters.

[0100] In some embodiments, when fusing multiple frames of images, pixels at the same position in the multiple frames can be fused, for example, the color value (RGB value), brightness value, grayscale value, etc. corresponding to each pixel can be fused.

[0101] Taking the fusion of long (L) exposure frame images and short (S) exposure frame images as an example, such as Figure 5 As shown, each square can represent a pixel. The image sensor can calculate the fusion weight of each pixel in the long (L) exposure frame image and the short (S) exposure frame image based on the exposure parameters. Based on the fusion weight, the pixels at the same position in the long (L) exposure frame image and the short (S) exposure frame image are fused to obtain the fused frame image, which is the target image.

[0102] When the controller acquires the target image, it can perform content recognition on the target image. For example, it can use brightness analysis and color analysis to perform content recognition on the target image. Combining the exposure parameters of the long (L) exposure frame image and the short (S) exposure frame image, it can obtain the fusion weight of each pixel when fusing the image. For example, the fusion weight of pixel 1 (the first pixel in the upper left corner) is 1 for the long (L) exposure frame image and 0 for the short (S) exposure frame image. That is, pixel 1 is completely referenced to pixel 1 in the long (L) exposure frame image during the fusion process.

[0103] When the fusion weight of each pixel is obtained, the controller can perform weighted processing on the value of each pixel according to the fusion weight to obtain the fusion ratio matrix.

[0104] For example, the fusion ratio matrix A can be as follows:

[0105]

[0106] Where α and β ∈ [0, 1], L represents that this pixel is completely referenced to a long exposure frame, S represents that this pixel is completely referenced to a short exposure frame, α*S+(1-α)*L represents that this pixel is partially referenced to a short exposure frame with a reference ratio of α, and partially referenced to a long exposure frame with a reference ratio of (1-α); β*S+(1-β)*L represents that this pixel is partially referenced to a short exposure frame with a reference ratio of β, and partially referenced to a long exposure frame with a reference ratio of (1-β).

[0107] S403. Based on the fusion ratio matrix and preset noise reduction parameters, obtain at least one noise reduction compensation parameter for the target image.

[0108] In some embodiments, when the controller obtains the fusion ratio matrix, it can split the fusion ratio matrix based on the exposure parameters to obtain reference matrices corresponding to different exposure parameters.

[0109] For example, if the fusion ratio matrix is ​​determined based on long (L) exposure frame images and short (S) exposure frame images, then by splitting the fusion ratio matrix based on long (L) exposure frames and short (S) exposure frames, a reference matrix for long (L) exposure frames and a reference matrix for short (S) exposure frames can be obtained.

[0110] For example, the reference matrix for a long (L) exposure frame can be as follows:

[0111]

[0112] The reference matrix for short (S) exposure frames can be shown below:

[0113]

[0114] In some embodiments, the multiple exposure modes of the image sensor include multiple exposure modes, such as long exposure mode and short exposure mode. The exposure parameters of different exposure modes are different, and different exposure modes are set with corresponding preset noise reduction parameters. For example, the long exposure mode (long exposure frame) corresponds to the first preset noise reduction parameter NR_L, and the short exposure mode (short exposure frame) corresponds to the second preset noise reduction parameter NR_S.

[0115] The controller can obtain the corresponding preset noise reduction parameters based on the exposure mode to which the exposure parameters belong. Taking the exposure time as an example, if the exposure time belongs to the long exposure mode, the corresponding preset noise reduction parameter is the first preset noise reduction parameter NR_L; if the exposure time belongs to the short exposure mode, the corresponding preset noise reduction parameter is the second preset noise reduction parameter NR_S.

[0116] In some embodiments, the preset noise reduction parameters can be determined based on experimental calibration or set based on prior knowledge, and this application does not limit this.

[0117] In some embodiments, when the reference matrix corresponding to different exposure parameters and the preset noise reduction parameters are obtained, the reference matrix and the preset noise reduction parameters can be multiplied to obtain the noise reduction compensation parameters.

[0118] For example, the noise reduction compensation parameters for long exposure frames are as follows:

[0119] ΔNR-L=AL*NR-L

[0120] The noise reduction compensation parameters for short exposure frames are as follows:

[0121] ΔNR-S=AL*NR-S

[0122] In some embodiments, the global noise reduction compensation parameters for the target image can be as follows:

[0123] ΔNR-ALL=ΔNR-L+ΔNR-S

[0124] S404. Perform noise reduction processing on the target image after global noise reduction based on at least one of the noise reduction compensation parameters.

[0125] In some embodiments, after the controller performs global noise reduction on the target image, if it determines, based on the noise reduction result, that the location of the reference long-exposure frame image region in the globally denoised target image needs to be further denoised, then the noise reduction compensation parameter corresponding to the long-exposure frame is used to perform secondary noise reduction on the target image. If it determines, based on the noise reduction result, that the location of the reference short-exposure frame image region in the globally denoised target image needs to be further denoised, then the noise reduction compensation parameter corresponding to the short-exposure frame is used to perform secondary noise reduction on the target image. If all regions need to be further denoised, then the global noise reduction compensation parameter can be used to perform secondary noise reduction on the target image.

[0126] Since the value of the pixel position referenced in A_L is 0, the noise reduction parameter after multiplying with NR_L is also 0. That is, ΔNR_L only applies to some pixels that reference the long exposure frame. In other words, if the position of the image area referenced by the long exposure frame needs to be denoised, using ΔNR_L to perform secondary noise reduction on the target image can achieve noise reduction of the target image without affecting other pixels, effectively improving the quality of the target image. The same principle applies to other areas.

[0127] The image processing method provided in this application involves acquiring a target image, which is an image obtained by fusing multiple frames of images acquired by the image sensor under different exposure parameters; the multiple frames of images are images acquired from the same scene.

[0128] The target image is processed to determine a fusion ratio matrix. This fusion ratio matrix is ​​used to fuse pixels at the same location from multiple frames acquired under different exposure parameters. Based on the fusion ratio matrix and preset noise reduction parameters, at least one noise reduction compensation parameter for the target image is obtained. Noise reduction processing is then performed on the globally denoised target image based on at least one of these noise reduction compensation parameters. Based on the preset noise reduction parameters corresponding to the reference region, noise reduction compensation parameters are obtained for secondary noise reduction of the fused image. Performing secondary noise reduction processing on the fused image based on these noise reduction compensation parameters can effectively improve image quality.

[0129] Based on the above embodiments, the following is combined with Figure 6 The image processing method provided in the embodiments of this application will be further described.

[0130] Figure 6 The second schematic flowchart of the image processing method provided in the embodiments of this application is as follows: Figure 6 As shown, it includes the following steps:

[0131] S601. Obtain the target image.

[0132] In some embodiments, the controller can acquire a target image sent by an image sensor, which is a target image obtained by fusing multiple frames of images with different exposure parameters acquired continuously in a multiple exposure mode.

[0133] In some embodiments, the controller can retrieve the target image from the memory. In this manner, the controller can determine whether the target image is an image fused with different exposure parameters based on the attribute information of the target image. If yes, the controller can execute the subsequent steps of this application embodiment; if no, the controller can execute the steps shown in S602 and then end the process.

[0134] S602. Perform global noise reduction processing on the target image.

[0135] In some embodiments, the controller may employ a preset noise reduction algorithm to perform global noise reduction on the target image, such as a filter-based noise reduction algorithm, a neural network model-based noise reduction algorithm, or a deep learning-based noise reduction algorithm. This application does not limit the scope of this embodiment.

[0136] S603. Determine whether the target image after global denoising needs denoising compensation. If yes, proceed with the steps shown in S604. If no, the process ends.

[0137] In some embodiments, the controller can perform noise distribution statistics on the target image after global denoising, and also perform noise distribution statistics on the target image before global denoising. Based on the noise distribution before and after global denoising, it determines whether the global denoising meets the requirements. For example, the controller can use an image quality evaluation algorithm to evaluate the denoising effect based on the noise distribution before and after denoising. If the evaluation index output by the image quality evaluation algorithm is less than or equal to a preset value, it indicates that the target image does not need denoising compensation; if the evaluation index output by the image quality evaluation algorithm is greater than the preset value, it indicates that the target image needs denoising compensation.

[0138] In some embodiments, to further improve the accuracy of determining whether the target image after global denoising needs noise reduction compensation, the controller can partition the target image into regions of a preset size to determine whether the target image after global denoising needs noise reduction compensation. For example, each region of each target image can be evaluated using an image quality assessment algorithm. If the evaluation index of a region is greater than a preset value, it indicates that the target image needs noise reduction compensation.

[0139] S604. Obtain the target region in the target image that needs noise reduction compensation.

[0140] In some embodiments, if noise reduction compensation is determined by partition recognition in S603, the controller can take the region in the target image where the evaluation index is greater than the preset value as the target region.

[0141] In some embodiments, if noise compensation is determined by global identification in S603, the controller can also use the noise distribution of the target image after global noise reduction as the target area, and the area with noise greater than a threshold as the target area.

[0142] S605. Process the target image to determine the fusion ratio matrix of the target image.

[0143] S606. The fusion ratio matrix is ​​split to obtain reference matrices corresponding to different exposure parameters.

[0144] S607. Based on the reference matrix and preset noise reduction parameters, obtain the noise reduction compensation parameters corresponding to different exposure parameters.

[0145] In some embodiments, the implementation of steps S605-S607 can be referred to Figure 4 The specific implementation methods in the illustrated embodiments will not be described in detail here.

[0146] In some embodiments, different exposure modes each correspond to a preset noise reduction parameter, such as Figure 3 The target image in the illustrated embodiment is an example of the fusion of long-exposure frame images and short-exposure frame images. The noise reduction compensation parameters corresponding to the long-exposure frame and the short-exposure frame are as follows:

[0147] ΔNR-L=AL*NR-L

[0148] ΔNR-S=AL*NR-S

[0149] ΔNR-ALL=ΔNR-L+ΔNR-S

[0150] In some embodiments, different exposure modes can correspond to multiple preset noise reduction parameters. For example, a long exposure mode corresponds to preset noise reduction parameter 1 (NR-L1), preset noise reduction parameter 2 (NR-L2), and preset noise reduction parameter 3 (NR-L3); a short exposure mode corresponds to preset noise reduction parameter 4 (NR-L4), preset noise reduction parameter 5 (NR-L5), and preset noise reduction parameter 6 (NR-L6). Each preset noise reduction parameter corresponds to an exposure parameter range; for example, preset noise reduction parameter 1 corresponds to exposure parameter range 1.

[0151] In this case, the controller can obtain preset noise reduction parameters corresponding to different exposure parameters based on the parameter range corresponding to different exposure parameters. Taking exposure duration as an example, if the exposure duration belongs to the exposure duration corresponding to the long exposure mode, the controller determines the exposure duration range to which the exposure duration belongs in the long exposure mode based on the exposure duration. According to the mapping relationship between the exposure duration range and the preset noise reduction parameters, the corresponding preset exposure parameter is determined. For example, if the exposure duration belongs to exposure duration range 1, the corresponding preset noise reduction parameter is preset noise reduction parameter 1 (NR-L1).

[0152] Correspondingly, the noise reduction compensation parameters for long exposure frames can be:

[0153] ΔNR-L=AL*NR-L1

[0154] It should be understood that the execution order of steps S604 and S605-S607 can be: S604 is executed first, followed by S605-S607; or S605-S607 is executed first, followed by S604; or S604 and S605-S607 are executed simultaneously. This application embodiment does not limit this.

[0155] S608. Determine the noise reduction compensation parameters corresponding to the target area.

[0156] In some embodiments, when determining a target region, the controller may determine the noise reduction compensation parameters corresponding to the target region based on the fusion weights of the pixels in the target region.

[0157] For example, a reference image frame for the target region during the fusion process is determined based on the reference matrix of the target region; the noise reduction compensation parameters corresponding to the reference image frame are used as the noise reduction compensation parameters corresponding to the target region.

[0158] The reference matrix for the target region can be constructed from the fusion weights of each pixel in the target region. For example, if the target region comprises M*N pixels, the fusion weights of pixels at the same position in the fusion ratio matrix are extracted based on the position of each pixel to obtain the reference matrix for the target region.

[0159] Taking the target image as an image fused from long and short exposure frames as an example, after determining the reference matrix of the target region, the reference matrix of the target region can be matched with the reference matrices of the long exposure frames and the short exposure frames. If the matching degree (e.g., similarity) between the reference matrix of the target region and the reference matrix of the long exposure frames is greater than or equal to a preset threshold, it can be said that the target region mainly references the long exposure frame image during the fusion process. If the matching degree (e.g., similarity) between the reference matrix of the target region and the reference matrix of the short exposure frames is greater than or equal to a preset threshold, it can be said that the target region mainly references the short exposure frame image during the fusion process. If the matching degree (e.g., similarity) between the reference matrix of the target region and the reference matrix of the long exposure frames is less than a preset threshold, and the matching degree (e.g., similarity) between the reference matrix of the target region and the reference matrix of the short exposure frames is less than a preset threshold, it can be said that the target region references both the short exposure frame image and the long exposure frame image during the fusion process.

[0160] In some embodiments, if the target region primarily references a long-exposure frame image during the fusion process, the noise reduction compensation parameter corresponding to the target region can be the noise reduction compensation parameter corresponding to the long-exposure frame (e.g., ΔNR-L). If the target region primarily references a short-exposure frame image during the fusion process, the noise reduction compensation parameter corresponding to the target region can be the noise reduction compensation parameter corresponding to the short-exposure frame (e.g., ΔNR-S). If the target region references both short-exposure and long-exposure frame images during the fusion process, the corresponding noise reduction compensation parameter can be a global noise reduction compensation parameter (e.g., ΔNR-ALL).

[0161] In some embodiments, if the target region references both short-exposure and long-exposure frames during the fusion process, and the ratio of the target region's size to the target image's size is greater than a preset ratio (i.e., the target region is large), the controller can set the corresponding noise reduction compensation parameter to a global noise reduction compensation parameter (e.g., ΔNR-ALL) to indicate the efficiency of determining the noise reduction compensation parameter corresponding to the target region. If the target region references both short-exposure and long-exposure frames during the fusion process, and the ratio of the target region's size to the target image's size is less than a preset ratio (i.e., the target region is small), the controller can correct the reference matrix corresponding to different exposure parameters based on the target region's reference matrix to obtain the corresponding noise reduction compensation parameter.

[0162] For example, the reference matrix for a long exposure frame can be as follows, including all pixels in the target image:

[0163]

[0164] The controller can set all values ​​outside the corresponding positions in the reference matrix of the long exposure frame to 0 based on the reference matrix of the target area, thus obtaining the corrected reference matrix of the long exposure frame. The processing method for the reference matrix of the short exposure frame is similar.

[0165] When determining the reference matrix for the corrected long and short exposure frames, the controller multiplies the corresponding preset noise reduction parameters by the corresponding reference matrix to obtain the noise reduction compensation parameters for each long and short exposure frame. These noise reduction compensation parameters are then summed to obtain the noise reduction compensation parameters for the target region. By using this method, the accuracy of obtaining the noise reduction compensation parameters for the target region can be improved when referencing long and short exposure frame images during the target region fusion process.

[0166] S609. Perform noise reduction processing on the target area according to the noise reduction compensation parameters.

[0167] In some embodiments, when the noise reduction compensation parameters of the target area are obtained, a noise reduction algorithm can be used to perform noise reduction processing on the target area based on the noise reduction compensation parameters.

[0168] The image processing method provided in this application embodiment can perform separate noise reduction processing on different regions of the target image by obtaining the target region to be denoised in the target image and the noise reduction compensation parameters corresponding to the target region. This enables secondary noise reduction of the target image without responding to other regions of the target image when the noise reduction effect in some regions of the target image is not good, thereby improving the image quality.

[0169] In some embodiments, within the controller, acquiring the reference matrix corresponding to different exposure parameters and acquiring the noise reduction compensation parameters can be performed by different modules. Since exposure fusion has pixel continuity, adjacent pixels generally do not experience abrupt reference changes; that is, pixels within a certain region reference either the L-frame or the S-frame. Therefore, when acquiring the reference matrix corresponding to different exposure parameters, only the data at the changed positions in the reference matrix can be recorded to reduce the amount of data transmitted subsequently and improve data transmission efficiency. For example, the reference matrix is ​​shown below:

[0170]

[0171] The reference matrix for the records can be as follows:

[0172]

[0173] The module that obtains the reference matrix sends the recorded reference matrix and the size of the original reference matrix to the module that obtains the noise reduction compensation parameters. The module that obtains the noise reduction compensation parameters can fill the received reference matrix according to the received reference matrix and the size of the original reference matrix to restore the source reference matrix.

[0174] Based on the above embodiments, this application also provides an image processing apparatus.

[0175] Figure 7 This is a schematic diagram of the structure of the image processing apparatus 70 provided in the embodiments of this application, as shown below. Figure 7 As shown, it includes:

[0176] The acquisition module 701 is used to acquire a target image, wherein the target image is an image obtained by fusing multiple frames of images acquired under different exposure parameters; the multiple frames of images are images acquired from the same scene.

[0177] The first processing module 702 is used to process the target image and determine the fusion ratio matrix of the target image; the fusion ratio matrix is ​​used to fuse pixels at the same position in multiple frames of images acquired under different exposure parameters.

[0178] The second processing module 703 is used to obtain at least one noise reduction compensation parameter of the target image based on the fusion ratio matrix and preset noise reduction parameters.

[0179] The noise reduction module 704 is used to perform noise reduction processing on the globally denoised target image based on at least one of the noise reduction compensation parameters.

[0180] In some embodiments, the first processing module 702 is further configured to perform content recognition on the target image according to the exposure parameters, and obtain the fusion weights of pixels at the same position in the multi-frame images under different exposure parameters; and to perform weighted processing on the values ​​of the pixels according to the fusion weights to obtain the fusion ratio matrix.

[0181] In some embodiments, the second processing module 703 is further configured to split the fusion ratio matrix to obtain reference matrices corresponding to different exposure parameters; obtain preset noise reduction parameters corresponding to the exposure modes to which different exposure parameters belong; different exposure modes correspond to one or more preset noise reduction parameters; and perform a product operation on the reference matrix and the preset noise reduction parameters to obtain the noise reduction compensation parameters.

[0182] In some embodiments, the second processing module 703 is further configured to determine the exposure duration range to which the exposure duration belongs; and to obtain the preset noise reduction parameters based on the exposure duration range and the mapping relationship between the exposure duration range and the preset noise reduction parameters.

[0183] In some embodiments, the noise reduction module 704 is further configured to obtain the target region in the target image that needs to be noise-reduced, and the noise reduction compensation parameters corresponding to the target region; and to perform noise reduction processing on the target region according to the noise reduction compensation parameters.

[0184] In some embodiments, the second processing module 703 is further configured to determine a reference image frame of the target region during the fusion process based on the reference matrix of the target region; and to use the noise reduction compensation parameter corresponding to the reference image frame as the noise reduction compensation parameter corresponding to the target region.

[0185] In some embodiments, the noise reduction module 704 is further configured to perform global noise reduction on the target image; based on the noise distribution of the target image before global noise reduction and the noise distribution of the target image after global noise reduction, the target image is partitioned for identification; if it is determined that there is a target region to be noise-reduced and compensated, the step of obtaining the fusion ratio matrix is ​​executed.

[0186] The image processing apparatus provided in this application is used to execute the image processing method provided in any of the foregoing embodiments. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0187] It should be noted that the division of the various modules in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. These modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented in software via processing element calls, while others are implemented in hardware. Each module can be a separate processing element, or it can be integrated into a chip within the device. Alternatively, it can be stored as program code in the device's memory, and its functions can be called and executed by a processing element. Furthermore, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. During implementation, each step of the above method or each of the above modules can be completed through integrated logic circuits in the processor element or through software instructions.

[0188] This application also provides a computer-readable storage medium, which may include various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk. Specifically, the computer-readable storage medium stores program instructions, which are used in the methods described in the above embodiments.

[0189] This application also provides a program product including executable instructions stored in a readable storage medium. At least one control module of an image processing apparatus can read the executable instructions from the readable storage medium, and the at least one control module executes the executable instructions to cause the image processing apparatus to implement the image processing methods provided in the various embodiments described above.

[0190] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

[0191] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be obtained based on the above teachings. The selection and description of the above embodiments are for the purpose of better explaining the principles and practical applications, thereby enabling those skilled in the art to better utilize the described embodiments and various different variations of embodiments suitable for specific use considerations.

Claims

1. An image processing device, characterized in that, The image processing device includes: an image sensor and a controller connected to the image sensor, the controller being configured to; Acquire a target image, which is an image obtained by fusing multiple frames of images acquired by the image sensor under different exposure parameters; the multiple frames of images are images acquired from the same scene; The target image is processed to determine the fusion ratio matrix of the target image; the fusion ratio matrix is ​​used to fuse pixels at the same position in multiple frames of images acquired under different exposure parameters; Based on the fusion ratio matrix and preset noise reduction parameters, at least one noise reduction compensation parameter of the target image is obtained; The target image after global denoising is denoised based on at least one of the denoising compensation parameters.

2. The image processing apparatus according to claim 1, characterized in that, The controller is configured to: Based on the exposure parameters, content recognition is performed on the target image to obtain the fusion weights of pixels at the same position in the multiple frames of images under different exposure parameters. The values ​​of the pixels are weighted according to the fusion weights to obtain the fusion ratio matrix.

3. The image processing apparatus according to claim 2, characterized in that, The controller is configured to: The fusion ratio matrix is ​​split to obtain reference matrices corresponding to different exposure parameters; Obtain the preset noise reduction parameters corresponding to the exposure modes of different exposure parameters; Different exposure modes correspond to one or more preset noise reduction parameters; The noise reduction compensation parameters are obtained by multiplying the reference matrix and the preset noise reduction parameters.

4. The image processing apparatus according to claims 1-3, characterized in that, The exposure parameters include the exposure duration, and the controller is configured to: Determine the exposure duration range to which the exposure duration belongs; Based on the exposure duration range and the mapping relationship between the exposure duration range and the preset noise reduction parameters, the preset noise reduction parameters are obtained.

5. The image processing apparatus according to claim 4, characterized in that, The controller is configured to: Obtain the target region in the target image that needs to be denoised, and the denoising compensation parameters corresponding to the target region; The target area is subjected to noise reduction processing based on the noise reduction compensation parameters.

6. The image processing apparatus according to claim 5, characterized in that, The controller is configured to: Based on the reference matrix of the target region, the reference image frame of the target region in the fusion process is determined; The noise reduction compensation parameters corresponding to the reference image frame are used as the noise reduction compensation parameters corresponding to the target region.

7. The image processing apparatus according to claim 1, characterized in that, The controller is configured to: Perform global noise reduction on the target image; Based on the noise distribution of the target image before global denoising and the noise distribution of the target image after global denoising, the target image is partitioned for identification. If it is determined that there is a target region that needs to be denoised and compensated, the step of obtaining the fusion ratio matrix is ​​executed.

8. An image processing method, characterized in that, include: Acquire a target image, which is an image obtained by fusing multiple frames acquired under different exposure parameters; The multiple frames are images captured from the same scene; The target image is processed to determine the fusion ratio matrix of the target image; The fusion ratio matrix is ​​used to fuse pixels at the same position in multiple frames of images acquired under different exposure parameters; Based on the fusion ratio matrix and preset noise reduction parameters, at least one noise reduction compensation parameter of the target image is obtained; The target image after global denoising is denoised based on at least one of the denoising compensation parameters.

9. A storage medium, characterized in that, The storage medium stores computer execution instructions, which, when executed by the controller, are used to implement the method as described in claim 8.

10. A computer program product, characterized in that, It includes a computer program that, when executed by the controller, implements the method of claim 8.

Images