A method suitable for low-power fast-start optimized AISP video denoising

Abstract

The application provides a method suitable for low-power fast-start optimization AISP video denoising, comprising the following steps: S1. light measurement switching; S2. initial EV setting; S3. kernel preprocessing; S4. obtaining pre-sequencing fusion weight information; and S5. model input and processing. The initial EV obtained by using the method enables AE to quickly converge to stability. The preprocessing of the kernel can enable the AISP 3D of the application layer to achieve good effects in the first frame, and avoid the problems of frame loss, fast start and timeliness.

Description

Technical Field

[0001] This invention belongs to the field of video noise reduction technology, and specifically relates to a method for low-power, fast-starting optimized AISP video noise reduction. Background Technology

[0002] When processing video and images, noise is often present in the images due to the characteristics of sensors and the influence of devices and the environment during signal transmission and processing, especially in low-light scenes. Videos and images acquired with high gain often contain a large amount of noise. AISP is a popular approach for handling such scenarios, but for noisy scenes, temporal denoising always requires processing several frames to achieve the desired effect. Since the model needs to be executed on the NPU, under normal circumstances, the model can only start after passing through uboot and the application layer of the kernel. Therefore, the time it takes for the model to converge to a stable state will be very long, which cannot meet the fast startup requirements of battery products.

[0003] The video denoising network used is recursively based on the principle of exponential moving average, that is, it processes one frame at a time. The model simultaneously inputs two frames, one reference frame and one current frame.

[0004] However, the existing technology has the following drawbacks:

[0005] In noisy scenes, temporal denoising always requires processing several frames to achieve the desired effect. Since the model needs to be executed on the NPU, under normal circumstances, the model can only start after passing through uboot and the application layer of the kernel. At this point, the time it takes for the model to converge to a stable state will be very long, which cannot meet the fast start-up requirements of battery products. It often takes tens of frames to converge. In extremely dark scenes, the frame rate is often reduced to improve image brightness and signal-to-noise ratio, which will take even longer, making the solution unfeasible.

[0006] In addition, the technical terms in this field include:

[0007] Deep learning is a new research direction in the field of machine learning (ML), which brings it closer to the original goal—artificial intelligence (AI). Deep learning learns the inherent patterns and hierarchical representations of sample data. The information gained during this learning process greatly aids in interpreting data such as text, images, and sound. Its ultimate goal is to enable machines to possess analytical and learning capabilities like humans, capable of recognizing data such as text, images, and sound. Deep learning is a complex machine learning algorithm that has achieved results in speech and image recognition far exceeding previous related technologies.

[0008] Low-power fast start-up: Low-power fast start-up technology refers to technologies that enable rapid startup while ensuring low power consumption. This technology is particularly suitable for devices that need to operate for extended periods and require fast startup speeds, such as wireless battery cameras. Summary of the Invention

[0009] To address the aforementioned issues, the purpose of this application is to improve the usability of the first frame output image of the model while ensuring fast startup speed, thereby optimizing the overall performance.

[0010] Specifically, the present invention provides a method for low-power fast-start optimized AISP video denoising, the method comprising the following steps:

[0011] S1. Metering Switch: Determine whether to enable AISP mode based on the current lighting conditions. When AISP mode is enabled in a low-light scene, the low-light scene can be considered as a scene with illumination below 0.1 lux, depending on the specific application requirements.

[0012] S2. Initial EV Settings: When AISP mode is enabled, set the exposure line and analog gain to the highest values ​​in the current fixed frame rate mode to make AE converge quickly to a stable exposure;

[0013] S3. Kernel preprocessing: The data sequence of kernel dropped frames is used for weighted averaging to generate a reference frame image with a relatively clean static area and the corresponding motion weight information, which is used as the input of the model;

[0014] S4. Obtaining the preceding fusion weight information: The number of frames currently being processed is counted by a counter, and the fusion weight is calculated based on the number of frames, where the weight of the nth frame is 1 / n;

[0015] S5. Model Input and Processing: Input the reference frame image, motion weight information, and current frame obtained in steps S2 and S3 into the model, perform denoising processing, and output the results.

[0016] In step S1, the light is first measured to determine whether the scene should be enabled by AISP. This method is based on the premise that the scene requires the use of AISP. For battery products, AISP is used in low-light scenes.

[0017] In step S2, the scene exposure line of the application is raised to the maximum according to the frame rate, and the analog gain is also at the maximum; assuming that the Bayer domain AI3D denoising model of the application is placed at the front of the entire ISP pipeline, the digital gain does not affect the model itself.

[0018] When switching to AISP mode after metering in the kernel, if the initial EV for AE convergence is high, the maximum exposure line and analog gain are set, then AE can quickly converge to a stable state. Moreover, if it is only convergence in digital gain, it will not affect the 3D effect of AISP at all.

[0019] During the experiment, the exposure line and simulation gain were directly increased to the highest level in the fixed frame rate mode, and the exposure in practice was able to be stabilized quickly.

[0020] Step S3 further includes:

[0021] S3.1, Obtain the current frame image of the video to be denoised and the previous fusion reference frame image in the cache. This is a recursive operation.

[0022] S3.2, Obtain the motion weight information corresponding to the preceding fusion reference frame; that is, the input model also includes the motion weight information (weight) corresponding to the preceding fusion reference frame. t ;

[0023] S3.3, the current frame image and the preceding fused reference frame image are denoised using the exponential moving average formula to obtain the denoised result. The formula is:

[0024] fusion t+1 =ft t weight t+1 +ft t+1 *(1-weight t+1 )

[0025] Here ft t The result of temporal denoising of the previous frame, ft t+1 For the current frame, the image to be denoised, fusion t+1 The deep learning model uses this formula to perform denoising operations on the denoised result;

[0026] S3.4, in kernel mode, average the lost frame data sequence to generate clean data in the static region as the reference frame for the model;

[0027] S3.5, input the reference frame and the corresponding motion weight information into the deep learning model so that the first frame output by the model is the converged result, thereby achieving fast convergence.

[0028] In step S3, the weighted average processing uses the exponential moving average formula, and its specific implementation includes:

[0029] avg_tmp = (src - avg) * weight

[0030] avg = avg_tmp + avg

[0031] avg = np.clip(avg, 0, 1)

[0032] Where src is the current frame image, avg is the reference frame image after weighted averaging, and weight is the fusion weight.

[0033] The fusion weight calculation in step S4 is implemented using the following pseudocode: weight_ind += 1

[0034] Weight = 1. / weight_ind

[0035] Here, weight_ind is a frame counter, indicating which frame is being processed at this time.

[0036] In step S4, the acquisition of the preceding fusion weights is assumed to be for an AI3D model, which has three inputs: two images and one weight. The images are the reference frames obtained in step S2. The current frame is directly output from the sensor. The fusion weights are originally the output of the model when processing the data of the previous frame. However, according to the logic of step S2, it can be deduced that for a simple weighted average of the sequence, the weights can be derived. The weight for two frames is 1 / 2, the weight for three frames is 1 / 3, and so on.

[0037] In step S5, the reference frame image, motion weight information, and the current frame obtained in steps S2 and S3 are input into the AI3D model, which can then iteratively output the denoised data.

[0038] Therefore, this solution has the following advantages:

[0039] 1. The initial EV causes AE to converge quickly to a stable state.

[0040] 2. Kernel preprocessing enables the application layer AISP3D to achieve better results in the first frame, avoiding frame drops and fast-start timeliness issues. Attached Figure Description

[0041] The accompanying drawings, which are provided to further illustrate the invention and form part of this application, are not intended to limit the scope of the invention.

[0042] Figure 1 This is a schematic diagram of the method in this application.

[0043] Figures 2(1) and 2(2) are schematic diagrams of the effect of the first frame outflow image before and after using this method in a specific embodiment of this application. Detailed Implementation

[0044] To better understand the technical content and advantages of the present invention, the present invention will now be described in further detail with reference to the accompanying drawings.

[0045] This invention relates to the technical field of improving the noise reduction effect of AISP3D in battery-powered camera products. Specifically, it is a solution that preprocesses the data before starting AISP under limited conditions, so that the data can quickly converge to a better effect after the model is officially started.

[0046] This invention proposes a method for low-power, fast-start optimized AISP video denoising. The specific implementation principle is explained in detail below. Figure 1 This is a flowchart, including:

[0047] S1. Metering Switch: Determine whether to enable AISP mode based on the current lighting conditions. When AISP mode is enabled in a low-light scene, the low-light scene can be considered as a scene with illumination below 0.1 lux, depending on the specific application requirements.

[0048] S2. Initial EV Settings: When AISP mode is enabled, set the exposure line and analog gain to the highest values ​​in the current fixed frame rate mode to make AE converge quickly to a stable exposure;

[0049] S3. Kernel preprocessing: The data sequence of kernel dropped frames is used for weighted averaging to generate a reference frame image with a relatively clean static area and the corresponding motion weight information, which is used as the input of the model;

[0050] S4. Obtaining the preceding fusion weight information: The number of frames currently being processed is counted by a counter, and the fusion weight is calculated based on the number of frames, where the weight of the nth frame is 1 / n;

[0051] S5. Model Input and Processing: Input the reference frame image, motion weight information, and current frame obtained in steps S2 and S3 into the model, perform denoising processing, and output the results.

[0052] The specific implementation steps of this method are described below:

[0053] Step S1. Metering Switch

[0054] For fast AISP activation in battery-related products, different modes are typically used in different scenarios. For example, a traditional ISP is used in bright light, while AISP is used in low-light scenarios. This is because AISP often requires a certain amount of computing power and has a relatively large load, while traditional ISPs cannot handle low-light scenarios. Therefore, at the very beginning of the entire logic, light metering is used to determine whether AISP should be enabled in the given scenario. The following discussion assumes that the scenario requires the use of AISP.

[0055] Specific instructions are not provided here, as different products have different usage methods.

[0056] Step S2. Initial EV

[0057] Regardless of the 3D optimization logic used, a stable exposure is necessary to stabilize the video stream. Since the model is mainly used for noise reduction in low-light scenes and has an advantage when the noise intensity is high, the exposure line is generally raised to the maximum based on the frame rate, and the analog gain is usually already at its maximum.

[0058] The Bayer domain AI3D denoising model is placed at the very beginning of the entire ISP pipeline, and the digital gain does not affect the model itself. Therefore, when we determine to switch to AISP mode after metering in the kernel, we give the AE convergence an initial EV that is very high, along with the maximum exposure line and analog gain. This allows the AE to converge quickly to a stable state, and if it is only convergence in terms of digital gain, it will not affect the 3D effect of AISP at all.

[0059] During the experiment, the exposure line and simulation gain were directly increased to the highest level in the fixed frame rate mode, and the exposure in practice was quickly stabilized.

[0060] Step S3. Kernel Preprocessing

[0061] The denoising formula for the model is the exponential moving average formula.

[0062] fusion t+1 =ft t weight t+1 +ft t+1 *(1-weight t+1 )

[0063] Here ft t The result of temporal denoising of the previous frame, ft t+1 For the current frame, the image to be denoised, fusion t+1 The result after denoising is used by deep learning models for denoising operations.

[0064] Based on this principle, we know that we first need to obtain the current frame image from the video frame to be denoised, and then obtain the reference frame image after previous fusion from the cache. This is a recursive operation. In addition, the input model also includes the motion weight information corresponding to the reference frame after previous fusion. t In this section, the core factors affecting model convergence stability lie in the reference frame image and the motion weight information. tThe reason why the model cannot converge to the expected value quickly after fast startup is that the reference frame at this time is the current frame. It can be understood that it starts to converge from the first frame and it will take several frames to converge and stabilize. However, at this time, the kernel has actually spawned several frames of data, and this data has been dropped.

[0065] Therefore, a relatively clean reference frame is obtained by utilizing the data from the kernel's dropped frames, along with the corresponding motion weight information. t As input to the model, the first frame output by the application layer is already the result of several converged frames, achieving the goal of fast convergence. The cleaner reference frame here is based on the logic of exponential moving average for temporal denoising. Theoretically, even using just one more frame would yield better results. In the embodiment, to complete the fast start-up within 1 second, about 5 frames were run in the kernel.

[0066] According to the formula above, for static scenes, the noise is different in each frame and fluctuates in the time series. However, after weighted averaging, the temporal noise is essentially zero.

[0067] I i =f i +N i

[0068] Here, I represents the noisy image, f represents the clean data, and N represents the noise. For static scenes,

[0069]

[0070] This is why temporal noise can be removed, and this is also the principle behind various temporal denoising models.

[0071] The simple approach is to average the data sequences that should have been dropped in the kernel state to obtain cleaner data in the static area as the reference frame for the model. For moving scenes, since the AI ​​model itself has the ability to recognize motion, the moving objects accumulated in the kernel will not cause serious artifacts in the output of the AI ​​model.

[0072] The Python code example is as follows:

[0073] avg_tmp = (src - avg) * weight

[0074] avg = avg_tmp + avg

[0075] avg = np.clip(avg, 0, 1)

[0076] The pseudocode here implements the logic of processing one frame at a time. The method for obtaining the weight is explained in step S4. Because the specific number of frames to be executed depends on the actual situation, this interface should be used as needed. In practice, executing 5 frames at a fixed frame rate of 10 frames per second can complete the fast-forward within one second.

[0077] Step S4. Pre-processing weight information t

[0078] For obtaining the preceding fusion weights, the AI3D model targeted in this invention has three inputs: two images and one weight. The images are the reference frames obtained in step S2, and the current frame is directly output from the sensor. The fusion weights are originally the output of the model when processing the data of the previous frame. However, according to the logic of step S2, for a simple weighted average of the sequence, the weights can be derived: the weight for two frames is 1 / 2, for three frames it is 1 / 3, and so on.

[0079] The following is an example code for Weight.

[0080] weight_ind+=1

[0081] Weight = 1. / weight_ind

[0082] In this pseudocode, weight_ind is a counter, indicating which frame is being processed at this time.

[0083] Step S5: Input Model

[0084] The information obtained in steps S2 and S3, along with the current frame, can be input into the AI3D model to iteratively output the denoised data.

[0085] Figures 2(1) and 2(2) show schematic diagrams of the first frame outflow image before and after using this method in the fast-start logic, respectively. The diagrams clearly demonstrate the difference between the two methods, namely, the significant denoising effect.

[0086] This technical solution can be applied to Ingenic T41 series chips and similar chips.

[0087] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the embodiments of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for low-power, fast-start optimized AISP video denoising, characterized in that, The method includes the following steps: S1. Metering Switch: Determine whether to enable AISP mode based on the current lighting conditions. When AISP mode is enabled in a low-light scene, the low-light scene can be considered as a scene with illumination below 0.1 lux, depending on the specific application requirements. S2. Initial EV Settings: When AISP mode is enabled, set the exposure line and analog gain to the highest values ​​in the current fixed frame rate mode to make AE converge quickly to a stable exposure; S3. Kernel preprocessing: The data sequence of kernel dropped frames is used for weighted averaging to generate a reference frame image with a relatively clean static area and the corresponding motion weight information, which is used as the input of the model; S4. Obtaining the preceding fusion weight information: The number of frames currently being processed is counted by a counter, and the fusion weight is calculated based on the number of frames, where the weight of the nth frame is 1 / n; S5. Model Input and Processing: Input the reference frame image, motion weight information, and current frame obtained in steps S2 and S3 into the model, perform denoising processing, and output the results.

2. The method for low-power, fast-start optimized AISP video denoising according to claim 1, characterized in that, In step S1, the light is first measured to determine whether the scene should be enabled by AISP. This method is based on the premise that the scene requires the use of AISP. For battery products, AISP is used in low-light scenes.

3. The method for low-power, fast-start optimized AISP video denoising according to claim 1, characterized in that, In step S2, the scene exposure line of the application is raised to the maximum according to the frame rate, and the analog gain is also at the maximum; assuming that the Bayer domain AI3D denoising model of the application is placed at the front of the entire ISP pipeline, the digital gain does not affect the model itself. When switching to AISP mode after metering in the kernel, if the initial EV for AE convergence is high, the maximum exposure line and analog gain are set, then AE can quickly converge to a stable state. Moreover, if it is only convergence in digital gain, it will not affect the 3D effect of AISP at all. During the experiment, the exposure line and simulation gain were directly increased to the highest level in the fixed frame rate mode, and the exposure in practice was able to be stabilized quickly.

4. The method for low-power, fast-start optimized AISP video denoising according to claim 3, characterized in that, Step S3 further includes: S3.1, Obtain the current frame image of the video to be denoised and the previous fusion reference frame image in the cache. This is a recursive operation. S3.2, Obtain the motion weight information corresponding to the preceding fusion reference frame; that is, the input model also includes the motion weight information (weight) corresponding to the preceding fusion reference frame. t ; S3.3, the current frame image and the preceding fused reference frame image are denoised using the exponential moving average formula to obtain the denoised result. The formula is: fusion t+1 =ft t *weight t+1 +ft t+1 *(1-weight t+1 ) Here ft t The result of temporal denoising of the previous frame, ft t+1 For the current frame, the image to be denoised, fusion t+1 The deep learning model uses this formula to perform denoising operations on the denoised result; S3.4, in kernel mode, average the lost frame data sequence to generate clean data in the static region as the reference frame for the model; S3.5, input the reference frame and the corresponding motion weight information into the deep learning model so that the first frame output by the model is the converged result, thereby achieving fast convergence.

5. A method for low-power, fast-start optimized AISP video denoising according to claim 4, characterized in that, In step S3, the weighted average processing uses the exponential moving average formula, and its specific implementation includes: avg_tmp = (src - avg) * weight avg = avg_tmp + avg avg = np.clip(avg, 0, 1) Where src is the current frame image, avg is the reference frame image after weighted averaging, and weight is the fusion weight.

6. The method for low-power, fast-start optimized AISP video denoising according to claim 1, characterized in that, The fusion weight calculation in step S4 is implemented using the following pseudocode: weight_ind += 1 Weight = 1. / weight_ind Here, weight_ind is a frame counter, indicating which frame is being processed at this time.

7. A method for low-power, fast-start optimized AISP video denoising according to claim 6, characterized in that, In step S4, the acquisition of the preceding fusion weights is assumed to be for an AI3D model, which has three inputs: two images and one weight. The images are the reference frames obtained in step S2. The current frame is directly output from the sensor. The fusion weights are originally the output of the model when processing the data of the previous frame. However, according to the logic of step S2, it can be deduced that for a simple weighted average of the sequence, the weights can be derived. The weight for two frames is 1 / 2, the weight for three frames is 1 / 3, and so on.

8. A method for low-power, fast-start optimized AISP video denoising according to claim 6, characterized in that, In step S5, the reference frame image, motion weight information, and the current frame obtained in steps S2 and S3 are input into the AI3D model, which can then iteratively output the denoised data.

Images