Noise reduction method, noise reduction device, electronic equipment and computer program product

Abstract

The invention discloses a noise reduction method, a noise reduction device, electronic equipment and a computer program product. The method is applied to the noise reduction system, the noise reduction system comprises a time domain noise reduction module and a space domain noise reduction module, the time domain noise reduction module is realized based on an artificial intelligence model, the space domain noise reduction module is realized based on a preset noise reduction algorithm, and the time domain noise reduction module and the space domain noise reduction module adopt a series structure. The time domain noise reduction module and the space domain noise reduction module are provided with independent control switches. The method comprises the following steps: determining a current noise reduction scene; determining a target noise reduction module in the time domain noise reduction module and the space domain noise reduction module according to the noise reduction scene; and carrying out noise reduction processing on the collected current image frame through the target noise reduction module. According to the scheme of the invention, a good noise reduction effect can be realized in various scenes.

Description

Technical Field

[0001] This application belongs to the field of image processing technology, and in particular relates to a noise reduction method, noise reduction device, electronic device and computer program product. Background Technology

[0002] In imaging systems, the entire process of converting light signals into digital signals inevitably introduces different types of noise. This makes noise reduction (Denoising) one of the most important steps in image signal processing (ISP), directly affecting the overall quality of the final image.

[0003] Currently, noise reduction techniques can be categorized into spatial domain noise reduction and temporal domain noise reduction based on the source of the referenced information. However, each of these two noise reduction techniques has its advantages and disadvantages. If only one noise reduction technique is used, it may not achieve the best noise reduction effect in some specific scenarios; if both noise reduction techniques are used simultaneously, it may lead to increased power consumption and processing time in other specific scenarios. Summary of the Invention

[0004] This application provides a noise reduction method, noise reduction device, electronic device, and computer program product, which can achieve good noise reduction effect in various scenarios.

[0005] Firstly, this application provides a noise reduction method applied to a noise reduction system. The noise reduction system includes a time-domain noise reduction module and a spatial-domain noise reduction module. The time-domain noise reduction module is implemented based on an artificial intelligence model, and the spatial-domain noise reduction module is implemented based on a preset noise reduction algorithm. The time-domain noise reduction module and the spatial-domain noise reduction module are connected in series, and each module has an independent control switch. The noise reduction method includes: Determine the current noise reduction scenario; In the time-domain noise reduction module and the spatial-domain noise reduction module, the target noise reduction module is determined according to the noise reduction scenario; The target noise reduction module performs noise reduction processing on the current image frame that has been acquired.

[0006] Secondly, this application provides a noise reduction device, the noise reduction method being applied to a noise reduction system. The noise reduction system includes a time-domain noise reduction module and a spatial-domain noise reduction module. The time-domain noise reduction module is implemented based on an artificial intelligence model, and the spatial-domain noise reduction module is implemented based on a preset noise reduction algorithm. The time-domain noise reduction module and the spatial-domain noise reduction module are connected in series, and each module has an independent control switch. The noise reduction device includes: The first determining module is used to determine the current noise reduction scenario; The second determining module is used to determine the target noise reduction module based on the noise reduction scenario in the time-domain noise reduction module and the spatial-domain noise reduction module. The noise reduction module is used to perform noise reduction processing on the current image frame acquired by the target noise reduction module.

[0007] Thirdly, this application provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the method described in the first aspect.

[0008] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described in the first aspect above.

[0009] Fifthly, this application provides a computer program product comprising a computer program that, when executed by one or more processors, implements the steps of the method described in the first aspect.

[0010] The advantages of this application compared to existing technologies are as follows: This application simultaneously sets up a temporal noise reduction module based on an artificial intelligence model and a spatial noise reduction module based on a preset algorithm in the noise reduction system, and adopts a series structure with independent control switches for each module. This allows for the intelligent selection of the target noise reduction module for processing based on the current noise reduction scenario. Therefore, on the one hand, it avoids the problem of poor performance of a single noise reduction method in specific scenarios, ensuring noise reduction effect and image quality in different scenarios; on the other hand, the independent control switches avoid the increased power consumption and extended processing time caused by simultaneously activating both modules when unnecessary, thus achieving a good balance between noise reduction effect and resource consumption.

[0011] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a line-of-sight diagram of the noise reduction method provided in the embodiments of this application; Figure 2This is a schematic diagram of the structure of the artificial intelligence model in the temporal domain noise reduction module provided in the embodiments of this application; Figure 3 This is a schematic diagram of the connection structure of the time-domain noise reduction module, the noise level adjustment module, and the spatial noise reduction module provided in the embodiments of this application; Figure 4 This is a structural block diagram of the noise reduction device provided in the embodiments of this application; Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0014] The embodiments of the technical solution of this application will now be described in detail with reference to the accompanying drawings. These embodiments are only used to more clearly illustrate the technical solution of this application and are therefore merely examples, and should not be used to limit the scope of protection of this application.

[0015] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0016] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly indicating the number, specific order, or primary and secondary relationship of the indicated technical features.

[0017] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0018] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0019] In the description of the embodiments of this application, the term "multiple" refers to two or more (including two), unless otherwise expressly and specifically defined.

[0020] This application proposes a noise reduction method. This noise reduction method can be applied to electronic devices equipped with a noise reduction system. To facilitate understanding of the noise reduction method proposed in this application, the noise reduction system is described below: the noise reduction system includes at least two modules: a time-domain noise reduction module and a spatial-domain noise reduction module.

[0021] The temporal noise reduction module is based on an artificial intelligence model. Its working principle can be simply described as follows: by modeling the correlation between adjacent image frames in a video sequence, the artificial intelligence model is used to extract cross-frame temporal features and distinguish noise from real signals, thereby achieving noise reduction at the temporal level.

[0022] The spatial domain noise reduction module is based on a preset noise reduction algorithm. Specifically, spatial domain noise reduction refers to reducing noise by smoothing or enhancing the pixel neighborhood information, relying solely on the spatial distribution characteristics of the current image frame.

[0023] The temporal and spatial denoising modules are connected in a preset order within the denoising system, allowing the same frame of image in the Raw domain to be processed sequentially by both modules. This cascaded structure ensures that the system can call different denoising methods in sequence as needed, thus enabling flexible combination processing.

[0024] Furthermore, the time-domain noise reduction module and the spatial-domain noise reduction module can each be equipped with corresponding control units, meaning that the time-domain noise reduction module and the spatial-domain noise reduction module have independent control switches, allowing them to be turned on or off independently. This switching method avoids the module from operating when a certain noise reduction method is not needed, thereby reducing unnecessary power consumption and processing latency. In this embodiment, the series connection order of the time-domain noise reduction module and the spatial-domain noise reduction module is not limited; the time-domain noise reduction module can be connected first, followed by the spatial-domain noise reduction module; or the spatial-domain noise reduction module can be connected first, followed by the time-domain noise reduction module.

[0025] Based on the above noise reduction system, please refer to Figure 1 , Figure 1 The implementation flow of a noise reduction method applied to electronic devices with noise reduction systems is presented, detailed below: Step 101: Determine the current noise reduction scenario.

[0026] The noise reduction system can analyze the current image acquisition environment to determine the illumination, gain, and / or motion conditions, thereby initially identifying the possible types and intensities of noise in the subsequent images. For example, in low-light scenes, noise mainly manifests as random noise caused by sensor dark current; in high-dynamic scenes, there is significant motion mismatch between video frames. Through scene judgment, the system can provide a basis for selecting the target noise reduction module.

[0027] Step 102: In the time-domain noise reduction module and the spatial-domain noise reduction module, determine the target noise reduction module according to the noise reduction scenario.

[0028] The noise reduction system can select a suitable target noise reduction module based on the current noise reduction scenario. It should be noted that this application does not limit the number of target noise reduction modules. For example, in typical high-illumination and low-gain scenarios, the target noise reduction module may only be a spatial noise reduction module, as spatial noise reduction is sufficient to meet image quality requirements. In this case, the temporal noise reduction module can be turned off, thereby saving power consumption of the noise reduction system. As another example, in scenarios where the image is essentially static, the target noise reduction module may only be a temporal noise reduction module, outputting a high-definition image through temporal noise reduction. Furthermore, in scenarios with low illumination and significant motion, the target noise reduction module can be both a temporal noise reduction module and a spatial noise reduction module, allowing for separate optimization for different regions.

[0029] Step 103: Denoise the acquired current image frame using the target noise reduction module.

[0030] The noise reduction system can input the acquired raw current image frame into the target noise reduction module, which then performs the corresponding noise reduction operation. For example, when the target noise reduction module is a spatial domain noise reduction module, the current image frame is smoothed using a preset noise reduction algorithm. As another example, when the target module is a temporal domain noise reduction module, in addition to the current image frame, historical image frames can also be input into the artificial intelligence model, which then outputs the noise-reduced result. Furthermore, when the target module is a combination of temporal and spatial domain noise reduction modules, a first noise reduction step followed by a second noise reduction step can be performed based on their cascaded structure, thereby further improving the noise reduction effect.

[0031] In some embodiments, as described above, the temporal denoising module can be determined as a target denoising module independently, or it can be determined together with the spatial denoising module. Based on this, when the target denoising module includes the temporal denoising module, the workflow of the temporal denoising module is described as follows: A1 uses an artificial intelligence model to process the first image frame and historical image frames to obtain the weight map of the first image frame output by the artificial intelligence model.

[0032] The first image frame serves as the input to the temporal denoising module, while the historical image frames are previously denoised image frames obtained by the temporal denoising module. Furthermore, the weight map of the first image frame has the same spatial scale as the first image frame, thus allowing for a pixel-by-pixel description of the weight ratio that should be assigned to the current frame or historical frames for the corresponding pixel position; that is, the weight map of the first image frame can be used to describe the weight of each pixel in the first image frame.

[0033] It is understandable that, unlike traditional temporal denoising methods that require matching in the surrounding area of ​​historical image frames, the temporal denoising module proposed in this application can directly compare the differences of the same pixel positions in two frames (the first image frame and the historical image frame): if the difference between the two frames at that position is small, it indicates that the position is close to stillness, and the pixel value of the historical image frame is more reliable and can be assigned greater weight; conversely, if the difference at that position is large, it indicates that there is motion, and more should be relied upon for the pixel value of the current frame, thus assigning greater weight to the current image frame. Through this artificial intelligence model, erroneous matching in high-noise environments can be avoided, and edge jitter caused by mismatches can also be reduced.

[0034] In some examples, the AI ​​model of the temporal denoising module can specifically adopt an N-layer pyramid structure. This N-layer pyramid structure can effectively suppress noise and improve the robustness of the model at different scales through layer-by-layer downsampling and upsampling. Specifically, such as Figure 2 As shown, Figure 2 A schematic diagram of the N-layer pyramid structure of this artificial intelligence model is given. Each layer of the model includes the following units: downsampling unit, frame difference calculation unit, noise model unit, convolution and nonlinear activation unit, and upsampling unit. The functions of each unit are briefly described below: The downsampling unit is used to reduce the resolution of the image input to the unit, thereby reducing the randomness of noise and obtaining a larger receptive field.

[0035] The frame difference calculation unit is used to calculate the difference between the current image frame and the historical image frames at the corresponding level. This difference can directly reflect the strength of the motion.

[0036] The noise model unit is used to model the noise level by combining preset noise model coefficients.

[0037] Convolutional and nonlinear activation units can extract temporal features and enhance nonlinear expressive power through multi-layer convolution and activation operations, and are used to predict the weight map at the corresponding level.

[0038] The upsampling unit is used to upsample the weight map obtained from the top layer to the bottom layer layer by layer, providing reference information for the prediction of the bottom layer.

[0039] In the N-layer pyramid structure of the proposed artificial intelligence model, layer 1 is the bottom layer and layer N is the top layer. The input to the downsampling unit in layer 1 is the original image frame, and the output of the upsampling unit in layer 1 is the final output of the model, i.e., the weight map of the first image frame. This N-layer pyramid structure allows for top-down, layer-by-layer information propagation, ensuring that the results of higher layers are constrained at lower layers, thus guaranteeing that the predictions at lower layers are more robust in terms of noise suppression.

[0040] Furthermore, the noise reduction system can start from the top layer and calculate the weight map corresponding to the first image frame (or its downsampled result) of each layer, and then upsample this weight map as a reference for the next layer. The connection relationships between the units in each layer are described below: The input to the downsampling unit in layer i is the output of the downsampling unit in layer (i-1), where i is a positive integer greater than 1 and not less than N. That is, the input to the downsampling units in layers 2 to N is the output of the downsampling unit in the next lower layer. It can be understood that by downsampling layer by layer, the image resolution gradually decreases, the impact of noise is mitigated, and the perception ability of the artificial intelligence model in a wide range of contexts is enhanced.

[0041] The input to the frame difference calculation unit of layer j is the output of the downsampling unit of layer j, where j is a positive integer greater than 0 and not less than N. That is, the input to the frame difference calculation unit of layers 1 to N (each layer) is the output of the downsampling unit of the same layer. It can be understood that calculating inter-frame differences at higher levels, i.e., at lower resolution, makes it easier to eliminate noise interference and makes motion detection more accurate.

[0042] The input to the noise model unit of layer j is the preset noise model coefficients and the output of the downsampling unit of layer j for the input originating from the first image frame. It can be understood that since the initial input to the artificial intelligence model is the first image frame and historical image frames, these two image frames will be downsampled multiple times by the downsampling unit layer by layer. Therefore, each downsampling unit will have two outputs: one for the input originating from the first image frame and one for the input originating from the historical image frames. Based on this, for the noise model unit, the input to the noise model units of layers 1 to N is the preset noise model coefficients and the output of the downsampling unit of the same layer for the input originating from the first image frame.

[0043] The inputs to the convolutional and nonlinear activation units of layer k are the channel concatenation results of the outputs of the frame difference calculation unit, the noise model unit, and the upsampling unit of layer k+1. The inputs to the convolutional and nonlinear activation units of layer N are the channel concatenation results of the outputs of the frame difference calculation unit and the noise model unit of layer N. The inputs to the upsampling units of layer j are the outputs of the convolutional and nonlinear activation units of layer j, where k is a positive integer less than N. As described earlier, the weight map calculation starts from the top layer. The inputs to the convolutional and nonlinear activation units of layer N are the channel concatenation results of the outputs of the frame difference calculation unit and the noise model unit of layer N. The inputs to the upsampling units of layers 1 to N (each layer) are the outputs of the convolutional and nonlinear activation units of the same layer. The inputs to the convolutional and nonlinear activation units of layers 1 to N-1 are the channel concatenation results of the outputs of the frame difference calculation unit, the noise model unit, and the upsampling unit of a higher layer. It is understandable that channel connectivity enables the fusion of multi-source features, thereby ensuring that the layer considers motion information, noise levels, and high-level global constraints when determining weights, making the weight prediction of the layer more accurate.

[0044] A2, based on the weight map of the first image frame, determine the weight map of the historical image frames.

[0045] It can be understood that, for each pixel's position, the sum of the weight of that position in the first image frame and the weight of that position in the historical image frame is 1; that is, the sum of the weights of the same pixel in the first image frame and the historical image frames is 1, which ensures stability during subsequent weighted fusion. Based on this, after obtaining the weight map of the first image frame, the denoising system can determine the weight map of the historical image frames. The weight map of the historical image frames has the same scale as the historical image frames and is used to describe the weight of each pixel in the historical image frames, that is, the degree of contribution of each pixel.

[0046] A3. Based on the weight map of the first image frame and the weight map of the historical image frames, the first image frame and the historical image frames are weighted and fused to obtain a denoised image that has undergone temporal denoising processing for the first image frame.

[0047] The denoising system can perform weighted fusion of the first image frame and historical image frames according to their respective weight maps to generate a temporally denoised output frame, which is the denoised image after temporal denoising processing of the first image frame. Generally, in static areas, the denoising result is closer to the historical frame, thus effectively utilizing cross-frame redundancy information to reduce noise; in moving areas, the denoising result retains more pixels of the current frame, thus avoiding motion blur caused by over-reliance on historical frames. The denoised image obtained in this step can be input into subsequent modules. For example, if a spatial denoising module is connected in series and is also identified as the target denoising module, the denoised image can be input into the spatial denoising module; in addition, the denoised image can also serve as a new historical frame to participate in the next round of temporal denoising calculation.

[0048] As can be seen, the temporal denoising module proposed in this application can use the pyramid structure in the artificial intelligence model to suppress noise layer by layer, thereby enhancing motion detection; and, under the multi-layer convolution and cross-level constraints of the artificial intelligence model, it can output the final weight map, so that the denoising result can make full use of the historical information of the static area to achieve strong denoising, and can retain the details of the current frame in the moving area, thus improving the overall video image quality.

[0049] In some embodiments, as described above, the spatial domain noise reduction module can be identified as a target noise reduction module independently, or it can be identified together with the temporal domain noise reduction module. Based on this, when the target noise reduction module includes the spatial domain noise reduction module, the workflow of the spatial domain noise reduction module is described as follows: B1, iterate through each pixel of the second image frame to determine the current pixel.

[0050] The second image frame serves as the input to the spatial domain denoising module. The denoising system can traverse each pixel of the second image frame, performing spatial domain denoising processing pixel by pixel to ensure that each pixel receives targeted spatial domain denoising. During the traversal, the pixel currently being processed at a given moment can be referred to as the current pixel.

[0051] B2 determines the noise level of the current block based on the average brightness of the current block where the current pixel is located.

[0052] Here, the current block has a preset first size, and the current block is centered on the current pixel; that is, the current block is a small area centered on the current pixel. In some examples, the size of the current block can be 5×5, or other suitable sizes, which are not limited here.

[0053] The noise reduction system can first calculate the average brightness of all pixels within the current block to characterize the overall brightness level of the current block. Then, based on this average brightness, the current imaging gain, and a pre-calibrated noise curve, the noise level of the current block can be calculated. It is understood that the noise level is introduced in this embodiment to quantify the noise intensity of the current block. Generally, in image signal processing, the noise level has the following functional relationship with brightness and gain: the lower the brightness and the higher the gain, the greater the noise level. In some examples, this noise curve can be modeled as y = kx + b, where x represents the average brightness, k and b represent pre-calibrated gain-related coefficients, and y represents the corresponding noise level.

[0054] Of course, the spatial noise reduction module can further introduce a noise scaling factor parameter to adjust the estimated noise level. In some examples, the noise scaling factor can be a preset fixed value; or, the noise scaling factor can be an indefinite value set by the noise reduction system based on the noise reduction processing of the time-domain noise reduction module. The setting of the noise scaling factor is not limited here.

[0055] B3 determines the weight of each other block based on the noise level of the current block and the brightness difference between the current block and each other block in the preset search range.

[0056] The search range is a preset second size, which is larger than the first size, and the search range is still centered on the current pixel; that is, the search range is a large area centered on the current pixel. In some examples, the size of this search range can be 21×21, or other suitable sizes, which are not limited here. Within this search range, the noise reduction system can find multiple other blocks with the same size as the current block. It should be noted that these other blocks are not arbitrarily determined by a sliding window, but are subject to the following constraint: the center pixel of each other block must be located in the same color channel as the center pixel of the current block.

[0057] In some examples, taking a first size of 5×5 and a second size of 21×21 as an example, since a safety distance of 5×5, or 2 pixels, needs to be left from the boundary of the search range, the center pixel may fall in a 9×9 grid, that is, the center pixel can have 9×9=81 positions; each center pixel can uniquely correspond to a 5×5 block, so 81 blocks can be divided; after removing the current block itself, 80 other blocks can be left in the search range.

[0058] For each other block, the noise reduction system can calculate the absolute value of the difference in brightness between it and the current block at each pixel position (that is, obtain the absolute value of 5×5 brightness differences), and then take the average of the absolute values ​​(that is, take the average of the absolute values ​​of 5×5 brightness differences). The result is the brightness difference between the other block and the current block.

[0059] In some examples, taking a first size of 5×5 and a second size of 21×21 as an example, since there are 80 other blocks, the brightness difference between these 80 other blocks and the current block can be obtained.

[0060] It's understandable that, for any other block, the smaller the brightness difference with the current block, the more similar that other block is to the current block, and its weight can be correspondingly larger; conversely, the larger the brightness difference with the current block, the less similar that other block is to the current block, and its weight can be correspondingly smaller. In addition, the noise level also affects the weight setting, with the aim of making the weight allocation more consistent with the actual noise intensity.

[0061] B4 determines the noise reduction brightness of the current pixel based on the weights of other blocks, the brightness of the center pixels of other blocks, and the brightness of the current pixel.

[0062] The brightness of the center pixel of each other block is weighted according to its corresponding weight; in addition, the current pixel itself also participates in the weighting to avoid over-reliance on other blocks. The final weighted result is the denoised brightness of the current pixel, representing the pixel value of the current pixel after spatial denoising. In this way, the denoising system can effectively smooth noise while preserving image details; especially in edge areas, the weights of blocks with large differences will naturally decrease, thereby avoiding excessive blurring of edges.

[0063] B5, after traversing each pixel of the second image frame, obtains the denoised image of the second image frame based on the denoised brightness of each pixel.

[0064] After all pixels in the second image frame have undergone the aforementioned noise reduction brightness calculation, the noise reduction system obtains a complete denoised image. This denoised image is the output of the spatial domain noise reduction module and can also be used as input for subsequent image processing steps (such as image enhancement), without limitation here. Compared to the first image frame, the noise in this denoised image is significantly reduced, while image structure and details are preserved. It is understood that the above spatial domain noise reduction processing can be implemented in hardware circuitry, with low power consumption, making it suitable for real-time applications.

[0065] It is important to note that when used in conjunction with temporal noise reduction, spatial noise reduction primarily targets residual noise in moving areas to ensure the overall visual consistency of the image, while temporal noise reduction mainly targets stationary areas.

[0066] In some embodiments, to ensure that the weights of each other block accurately reflect the contribution of each other block, step B3 may include: C1 determines the reference value as the product of the noise level of the current block and the preset noise reduction intensity.

[0067] The noise reduction system multiplies the noise level of the current block by a preset noise reduction intensity to obtain a reference value. Here, the noise level of the current block reflects the strength of noise within that block's region; the noise reduction intensity is a preset parameter of the noise reduction system, configurable via registers, allowing for overall adjustment of the noise reduction effort. By multiplying these two values, the noise level and noise reduction intensity are combined to form a benchmark index, or reference value, used to determine subsequent weighting levels.

[0068] C2, multiply the reference value by each of the M preset coefficients to obtain the M thresholds.

[0069] The noise reduction system can further multiply the obtained reference values ​​by M preset coefficients to obtain M thresholds, where M is a positive integer. These M coefficients can be understood as dividing the brightness difference into different levels, thus enabling discretization. Generally, the larger the value of M, i.e., the more coefficients, the finer the weight adjustment can be, but the circuit area and power consumption will increase accordingly. In some examples, M can be 63; of course, coefficients can be added or removed according to the actual situation, which is not limited here.

[0070] For the calculated M thresholds, each threshold is associated with a specific weight, thereby achieving a hierarchical mapping of brightness differences in other blocks; that is, each threshold is associated with a different weight. Generally speaking, smaller thresholds are associated with larger weights, while larger thresholds are associated with smaller weights.

[0071] C3 compares each brightness difference with M thresholds to determine the threshold that is closest to each brightness difference.

[0072] For each other block within the search range, the denoising system compares its brightness difference with the current block against one of the calculated M thresholds. In some examples, a binary search method can be used to reduce the number of actual comparisons. Through these comparisons, the denoising system finds the thresholds that best match each brightness difference. This process ensures that each other block is classified into a suitable threshold range, preventing overly discrete or abrupt weight distributions.

[0073] C4 determines the weight of each other block based on the threshold closest to each brightness difference and the correlation between the threshold and the weight.

[0074] The noise reduction system can determine the weights of other blocks based on the threshold closest to each brightness difference and a pre-defined correlation between the threshold and the weight. Taking any other block as an example, after knowing the threshold closest to the brightness difference between the other block and the current block, the weight corresponding to that closest threshold can be assigned to the other block; that is, the weight corresponding to the closest threshold is the weight of the other block. Thus, if the brightness difference between a certain other block and the current block is small, its closest threshold will also be small, and the other block can be assigned a larger weight; conversely, if the brightness difference between a certain other block and the current block is large, its closest threshold will also be large, and the other block can be assigned a smaller weight.

[0075] This weighting mechanism allows blocks with similar textures to occupy a larger proportion in the spatial domain denoising process, while blocks with significant differences contribute less to the final result, thereby improving the detail preservation of the denoised image (i.e., the denoised image).

[0076] In some embodiments, to ensure that the most similar other blocks and the current block itself occupy a larger proportion in the noise reduction process and to avoid excessive smoothing of the image, step B4 may include: D1 sums the weights of all other blocks, and additionally sums the maximum weight among the weights of all other blocks to obtain the weight sum.

[0077] The noise reduction system can accumulate the weights of all other blocks within the search range; on this basis, an additional maximum weight is added; the result is the weight sum. This weight sum not only reflects the overall contribution of all other blocks, but also enhances the influence of the current block by adding the maximum weight.

[0078] D2 multiplies the brightness of the center pixel of each other block by the weight of the corresponding other block to obtain the weighted brightness of the center pixel of each other block.

[0079] The noise reduction system multiplies the brightness of the center pixel of each other block by the weight of that other block to obtain a weighted brightness value. It can be understood that the brightness of the center pixel, as a representative value of the corresponding other block, effectively reflects the overall texture and lighting characteristics of that other block. By combining it with the corresponding weight, the contribution of other blocks with brightness similar to the current block becomes larger, while the contribution of other blocks with significantly different brightness becomes smaller.

[0080] D3 multiplies the brightness of the current pixel by the maximum weight to obtain the weighted brightness of the current pixel.

[0081] The noise reduction system can also multiply the brightness of the current pixel by the maximum weight corresponding to other blocks to obtain the weighted brightness of the current pixel. It can be understood that by introducing the maximum weight to act on the current pixel alone, it can be ensured that the current pixel itself does not lose its influence in the noise reduction process, thereby avoiding blurring and ghosting caused by excessive averaging of other blocks.

[0082] D4 sums the weighted brightness of the center pixels of each other block, and additionally sums the weighted brightness of the current pixel to obtain the weighted brightness sum.

[0083] Similar to the calculation of the weighted sum, the noise reduction system can accumulate the weighted brightness of each other block and add the weighted brightness of the current pixel to obtain a weighted brightness sum. In this way, the weighted brightness sum not only includes the contributions of other blocks but also retains the information of the current pixel itself.

[0084] D5 determines the quotient of the weighted brightness sum and the weighted sum as the noise-reduced brightness of the current pixel.

[0085] The noise reduction system can ultimately obtain the noise-reduced brightness of the current pixel by dividing the weighted brightness sum by the weight sum. This operation is essentially a normalized weighted averaging process. Through the weight assignment mechanism proposed above, the noise-reduced brightness of the current pixel can be balanced between other blocks in the neighborhood and the pixel itself.

[0086] It is understandable that, based on the various operations of the proposed spatial denoising module, the module can possess the following functions: First, it can calculate the noise level based on the gain and brightness values; second, it can control the weight of other blocks in the space based on the noise level; third, it can perform block-based search and comparison, thereby improving robustness to noise; fourth, when the search range is large enough, it can improve the denoising effect under high gain; and fifth, it can distinguish between flat areas and edge areas in the image and perform targeted denoising processing.

[0087] In some embodiments, the series structure of the temporal denoising module and the spatial denoising module can specifically be that the temporal denoising module comes first, followed by the spatial denoising module. As described above, temporal denoising mainly targets stationary regions, while spatial denoising mainly targets moving regions. In temporal denoising, different regions of the entire image have different weighting weights, resulting in different noise levels after weighting. Since spatial denoising is serially connected after temporal denoising, if spatial denoising uses the same denoising intensity to process different regions of the entire image, it will lead to over-denoising in stationary regions or insufficient denoising in moving regions. Based on this, embodiments of this application also propose a noise level adjustment module, which can output the changes in noise levels in different regions of the entire image based on the weighting of the temporal denoising part, thereby enabling spatial denoising to achieve more targeted processing. Based on this, when the target denoising module includes both a spatial denoising module and a temporal denoising module, step 103 may include: E1 performs temporal denoising on the current image frame using the temporal denoising module to obtain a first denoised image, which is a denoised image that has undergone temporal denoising on the current image frame.

[0088] The denoising system can perform temporal denoising on the current image frame. It's understandable that historical image frames are needed as input to the artificial intelligence model of the temporal denoising module during this process; the denoised image output by the temporal denoising module can be recorded as the first denoised image; of course, the temporal denoising module can also output a weight map of the current image frame. Since the workflow of the temporal denoising module has already been described above, it will not be repeated here.

[0089] E2 processes the current noise level of the current image frame and the historical noise level of historical image frames through the noise level adjustment module to obtain the noise scaling factor.

[0090] Among them, the historical image frame is the denoised image frame obtained by the temporal denoising module in the previous process, which has been described above and will not be repeated here; the historical noise level is the fused noise level obtained by the noise level adjustment module in the previous process.

[0091] For the noise level adjustment module, it can read the current noise level of the current image frame and the historical noise level of historical image frames. The current noise level of the current image frame is obtained by interpolating the block brightness, system gain, and pre-calibrated noise curve of the current frame, as described earlier and will not be repeated here. Subsequently, the noise level adjustment module can perform fusion and calculation processing on these two data points, specifically: E21, based on the weight map of the current image frame, fuses the current noise level and the historical noise level to obtain the fused noise level.

[0092] The noise reduction system performs a weighted fusion of the current noise level and historical noise levels at each pixel location according to the weight map of the current image frame. This fusion logic is similar to that of temporal noise reduction: the more the weights are biased towards the current image frame, the closer the fused noise level is to the current noise level; conversely, if the weights are biased towards historical image frames, the fused noise level is closer to the historical noise level. Through this process, the fused noise level can be obtained for the location of each pixel in the entire image.

[0093] E22 calculates the ratio between the current noise level and the combined noise level to obtain the noise scaling factor.

[0094] In some examples, this ratio calculation can be specifically: fused noise level / current noise level; that is, the noise reduction system can perform a pixel-by-pixel division between the fused noise level and the current noise level. Of course, other methods can also be used to calculate the ratio between the two to obtain the noise scaling factor, which is not limited here.

[0095] E3, based on the noise scaling factor, performs spatial denoising on the first denoised image through the spatial denoising module to obtain the second denoised image.

[0096] As described earlier in the workflow description of the spatial domain denoising module, it can be explained that the noise level can be adjusted accordingly. Specifically, the noise scaling factor can be used by the spatial domain denoising module to adjust the noise level of the first denoised image during the spatial domain denoising process. The noise scaling factor used in this adjustment can be a fixed value or a variable value. In some examples, the noise level of the current block = the original noise level of the current block estimated by gain * the noise scaling factor corresponding to the center pixel of the current block.

[0097] It is understood that the embodiments of this application actually propose a specific method for determining the noise scaling factor. The noise scaling factor for each pixel is dynamically calculated by the noise level adjustment module and passed to the spatial domain denoising module. This affects the spatial domain denoising processing performed by the spatial domain denoising module on the first denoised image, achieving different denoising intensities in different regions, resulting in a more balanced overall denoising effect in the final denoising result. For ease of distinction, the denoising result obtained by this spatial domain denoising module can be referred to as the second denoised image.

[0098] Please see Figure 3 , Figure 3 A schematic diagram of the connection structure of the time-domain noise reduction module, the noise level adjustment module, and the spatial noise reduction module is given.

[0099] As can be seen from the above, in this embodiment, a temporal noise reduction module based on an artificial intelligence model and a spatial noise reduction module based on a preset algorithm are simultaneously set in the noise reduction system. These modules are connected in series and each module is equipped with an independent control switch, enabling the system to intelligently select the target noise reduction module for processing based on the current noise reduction scenario. This avoids the problem of a single noise reduction method performing poorly in specific scenarios, ensuring noise reduction performance and image quality across different scenarios. Furthermore, the independent control switches prevent increased power consumption and prolonged processing time caused by simultaneously activating both modules unnecessarily, thus achieving a good balance between noise reduction performance and resource consumption.

[0100] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0101] Corresponding to the noise reduction method provided above, this application also provides a noise reduction device applied to a noise reduction system. The noise reduction system includes a time-domain noise reduction module and a spatial-domain noise reduction module. The time-domain noise reduction module is implemented based on an artificial intelligence model, and the spatial-domain noise reduction module is implemented based on a preset noise reduction algorithm. The time-domain and spatial-domain noise reduction modules are connected in series, and each module has an independent control switch. Please refer to [link to relevant documentation]. Figure 4 The noise reduction device 4 in this embodiment includes: The first determining module 401 is used to determine the current noise reduction scenario; The second determining module 402 is used to determine the target noise reduction module based on the noise reduction scenario in the time-domain noise reduction module and the spatial-domain noise reduction module. The noise reduction module 403 is used to perform noise reduction processing on the acquired current image frame through the target noise reduction module.

[0102] In some embodiments, the time-domain noise reduction module includes: The first processing sub-module is used to process the first image frame and historical image frames through an artificial intelligence model when the target noise reduction module includes a temporal noise reduction module, to obtain the weight map of the first image frame output by the artificial intelligence model. The first image frame is the input of the temporal noise reduction module, and the historical image frames are the noise-reduced image frames obtained by the temporal noise reduction module in the previous time. The weight map of the first image frame has the same scale as the first image frame and is used to describe the weight of each pixel in the first image frame. The first determining sub-module is used to determine the weight map of historical image frames based on the weight map of the first image frame. The weight map of the historical image frames has the same scale as the historical image frames and is used to describe the weight of each pixel in the historical image frames. The first noise reduction submodule is used to perform weighted fusion of the first image frame and the historical image frames based on the weight map of the first image frame and the weight map of the historical image frames, so as to obtain a noise-reduced image that has undergone temporal noise reduction processing for the first image frame.

[0103] In some embodiments, the artificial intelligence model adopts an N-layer pyramid structure, each layer including a downsampling unit, a frame difference calculation unit, a noise model unit, a convolution and nonlinear activation unit, and an upsampling unit. The input of the downsampling unit of the first layer is the input of the artificial intelligence model, and the output of the upsampling unit of the first layer is the output of the artificial intelligence model.

[0104] In some embodiments, the input of the downsampling unit of the i-th layer is the output of the downsampling unit of the (i-1)-th layer, where i is a positive integer greater than 1 and not less than N; The input to the frame difference calculation unit of the j-th layer is the output of the downsampling unit of the j-th layer, where j is a positive integer greater than 0 and not less than N; The input to the noise model unit of the j-th layer is the preset noise model coefficients and the output of the downsampling unit of the j-th layer for the input originating from the first image frame; The input to the convolution and nonlinear activation unit of the k-th layer is the channel concatenation result of the output of the frame difference calculation unit of the k-th layer, the output of the noise model unit of the k-th layer, and the output of the upsampling unit of the (k+1)-th layer. The input to the convolution and nonlinear activation unit of the N-th layer is the channel concatenation result of the output of the frame difference calculation unit of the N-th layer and the output of the noise model unit of the N-th layer. k is a positive integer less than N. The input to the upsampling unit in layer j is the output of the convolution and nonlinear activation units in layer j.

[0105] In some embodiments, the spatial noise reduction module includes: The second determining sub-module is used to traverse each pixel of the second image frame and determine the current pixel when the target noise reduction module includes the spatial noise reduction module. The second image frame is the input of the spatial noise reduction module. The third determining sub-module is used to determine the noise level of the current block based on the average brightness of the current block where the current pixel is located, wherein the current block is a preset first size and the current block is centered on the current pixel. The fourth sub-module is determined based on the noise level of the current block and the brightness difference between the current block and each other block in the preset search range. The search range is a preset second size, which is larger than the first size, and the search range is centered on the current pixel. The fifth sub-module is used to determine the noise reduction brightness of the current pixel based on the weights of other blocks, the brightness of the center pixels of other blocks, and the brightness of the current pixel. The second noise reduction submodule is used to obtain a denoised image of the second image frame after completing the traversal of each pixel of the second image frame and based on the noise reduction brightness of each pixel.

[0106] In some embodiments, the fourth determination of a sub-module includes: The first determining unit is used to determine the product of the noise level of the current block and the preset noise reduction intensity as a reference value; The first calculation unit is used to multiply the reference value by M preset coefficients to obtain M thresholds, each of which is associated with a different weight. The second determining unit is used to compare each brightness difference with M thresholds and determine the threshold closest to each brightness difference. The third determining unit is used to determine the weight of each other block based on the threshold closest to each brightness difference and the correlation between the threshold and the weight.

[0107] In some embodiments, the fifth determination of the sub-module includes: The second calculation unit is used to accumulate the weights of each other block, and additionally accumulate the maximum weight among the weights of each other block to obtain the weight sum; The third calculation unit is used to multiply the brightness of the center pixel of each other block by the weight of the corresponding other block to obtain the weighted brightness of the center pixel of each other block. The fourth calculation unit is used to multiply the brightness of the current pixel by the maximum weight to obtain the weighted brightness of the current pixel; The fifth calculation unit is used to accumulate the weighted brightness of the center pixels of each other block, and additionally accumulate the weighted brightness of the current pixel to obtain the weighted brightness sum; The fourth determining unit is used to determine the quotient of the weighted brightness sum and the weighted sum as the noise-reduced brightness of the current pixel.

[0108] In some embodiments, the noise reduction system further includes a noise level adjustment module, the noise reduction module 403, comprising: The first noise reduction submodule is used to perform temporal noise reduction processing on the current image frame through the temporal noise reduction module when the target noise reduction module includes a spatial noise reduction module and a temporal noise reduction module, so as to obtain a first noise-reduced image. The first noise-reduced image is a noise-reduced image that has undergone temporal noise reduction processing for the current image frame. The noise adjustment submodule is used to process the current noise level of the current image frame and the historical noise level of the historical image frame through the noise level adjustment module to obtain the noise scaling coefficient. The historical image frame is the denoised image frame obtained by the temporal denoising module in the previous process, and the historical noise level is the fused noise level obtained by the noise level adjustment module in the previous process. The second noise reduction submodule is used to perform spatial noise reduction processing on the first denoised image based on the noise scaling factor through the spatial noise reduction module to obtain a second denoised image. The second denoised image is a denoised image that has undergone spatial noise reduction processing on the first denoised image. The noise scaling factor is used by the spatial noise reduction module to adjust the noise level of the first denoised image during the spatial noise reduction process.

[0109] In some embodiments, the noise level adjustment module includes: The fusion unit is used to fuse the current noise level and the historical noise level based on the weight map of the current image frame to obtain the fused noise level. The weight map of the current image frame is obtained through the temporal denoising module. The sixth calculation unit is used to perform proportional calculations on the current noise level and the fused noise level to obtain the noise scaling factor.

[0110] As can be seen from the above, in this embodiment, a temporal noise reduction module based on an artificial intelligence model and a spatial noise reduction module based on a preset algorithm are simultaneously set in the noise reduction system. These modules are connected in series and each module is equipped with an independent control switch, enabling the system to intelligently select the target noise reduction module for processing based on the current noise reduction scenario. This avoids the problem of a single noise reduction method performing poorly in specific scenarios, ensuring noise reduction performance and image quality across different scenarios. Furthermore, the independent control switches prevent increased power consumption and prolonged processing time caused by simultaneously activating both modules unnecessarily, thus achieving a good balance between noise reduction performance and resource consumption.

[0111] Corresponding to the noise reduction method provided above, this application also provides an electronic device equipped with a noise reduction system. The noise reduction system includes a time-domain noise reduction module and a spatial-domain noise reduction module. The time-domain noise reduction module is implemented based on an artificial intelligence model, and the spatial-domain noise reduction module is implemented based on a preset noise reduction algorithm. The time-domain and spatial-domain noise reduction modules are connected in series, and each module has an independent control switch. Please refer to [link to relevant documentation]. Figure 5 The electronic device 5 in this application embodiment includes: a memory 501, and one or more processors 502. Figure 5 (Only one is shown in the image) and a computer program stored in memory 501 and executable on the processor. Specifically, the processor 502 performs the following steps by running the aforementioned computer program stored in memory 501: Determine the current noise reduction scenario; In the time-domain noise reduction module and the spatial-domain noise reduction module, the target noise reduction module is determined according to the noise reduction scenario; The target noise reduction module performs noise reduction processing on the current image frame that has been acquired.

[0112] Assuming the above is the first possible implementation, in the second possible implementation based on the first possible implementation, when the target noise reduction module includes a time-domain noise reduction module, the workflow of the time-domain noise reduction module is as follows: The first image frame and historical image frames are processed by an artificial intelligence model to obtain the weight map of the first image frame output by the artificial intelligence model. The first image frame is the input of the temporal denoising module, and the historical image frames are the previously denoised image frames obtained by the temporal denoising module. The weight map of the first image frame has the same scale as the first image frame and is used to describe the weight of each pixel in the first image frame. Based on the weight map of the first image frame, the weight map of the historical image frame is determined. The weight map of the historical image frame has the same scale as the historical image frame and is used to describe the weight of each pixel in the historical image frame. Based on the weight map of the first image frame and the weight map of the historical image frames, the first image frame and the historical image frames are weighted and fused to obtain a denoised image that has undergone temporal denoising processing for the first image frame.

[0113] In the third possible implementation based on the second possible implementation described above, the artificial intelligence model adopts an N-layer pyramid structure, each layer including a downsampling unit, a frame difference calculation unit, a noise model unit, a convolution and nonlinear activation unit, and an upsampling unit. The input of the downsampling unit of the first layer is the input of the artificial intelligence model, and the output of the upsampling unit of the first layer is the output of the artificial intelligence model.

[0114] In the fourth possible implementation based on the third possible implementation described above, the input of the downsampling unit of the i-th layer is the output of the downsampling unit of the (i-1)-th layer, where i is a positive integer greater than 1 and not less than N. The input to the frame difference calculation unit of the j-th layer is the output of the downsampling unit of the j-th layer, where j is a positive integer greater than 0 and not less than N; The input to the noise model unit of the j-th layer is the preset noise model coefficients and the output of the downsampling unit of the j-th layer for the input originating from the first image frame; The input to the convolution and nonlinear activation unit of the k-th layer is the channel concatenation result of the output of the frame difference calculation unit of the k-th layer, the output of the noise model unit of the k-th layer, and the output of the upsampling unit of the (k+1)-th layer. The input to the convolution and nonlinear activation unit of the N-th layer is the channel concatenation result of the output of the frame difference calculation unit of the N-th layer and the output of the noise model unit of the N-th layer. k is a positive integer less than N. The input to the upsampling unit in layer j is the output of the convolution and nonlinear activation units in layer j.

[0115] In the fifth possible implementation provided based on the first possible implementation described above, when the target noise reduction module includes a spatial noise reduction module, the workflow of the spatial noise reduction module is as follows: The process iterates through each pixel of the second image frame to determine the current pixel, where the second image frame is the input to the spatial domain noise reduction module. The noise level of the current block is determined based on the average brightness of the current block where the current pixel is located. The current block is a preset first size and is centered on the current pixel. Based on the noise level of the current block and the brightness difference between the current block and each other block in the preset search range, the weight of each other block is determined. The search range is a preset second size, which is larger than the first size, and the search range is centered on the current pixel. The noise reduction brightness of the current pixel is determined based on the weights of each other block, the brightness of the center pixel of each other block, and the brightness of the current pixel. After traversing each pixel of the second image frame, the denoised image of the second image frame is obtained based on the denoised brightness of each pixel.

[0116] In a sixth possible implementation based on the fifth possible implementation described above, the weights of each other block are determined according to the noise level of the current block and the brightness differences between the current block and each other block within a preset search range, including: The product of the noise level of the current block and the preset noise reduction intensity is determined as the reference value; The reference value is multiplied by M preset coefficients to obtain M thresholds, each of which is associated with a different weight. Compare each brightness difference with M thresholds to determine the threshold that is closest to each brightness difference; The weights of each other block are determined based on the threshold closest to each brightness difference and the correlation between the threshold and the weight.

[0117] In the seventh possible implementation provided based on the fifth possible implementation described above, the noise reduction brightness of the current pixel is determined according to the weights of each other block, the brightness of the center pixel of each other block, and the brightness of the current pixel, including: The weights of each other block are summed, and the maximum weight among the weights of each other block is additionally summed to obtain the weight sum; The brightness of the center pixel of each other block is multiplied by the weight of the corresponding other block to obtain the weighted brightness of the center pixel of each other block; Multiply the brightness of the current pixel by the maximum weight to obtain the weighted brightness of the current pixel; The weighted brightness of the center pixels of each other block is accumulated, and the weighted brightness of the current pixel is additionally accumulated to obtain the weighted brightness sum; The quotient of the weighted brightness sum and the weighted sum is determined as the noise-reduced brightness of the current pixel.

[0118] In a seventh possible implementation based on the first possible implementation described above, the noise reduction system further includes a noise level adjustment module. When the target noise reduction module includes a spatial noise reduction module and a temporal noise reduction module, the target noise reduction module performs noise reduction processing on the acquired current image frame, including: The current image frame is denoised by the temporal domain denoising module to obtain the first denoised image, which is the denoised image of the current image frame after temporal domain denoising. The noise level adjustment module processes the current noise level of the current image frame and the historical noise level of the historical image frames to obtain the noise scaling factor. The historical image frame is the denoised image frame obtained by the temporal denoising module in the previous process, and the historical noise level is the fused noise level obtained by the noise level adjustment module in the previous process. Based on the noise scaling factor, the first denoised image is subjected to spatial denoising processing by the spatial denoising module to obtain the second denoised image. The second denoised image is the denoised image after spatial denoising processing of the first denoised image. The noise scaling factor is used by the spatial denoising module to adjust the noise level of the first denoised image during the spatial denoising process.

[0119] In the seventh possible implementation provided based on the first possible implementation described above, the noise level of the current image frame and the historical noise level of historical image frames are processed by a noise level adjustment module to obtain a noise scaling factor, including: The current noise level and historical noise level are fused based on the weight map of the current image frame to obtain the fused noise level. The weight map of the current image frame is obtained through a temporal denoising module. The noise scaling factor is obtained by proportionally calculating the current noise level and the combined noise level.

[0120] It should be understood that, in the embodiments of this application, the processor 502 may be a central processing unit (CPU), but it may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0121] Memory 501 may include read-only memory and random access memory, and provides instructions and data to processor 502. Some or all of memory 501 may also include non-volatile random access memory. For example, memory 501 may also store device type information.

[0122] As can be seen from the above, in this embodiment, a temporal noise reduction module based on an artificial intelligence model and a spatial noise reduction module based on a preset algorithm are simultaneously set in the noise reduction system. These modules are connected in series and each module is equipped with an independent control switch, enabling the system to intelligently select the target noise reduction module for processing based on the current noise reduction scenario. This avoids the problem of a single noise reduction method performing poorly in specific scenarios, ensuring noise reduction performance and image quality across different scenarios. Furthermore, the independent control switches prevent increased power consumption and prolonged processing time caused by simultaneously activating both modules unnecessarily, thus achieving a good balance between noise reduction performance and resource consumption.

[0123] This application also provides a computer program product that, when run on an electronic device, enables the electronic device to perform the steps described in the various method embodiments above.

[0124] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the above device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0125] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0126] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of external device software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0127] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For instance, the division of modules or units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units through some interfaces, and may be electrical, mechanical, or other forms.

[0128] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0129] If the integrated units described above are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing associated hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable storage medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer-readable storage device, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc. It should be noted that the contents of the aforementioned computer-readable storage media may be appropriately added to or subtracted from the contents according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable storage media may not include electrical carrier signals and telecommunication signals.

[0130] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A noise reduction method, characterized in that, The noise reduction method is applied to a noise reduction system, which includes a time-domain noise reduction module and a spatial noise reduction module. The time-domain noise reduction module is implemented based on an artificial intelligence model, and the spatial noise reduction module is implemented based on a preset noise reduction algorithm. The time-domain noise reduction module and the spatial noise reduction module are connected in series, and each module has an independent control switch. The noise reduction method includes: Determine the current noise reduction scenario; In the time-domain noise reduction module and the spatial-domain noise reduction module, the target noise reduction module is determined according to the noise reduction scenario; The target noise reduction module performs noise reduction processing on the acquired current image frame.

2. The noise reduction method as described in claim 1, characterized in that, When the target noise reduction module includes the time-domain noise reduction module, the workflow of the time-domain noise reduction module is as follows: The first image frame and historical image frames are processed by the artificial intelligence model to obtain the weight map of the first image frame output by the artificial intelligence model. The first image frame is the input of the temporal denoising module, and the historical image frame is the previously denoised image frame obtained by the temporal denoising module. The weight map of the first image frame has the same scale as the first image frame and is used to describe the weight of each pixel in the first image frame. Based on the weight map of the first image frame, a weight map of the historical image frame is determined. The weight map of the historical image frame has the same scale as the historical image frame and is used to describe the weight of each pixel in the historical image frame. Based on the weight map of the first image frame and the weight map of the historical image frames, the first image frame and the historical image frames are weighted and fused to obtain a denoised image for which temporal denoising processing has been performed on the first image frame.

3. The noise reduction method as described in claim 2, characterized in that, The artificial intelligence model adopts an N-layer pyramid structure. Each layer includes a downsampling unit, a frame difference calculation unit, a noise model unit, a convolution and nonlinear activation unit, and an upsampling unit. The input of the downsampling unit in the first layer is the input of the artificial intelligence model, and the output of the upsampling unit in the first layer is the output of the artificial intelligence model.

4. The noise reduction method as described in claim 3, characterized in that, The input of the downsampling unit of the i-th layer is the output of the downsampling unit of the (i-1)-th layer, where i is a positive integer greater than 1 and not less than N; The input to the frame difference calculation unit of the j-th layer is the output of the downsampling unit of the j-th layer, where j is a positive integer greater than 0 and not less than N; The input to the noise model unit of the j-th layer is the preset noise model coefficients and the output of the downsampling unit of the j-th layer for the input originating from the first image frame; The input to the convolution and nonlinear activation unit of the k-th layer is the channel concatenation result of the output of the frame difference calculation unit of the k-th layer, the output of the noise model unit of the k-th layer, and the output of the upsampling unit of the (k+1)-th layer. The input to the convolution and nonlinear activation unit of the N-th layer is the channel concatenation result of the output of the frame difference calculation unit of the N-th layer and the output of the noise model unit of the N-th layer. k is a positive integer less than N. The input to the upsampling unit in layer j is the output of the convolution and nonlinear activation units in layer j.

5. The noise reduction method as described in claim 1, characterized in that, When the target noise reduction module includes the spatial noise reduction module, the workflow of the spatial noise reduction module is as follows: The pixels of the second image frame are traversed to determine the current pixel, wherein the second image frame is the input of the spatial domain noise reduction module; The noise level of the current block is determined based on the average brightness of the current block in which the current pixel is located, wherein the current block has a preset first size and the current block is centered on the current pixel. Based on the noise level of the current block and the brightness difference between the current block and each other block in the preset search range, the weight of each other block is determined. The search range is a preset second size, which is larger than the first size, and the search range is centered on the current pixel. The noise reduction brightness of the current pixel is determined based on the weights of each other block, the brightness of the center pixel of each other block, and the brightness of the current pixel. After traversing each pixel of the second image frame, a denoised image is obtained based on the denoised brightness of each pixel, which has undergone spatial denoising processing for the second image frame.

6. The noise reduction method as described in claim 5, characterized in that, The step of determining the weight of each other block based on the noise level of the current block and the brightness difference between the current block and each other block within a preset search range includes: The product of the noise level of the current block and the preset noise reduction intensity is determined as a reference value; The reference value is multiplied by M preset coefficients to obtain M thresholds, each of which is associated with a different weight. Each of the brightness differences is compared with the M thresholds to determine the threshold that is closest to each of the brightness differences; The weights of each other block are determined based on the threshold closest to each of the brightness differences and the correlation between the threshold and the weight.

7. The noise reduction method as described in claim 5, characterized in that, The step of determining the noise reduction brightness of the current pixel based on the weights of other blocks, the brightness of the center pixels of other blocks, and the brightness of the current pixel includes: The weights of each of the other blocks are summed, and the maximum weight among the weights of each of the other blocks is additionally summed to obtain the weight sum; The brightness of the center pixel of each of the other blocks is multiplied by the weight of the corresponding other block to obtain the weighted brightness of the center pixel of each of the other blocks; Multiply the brightness of the current pixel by the maximum weight to obtain the weighted brightness of the current pixel; The weighted brightness of the center pixels of each of the other blocks is accumulated, and the weighted brightness of the current pixel is additionally accumulated to obtain the weighted brightness sum; The quotient of the weighted brightness and the weighted sum is determined as the noise-reduced brightness of the current pixel.

8. The noise reduction method as described in claim 1, characterized in that, The noise reduction system further includes a noise level adjustment module. When the target noise reduction module includes both the spatial noise reduction module and the temporal noise reduction module, the noise reduction processing of the acquired current image frame using the target noise reduction module includes: The temporal denoising module performs temporal denoising on the current image frame to obtain a first denoised image, which is a denoised image that has undergone temporal denoising on the current image frame. The noise level adjustment module processes the current noise level of the current image frame and the historical noise level of the historical image frames to obtain a noise scaling factor. The historical image frame is the previously denoised image frame obtained by the temporal denoising module, and the historical noise level is the fused noise level obtained in the previous processing of the noise level adjustment module. Based on the noise scaling factor, the first denoised image is subjected to spatial denoising processing by the spatial denoising module to obtain a second denoised image. The second denoised image is a denoised image that has undergone spatial denoising processing on the first denoised image. The noise scaling factor is used by the spatial denoising module to adjust the noise level of the first denoised image during the spatial denoising process.

9. The noise reduction method as described in claim 8, characterized in that, The step of processing the current noise level of the current image frame and the historical noise levels of historical image frames through the noise level adjustment module to obtain a noise scaling factor includes: The current noise level and the historical noise level are fused based on the weight map of the current image frame to obtain a fused noise level, wherein the weight map of the current image frame is obtained through the temporal denoising module; The noise scaling factor is obtained by proportionally calculating the current noise level and the fused noise level.

10. A noise reduction device, characterized in that, The noise reduction device is applied to a noise reduction system, which includes a time-domain noise reduction module and a spatial noise reduction module. The time-domain noise reduction module is implemented based on an artificial intelligence model, and the spatial noise reduction module is implemented based on a preset noise reduction algorithm. The time-domain noise reduction module and the spatial noise reduction module are connected in series, and each module has an independent control switch. The noise reduction device includes: The first determining module is used to determine the current noise reduction scenario; The second determining module is used to determine the target noise reduction module based on the noise reduction scenario in the time-domain noise reduction module and the spatial-domain noise reduction module. The noise reduction module is used to perform noise reduction processing on the acquired current image frame through the target noise reduction module.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 9.

12. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by one or more processors, implements the method as described in any one of claims 1 to 9.

Images