Image denoising processing method, device, storage medium and electronic device

Abstract

The present disclosure provides an image denoising processing method, device, computer-readable storage medium and electronic device, which relate to the field of image and video processing technology. The image denoising processing method includes: obtaining an image to be processed, and obtaining inertial sensor data corresponding to the image to be processed and scene information of the image to be processed; determining the motion information of the image to be processed according to the inertial sensor data and the scene information; fusing the image to be processed with a reference image based on the motion information of the image to be processed to obtain a denoised image corresponding to the image to be processed; the reference image is an image taken at a different time for the same shooting scene as the image to be processed. The present disclosure improves the image denoising effect, and improves the motion blur caused by excessive denoising strength in the time domain denoising processing, and the problem of more image noise being retained due to insufficient denoising strength, thereby improving the image quality after denoising processing.

Description

Background Art

[0002] Image noise refers to the interference information existing in image data, which usually comes from external influences on the image system and errors and jitters within the image system, which will affect the visual effect of the image. In related technologies, the effect of image denoising needs to be improved.

[0003] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to those skilled in the art. Summary of the invention

[0004] The present disclosure provides an image denoising processing method, an image denoising processing device, a computer-readable storage medium and an electronic device, so as to improve the effect of image denoising processing at least to a certain extent.

[0005] According to a first aspect of the present disclosure, there is provided an image denoising processing method, comprising: acquiring an image to be processed, and acquiring inertial sensor data corresponding to the image to be processed and scene information of the image to be processed; determining motion information of the image to be processed according to the inertial sensor data and the scene information; fusing the image to be processed with a reference image based on the motion information of the image to be processed to obtain a denoised image corresponding to the image to be processed; the reference image is an image shot at a different time for the same shooting scene as the image to be processed.

[0006] According to a second aspect of the present disclosure, there is provided an image denoising processing device, comprising: an image acquisition module, configured to acquire an image to be processed, and acquire inertial sensing data corresponding to the image to be processed and scene information of the image to be processed; a motion information determination module, configured to determine the motion information of the image to be processed according to the inertial sensing data and the scene information; an image fusion module, configured to fuse the image to be processed with a reference image based on the motion information to obtain a denoised image corresponding to the image to be processed; the reference image is an image shot at a different time for the same shooting scene as the image to be processed.

[0007] According to a third aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the image denoising processing method of the first aspect and possible implementation methods thereof are implemented.

[0008] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the image denoising processing method of the above-mentioned first aspect and its possible implementation method by executing the executable instructions.

[0009] The technical solution disclosed in this disclosure has the following beneficial effects:

[0010] Based on the above method, on the one hand, the motion information of the image to be processed is determined by combining the two types of information, inertial sensor data and scene information, which can match the actual motion conditions of the image to be processed and the needs and preferences for the image denoising strength, thereby accurately guiding the fusion ratio of the image to be processed and the reference image in the image time domain denoising, and improving the denoising effect. On the other hand, the motion information used in this scheme corresponds to a suitable fusion ratio, which is equivalent to a suitable image denoising strength in the time domain denoising process, thereby reducing the motion blur caused by excessive denoising strength, or the situation where more image noise is retained due to insufficient denoising strength, thereby improving the image quality after denoising.

[0011] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0013] Figure 1 A schematic diagram showing a system architecture in this exemplary embodiment;

[0014] Figure 2 A schematic diagram showing the structure of an electronic device in this exemplary embodiment is shown;

[0015] Figure 3 A flowchart of an image denoising method according to the exemplary embodiment is shown;

[0016] Figure 4 A flow chart of determining motion information in this exemplary embodiment is shown;

[0017] Figure 5 A schematic diagram showing a method of determining a global motion degree value in this exemplary embodiment is shown;

[0018] Figure 6 A flow chart of determining motion information in this exemplary embodiment is shown;

[0019] Figure 7 A schematic diagram showing an image denoising method in this exemplary embodiment;

[0020] Figure 8A schematic structural diagram of an image denoising processing device in this exemplary embodiment is shown. DETAILED DESCRIPTION

[0021] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0022] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0023] Temporal Noise Reduction (TNR) refers to denoising based on the continuity of image signals in the time domain. It usually fuses multiple frames of images that are continuous in time to suppress the noise of a single frame. In related technologies, it is difficult to determine the appropriate fusion ratio when fusing multiple frames of images, which affects the denoising effect.

[0024] In view of the above problems, an exemplary embodiment of the present disclosure provides an image denoising method. Figure 1 The system architecture and application scenarios of the operating environment of this exemplary embodiment are exemplarily described.

[0025] Figure 11 shows a schematic diagram of a system architecture, which may include a terminal 110 and a server 120. The terminal 110 may be a terminal device such as a smart phone, a tablet computer, a desktop computer, a laptop computer, etc. The server 120 generally refers to a background system that provides image denoising processing related services in this exemplary embodiment, and may be a server or a cluster formed by multiple servers. The terminal 110 and the server 120 may be connected via a wired or wireless communication link to exchange data.

[0026] In one implementation, the image denoising method in this exemplary implementation may be performed by the terminal 110. For example, when a user uses the terminal 110 to shoot a video image or video, or when a user selects an image or video in a local library of the terminal 110, the terminal 110 performs denoising on one or more frames of the image or video, and outputs the denoised image or video.

[0027] In one implementation, the image denoising method may be performed by the server 120. For example, after a user uses the terminal 110 to shoot an image or video, or after a user selects an image or video from a local library of the terminal 110, the terminal 110 uploads the image or video to the server 120, and the server 120 performs denoising on the image or one or more frames of the image in the video, and returns the denoised image or video to the terminal 110.

[0028] As can be seen from the above, the execution subject of the image denoising processing method in this exemplary embodiment may be the terminal 110 or the server 120, and the present disclosure does not limit this.

[0029] The exemplary embodiment of the present disclosure also provides an electronic device for executing the image denoising processing method, and the electronic device may be the above-mentioned terminal 110 or the server 120. Generally, the electronic device may include a processor and a memory, the memory is used to store executable instructions of the processor, and the processor is configured to execute the above-mentioned image denoising processing method by executing the executable instructions.

[0030] Below Figure 2 The mobile terminal 200 in FIG. 1 is taken as an example to exemplify the structure of the electronic device. It should be understood by those skilled in the art that, in addition to the components specifically used for mobile purposes, Figure 2 The construction in can also be applied to fixed type equipment.

[0031] like Figure 2 As shown, the mobile terminal 200 may specifically include: a processor 201, a memory 202, a bus 203, a mobile communication module 204, an antenna 1, a wireless communication module 205, an antenna 2, a display screen 206, a camera module 207, an audio module 208, a power module 209 and a sensor module 210.

[0032] The processor 201 may include one or more processing units, for example, the processor 201 may include an AP (Application Processor), a modem processor, a GPU (Graphics Processing Unit), an ISP (Image Signal Processor), a controller, an encoder, a decoder, a DSP (Digital Signal Processor), a baseband processor and / or an NPU (Neural-Network Processing Unit), etc. The image denoising processing method in this exemplary embodiment may be executed by an AP, a GPU or a DSP. When the method involves processing related to a neural network, it may be executed by an NPU, for example, the NPU may load neural network parameters and execute algorithm instructions related to the neural network.

[0033] The encoder can encode (i.e. compress) an image or video to reduce the data size for easy storage or transmission. The decoder can decode (i.e. decompress) the encoded data of an image or video to restore the image or video data. The mobile terminal 200 can support one or more encoders and decoders, such as: image formats such as JPEG (Joint Photographic Experts Group), PNG (Portable Network Graphics), BMP (Bitmap), and video formats such as MPEG (Moving Picture Experts Group) 1, MPEG2, H.263, H.264, and HEVC (High Efficiency Video Coding).

[0034] The processor 201 may be connected to the memory 202 or other components via a bus 203 .

[0035] The memory 202 can be used to store computer executable program codes, which include instructions. The processor 201 executes various functional applications and data processing of the mobile terminal 200 by running the instructions stored in the memory 202. The memory 202 can also store application data, such as images, videos and other files.

[0036] The communication function of the mobile terminal 200 can be realized by the mobile communication module 204, the antenna 1, the wireless communication module 205, the antenna 2, the modulation and demodulation processor, and the baseband processor. The antenna 1 and the antenna 2 are used to transmit and receive electromagnetic wave signals. The mobile communication module 204 can provide 2G, 3G, 4G, 5G and other mobile communication solutions applied to the mobile terminal 200. The wireless communication module 205 can provide wireless communication solutions such as wireless LAN, Bluetooth, and near field communication applied to the mobile terminal 200.

[0037] The display screen 206 is used to implement display functions, such as displaying user interfaces, images, videos, etc. The camera module 207 is used to implement shooting functions, such as shooting images, videos, etc. The audio module 208 is used to implement audio functions, such as playing audio, collecting voice, etc. The power module 209 is used to implement power management functions, such as charging the battery, powering the device, monitoring the battery status, etc.

[0038] The sensor module 210 may include an inertial sensor 2101 for detecting the motion posture of the mobile terminal 200 and outputting inertial sensing data. In one embodiment, the inertial sensor 2101 may include a gyroscope, which may output the three-axis angular velocity of the mobile terminal 200 during the motion. In one embodiment, the inertial sensor 2101 may also include an accelerometer, which may output the three-axis acceleration of the mobile terminal 200 during the motion.

[0039] In addition, the sensor module 210 may also include other types of sensors for implementing corresponding sensing detection functions.

[0040] Combine the following Figure 3 The image denoising processing method in this exemplary embodiment is described. Figure 3 An exemplary process of the image denoising method is shown, which may include:

[0041] Step S310, obtaining an image to be processed, and obtaining inertial sensor data corresponding to the image to be processed and scene information of the image to be processed;

[0042] Step S320, determining motion information of the image to be processed according to the inertial sensing data and the scene information;

[0043] Step S330 , fusing the image to be processed with the reference image based on the motion information of the image to be processed to obtain a denoised image corresponding to the image to be processed.

[0044] Based on the above method, on the one hand, the motion information of the image to be processed is determined by combining the two types of information, inertial sensor data and scene information, which can match the actual motion conditions of the image to be processed and the needs and preferences for the image denoising strength, thereby accurately guiding the fusion ratio of the image to be processed and the reference image in the image time domain denoising, and improving the denoising effect. On the other hand, the motion information used in this scheme corresponds to a suitable fusion ratio, which is equivalent to a suitable image denoising strength in the time domain denoising process, thereby reducing the motion blur caused by excessive denoising strength, or the situation where more image noise is retained due to insufficient denoising strength, thereby improving the image quality after denoising.

[0045] Below Figure 3 Each step in the procedure is described in detail.

[0046] refer to Figure 3 In step S310, the image to be processed is obtained, and the inertial sensing data corresponding to the image to be processed and the scene information of the image to be processed are obtained.

[0047] The image to be processed is an image that needs to be denoised. The present disclosure does not limit the source of the image to be processed. For example, the image to be processed may be a currently captured image, an image selected by a user in a local library, or an image obtained from other devices.

[0048] In one implementation, the image to be processed may be a frame image among a plurality of continuous frame images, for example, may be a current frame image in a video.

[0049] The inertial sensor data corresponding to the image to be processed refers to the inertial sensor data when the device shoots the image to be processed. For example, when the mobile terminal 200 shoots the image to be processed, its inertial sensor 2101 synchronously outputs the inertial sensor data.

[0050] In one embodiment, both the image to be processed and the inertial sensor data carry a timestamp, the timestamp of the image to be processed indicates the time when the image to be processed is captured or output, and the timestamp of the inertial sensor data indicates the time when the inertial sensor data is detected or output. By making the timestamp of the image to be processed correspond to the timestamp of the inertial sensor data, the image to be processed and the inertial sensor data can be made correspondent.

[0051] Scene information is information reflecting the shooting scene of the image to be processed (the shooting scene in this article includes the shooting object). In one embodiment, the scene information of the image to be processed may include a scene tag (Scene Tag), and the scene tag represents the classification tag of the shooting scene of the image to be processed, for example, it may be scenery, beach, blue sky, green grass, snow scene, night scene, sunset, fireworks, spotlight, indoor, text, portrait, baby, cat, dog, food, etc. In one embodiment, the scene information of the image to be processed may include multiple scene tags. For example, the image to be processed includes multiple shooting objects, and different shooting objects may correspond to different scene tags, or the foreground and background in the image to be processed correspond to different scene tags. Exemplarily, the scene information of the image to be processed may include portraits and other scene tags. For example, if the image to be processed is a portrait shot in the snow, the scene information may include two scene tags: portrait and snow scene.

[0052] The present disclosure does not limit the method for obtaining scene information. For example, when a user shoots an image to be processed, a plurality of optional shooting modes are provided, such as portrait, night scene, indoor, text, etc., so that the scene information of the image to be processed can be determined according to the shooting mode selected by the user.

[0053] In one embodiment, scene detection may be performed on the image to be processed after it is acquired to obtain scene information. Scene detection may be achieved by an artificial intelligence method or a non-artificial intelligence image processing method. Exemplarily, the artificial intelligence method may include using a pre-trained neural network to perform scene detection on the image to be processed to output scene information. The non-artificial intelligence image processing method may include performing color detection on the image to be processed, and determining the scene information based on the color detection result. For example, if it is detected that the image to be processed includes a large area of ​​sky blue, then the scene of the image to be processed is determined to be blue sky.

[0054] In one embodiment, the above-mentioned artificial intelligence method and non-artificial intelligence image processing method can be combined to realize scene detection. For example, after using neural network and color detection to perform scene detection respectively, the scene information is determined by combining the detection results of the two aspects.

[0055] In one implementation, the scene information may include global scene information of the image to be processed and / or local scene information of at least one local area in the image to be processed.

[0056] Global scene information refers to the scene information for the entire image to be processed. Generally, the image to be processed is macroscopically manifested as a certain shooting scene, and the information of the scene is the global scene information of the image to be processed. In contrast, local scene information refers to the scene area of ​​the local area of ​​the image to be processed, usually the scene information corresponding to a local shooting object. For example, the image to be processed is an image of a person and a dog shot in the snow. When the proportion of people and dogs in the image to be processed is not high, the global scene information of the image to be processed can be snow, the local scene information of the local area where the person is located can be a portrait, and the local scene information of the local area where the dog is located can be a dog; when the proportion of people in the image to be processed is high, the global scene information of the image to be processed can be a portrait, the local scene information of the local area where the dog is located can be a dog, and the local scene information of the local area of ​​the snow can be snow.

[0057] In one embodiment, a global scene detection model and a local scene detection model may be trained to detect the global scene information and the local scene information of the image to be processed, respectively. For example, the global scene detection model may be an image classification model, which is used to output a scene label for globally classifying the image to be processed; the local scene detection model may be a semantic segmentation model, which is used to output a semantic classification result (i.e., a scene label) for each local area (or each pixel) in the image to be processed.

[0058] In one embodiment, a model may be trained to perform global scene classification and local scene classification at the same time, and output global scene information and local scene information.

[0059] In one implementation, the global scene information of the image to be processed may be determined by a global scene detection model, and then a non-artificial intelligence image processing method (such as color detection) may be used to determine the local scene information of the image to be processed.

[0060] Continue to refer Figure 3 In step S320, the motion information of the image to be processed is determined based on the inertial sensing data and the scene information.

[0061] The motion information indicates the degree of motion of the captured content (such as people, animals, buildings, etc.) or the whole in the image to be processed. In one embodiment, the motion information may indicate the degree of motion of the captured content of different parts in the image to be processed. That is, different degrees of motion may be determined for different parts in the image to be processed. Exemplarily, the motion information may be represented by a motion map, and the motion map may have the same number of pixels as the image to be processed, and each pixel in the motion map indicates the degree of motion of the corresponding pixel in the image to be processed.

[0062] Motion information can be used for subsequent image temporal denoising. Generally, the higher the degree of motion of a pixel or a region in the image to be processed, the easier it is for the pixel or region to produce motion blur when the image to be processed is fused with the reference image. Therefore, the lower the proportion of the reference image in the pixel or region, the weaker the image denoising.

[0063] This exemplary embodiment determines the motion information of the image to be processed based on two types of information: inertial sensor data and scene information. Among them, the inertial sensor data represents the motion posture of the device that shoots the image to be processed, which can be quantified as the degree of motion of the entire image to be processed. The scene information can affect the motion information of the image to be processed in two ways. On the one hand, the shooting scene of the image to be processed may belong to a dynamic scene or a static scene, thereby determining the motion information of the image to be processed to a certain extent. On the other hand, the scene information of the image to be processed can reflect the degree of noise that may exist in the shooting scene of the image to be processed, as well as the demand and preference of the image to be processed for the intensity of image denoising, which essentially represents the impact of image denoising or motion blur on the visual effect in the shooting scene of the image to be processed. Based on this idea, the motion information can be adjusted according to the scene information of the image to be processed to achieve the adjustment of the intensity of image denoising.

[0064] In one embodiment, the scene information may include global scene information of the image to be processed. Figure 4 As shown, the above-mentioned determination of the motion information of the image to be processed based on the inertial sensing data and the scene information may include the following steps S410 to S430:

[0065] Step S410, determining corresponding global motion discrimination information according to the global scene information.

[0066] The global motion discrimination information is information used to discriminate the global motion degree of the image to be processed. For example, the global motion discrimination information may be a numerical mapping parameter between inertial sensor data and the global motion degree.

[0067] In one embodiment, the global motion discrimination information may include n global motion discrimination value intervals, where n is a positive integer not less than 2. The n global motion discrimination value intervals may be arranged in order from small to large, respectively representing different degrees of motion. For example, the smallest global motion discrimination value interval may indicate that the image to be processed is basically in a static state and is not prone to motion blur; the middle global motion discrimination value interval may indicate that the image to be processed is in a medium motion state and may produce motion blur; the largest global motion discrimination value interval may indicate that the image to be processed is in a violent motion state and is prone to motion blur.

[0068] The above n global motion discrimination value intervals can correspond to n motion degree values ​​respectively. For example, four global motion discrimination value intervals can be determined, from small to large, [0, th1), [th1, th2), [th2, th3), [th3, ∞), where th1, th2, th3 represent the segmentation thresholds used to determine the global motion discrimination value intervals, and the four global motion discrimination value intervals correspond to the motion degree values ​​k1, k2, k3, k4 that increase in sequence.

[0069] In one embodiment, the segmentation thresholds th1, th2, and th3 can be determined according to the global scene information to obtain n global motion discrimination value intervals. It can be seen that the segmentation threshold or the global motion discrimination value interval actually represents the discrimination standard for the global motion degree. According to the global scene information, the possible noise level and the demand and preference for image denoising in the shooting scene of the image to be processed can be judged, and then the global motion discrimination value interval is set accordingly. For example, when the global scene information is the sky, the preference for image details is low, and the corresponding th1=10, th2=20, and th3=40 can be determined, thereby obtaining a relatively large global motion discrimination value interval, tending to judge the global motion degree of the image to be processed as a lower value to improve the image denoising strength. When the global scene information is grass, the preference for image details is high, and the corresponding th1=5, th2=15, and th3=25 can be determined, thereby obtaining a relatively small global motion discrimination value interval, tending to judge the global motion degree of the image to be processed as a higher value to reduce the image denoising strength.

[0070] Step S420: determining first intermediate motion information according to the relationship between the inertial sensor data and the global motion discrimination information.

[0071] The first intermediate motion information refers to motion information determined by scene information, inertial sensor data, and other information instead of using the difference information between image frames. Correspondingly, the second intermediate motion information refers to motion information determined by using the difference information between image frames (usually the difference between the image to be processed and its reference image). Both the first intermediate motion information and the second intermediate motion information can be used as motion information of the image to be processed separately, or the two can be combined to obtain the motion information of the image to be processed.

[0072] In one embodiment, the first intermediate motion information may be used to represent the global motion degree value of the image to be processed. For example, the first intermediate motion information may be a first intermediate motion image, which has the same size as the image to be processed, and has the same values ​​of each point in the image, which are all global motion degree values.

[0073] Based on the global motion determination information, it can be determined which global motion degree value or values ​​the inertial sensor data corresponds to, thereby obtaining the first intermediate motion information.

[0074] In one implementation, the determining of the first intermediate motion information based on the relationship between the inertial sensor data and the global motion discrimination information may include the following steps:

[0075] Fuse inertial sensor data of multiple dimensions to obtain quantified values ​​of inertial sensor data;

[0076] The first intermediate motion information is determined according to the global motion discrimination value interval in which the quantized value of the inertial sensor data is located.

[0077] The inertial sensing data of each dimension can characterize the degree of movement in a specific direction. The inertial sensing data of multiple dimensions can include three-axis angular velocity, or three-axis angular velocity and three-axis acceleration. By fusing the inertial sensing data of multiple dimensions, a quantitative value of the inertial sensing data that can characterize the overall degree of movement of the device is obtained.

[0078] In one embodiment, the inertial sensor data of multiple dimensions may be normalized and added together to serve as the quantized value of the inertial sensor data. Corresponding weights may also be set for different dimensions, and the inertial sensor data of multiple dimensions may be weighted and summed after normalization to obtain the quantized value of the inertial sensor data. For example, the three-axis angular velocity represents the rotation of the device, and the three-axis acceleration represents the translation of the device. In comparison, rotation is more likely to introduce image noise than translation, so a larger weight may be set for the three-axis angular velocity, and a smaller weight may be set for the three-axis acceleration.

[0079] By comparing the quantized value of the inertial sensor data with the above n global motion discrimination value intervals, the interval in which it is located is determined, and then the motion degree value corresponding to the interval is used as the global motion degree value of the image to be processed to obtain the first intermediate motion information.

[0080] refer to Figure 5Specific description is given. The inertial sensor data corresponding to the image to be processed is multi-dimensionally fused to obtain the quantized value of the inertial sensor data, assuming that its value is 18. If the global scene information of the image to be processed is the sky, the global motion discrimination value intervals corresponding to the sky scene are [0,10), [10,20), [20,40), [40,∞), the inertial sensor data quantized value 18 belongs to the interval [10,20), and the corresponding global motion degree value is k2, thereby obtaining the first intermediate motion information. If the global scene information of the image to be processed is grass, the global motion discrimination value intervals corresponding to the grass scene are [0,5), [5,15), [15,25), [25,∞), which are generally offset to the left (towards the direction of smaller values) compared to the intervals under the sky scene. The inertial sensor data quantized value 18 belongs to the interval [15,25), and the corresponding global motion degree value is k3, thereby obtaining the first intermediate motion information. It can be seen that the same inertial sensor data quantization value 18, the corresponding global motion degree value in the sky scene is k2, and the corresponding global motion degree value in the grass scene is k3. Therefore, the image denoising strength tends to be increased in the sky scene, and tends to be reduced in the grass scene. In this way, the global motion degree value can be determined according to the specific scene, which is conducive to using a suitable fusion ratio in the image time domain denoising and improving the denoising effect.

[0081] In one embodiment, the global motion discrimination information may be a numerical mapping parameter between the inertial sensor data and the global motion degree. Then in step S420, the inertial sensor data may be mapped to a corresponding global motion degree value using the numerical mapping parameter to obtain the first intermediate motion information. For example, the global motion discrimination information may be a numerical mapping ratio, the numerical mapping ratio corresponding to the sky scene is lower, and the numerical mapping ratio corresponding to the grass scene is higher. Then, for the same inertial sensor data, it may be mapped to a lower global motion degree value in the sky scene, and may be mapped to a higher global motion degree value in the grass scene. Then, an image fusion ratio suitable for a specific scene is obtained to improve the denoising effect.

[0082] Step S430: determining motion information of the image to be processed based on the first intermediate motion information.

[0083] The first intermediate motion information may be used alone as the motion information of the image to be processed, or the first intermediate motion information may be combined with other aspects of motion information (such as the second intermediate motion information) to determine the motion information of the image to be processed.

[0084] In one implementation, the scene information may also include local scene information of at least one local area in the image to be processed. Figure 6As shown, the above-mentioned determination of the motion information of the image to be processed based on the first intermediate motion information may include the following steps S610 and S620:

[0085] Step S610: determining second intermediate motion information according to the difference between the image to be processed and the reference image and the local scene information of the local area.

[0086] The reference image is an image captured at a different time for the same shooting scene as the image to be processed. The shooting time of the reference image and the image to be processed is usually within a certain time domain to avoid too long a shooting time interval between the two images. The disclosure does not limit the size of the time domain, which can be determined according to the frame rate of the image shooting and the actual situation. For example, the time domain can be 1 second, and the shooting time interval between the reference image and the image to be processed will not exceed 1 second.

[0087] Exemplarily, the image to be processed is a frame image in a continuous multi-frame image, and the reference image may be another frame image in the continuous multi-frame image, such as an adjacent frame image of the image to be processed. In one embodiment, the image to be processed is a current frame image in a video, and the reference image may be the first m frames of the current image, where m is a positive integer.

[0088] As described above, the second intermediate motion information refers to motion information determined using the difference information between image frames, and the difference information between image frames may include the difference between the image to be processed and the reference image. In this exemplary embodiment, the second intermediate motion information may also be determined in combination with the local scene information of the local area.

[0089] In one embodiment, the second intermediate motion information may be used to represent the local motion degree value of the image to be processed. For example, the second intermediate motion information may be a second intermediate motion image, which has the same size as the image to be processed, and the values ​​of each point in the image are the local motion degree values ​​of each point.

[0090] Generally, if the difference between a pixel point or a region of the image to be processed and the reference image is large, it means that the local motion degree of the pixel point or the region is high. Considering the local scene information of the local region, the local motion degree value can be affected according to the needs and preferences of the local region for image denoising. In this way, the local motion degree value is obtained by combining the difference between the image to be processed and the reference image and the local scene information of the local region, which is conducive to improving the precision and accuracy of the local motion degree value.

[0091] In one implementation, the determining of the second intermediate motion information according to the difference between the image to be processed and the reference image and the local scene information of the local area may include the following steps:

[0092] Determine corresponding local motion discrimination information according to local scene information of the local area;

[0093] The second motion information is determined according to the difference between the image to be processed and the reference image and the local motion discrimination information.

[0094] The local motion discrimination information is information for discriminating the local motion degree of the image to be processed. For example, the local motion discrimination information may be a numerical mapping parameter between an image difference (ie, a difference between the image to be processed and the reference image) and the local motion degree.

[0095] In one embodiment, the local motion discrimination information may include h local motion discrimination value intervals, where h is a positive integer not less than 2. The h local motion discrimination value intervals may be arranged in order from small to large, respectively representing different degrees of motion. For example, the smallest local motion discrimination value interval may indicate that the image to be processed is basically in a static state and is not prone to motion blur; the middle local motion discrimination value interval may indicate that the image to be processed is in a medium motion state and may produce motion blur; the largest local motion discrimination value interval may indicate that the image to be processed is in a violent motion state and is prone to motion blur.

[0096] The above h local motion discrimination value intervals can correspond to h motion degree values. For example, three local motion discrimination value intervals can be determined, from small to large, [0, th4), [th4, th5), [th5, ∞), where th4 and th5 represent segmentation thresholds for determining the local motion discrimination value intervals, and the three local motion discrimination value intervals correspond to the motion degree values ​​k5, k6, and k7 that increase in sequence.

[0097] In one embodiment, the segmentation thresholds th4 and th5 can be determined according to the local scene information to obtain h local motion discrimination value intervals. It can be seen that the segmentation threshold or the local motion discrimination value interval actually represents the discrimination criterion for the degree of local motion. According to the local scene information, the possible noise level and the demand and preference for image denoising in the shooting scene of the image to be processed can be judged, and then the local motion discrimination value interval can be set accordingly.

[0098] Based on the local motion determination information, it can be determined which local motion degree value or values ​​the image difference value corresponds to, thereby obtaining the second intermediate motion information.

[0099] In one embodiment, after determining the image difference of the local area, it is determined which interval of the above-mentioned h local motion judgment value intervals the image difference is in, and then the local motion degree value corresponding to the interval is used as the local motion degree value of the local area to obtain the second intermediate motion information.

[0100] In one implementation, the local motion discrimination information may be a numerical mapping parameter between the image difference and the local motion degree, and the image difference may be mapped to a corresponding local motion degree value using the numerical mapping parameter to obtain the second intermediate motion information.

[0101] Step S620: fusing the first intermediate motion information and the second intermediate motion information to obtain motion information of the image to be processed.

[0102] The first intermediate motion information and the second intermediate motion information may be fused at the pixel level. Exemplarily, the first intermediate motion information and the second intermediate motion information are averaged or maximized at each point to obtain fused motion information.

[0103] In one embodiment, the second intermediate motion information may be adjusted according to the first intermediate motion information, which is equivalent to fusing the first intermediate motion information with the second intermediate motion information to obtain motion information of the image to be processed. For example, the first intermediate motion information is used to represent the global motion degree value of the image to be processed, and the average value or median value of the second intermediate motion information may be calculated and compared with the global motion degree value, such as calculating the difference, to obtain an adjustment amount; and then the second intermediate motion information is adjusted as a whole according to the adjustment amount to obtain the final motion information.

[0104] By fusing the first intermediate motion information with the second intermediate motion information, it is equivalent to combining inertial sensor data, scene information, image difference and other information to determine the motion information, which is beneficial to improving the accuracy of the motion information and the quality of image denoising.

[0105] In one embodiment, the scene information may include local scene information of at least one local area in the image to be processed. The above-mentioned determination of the motion information of the image to be processed based on the inertial sensor data and the scene information may include the following steps:

[0106] Determining global motion information of an image to be processed based on inertial sensor data;

[0107] Adjusting the global motion information based on the local scene information of the local area to obtain first intermediate motion information;

[0108] The motion information of the image to be processed is determined based on the first intermediate motion information.

[0109] As can be seen from the above, the global motion information of the image to be processed may be determined based on the inertial sensing data only without using the scene information, and the global motion information is used to indicate the degree of global motion of the image to be processed.

[0110] Exemplarily, fixed global motion discrimination information may be preset, and after the inertial sensor data is obtained, the global motion information is determined according to the inertial sensor data and the global motion discrimination information.

[0111] The local scene information of the local area reflects the demand and preference of the local area for image denoising, and thus can affect the local motion degree value. Therefore, the global motion information can be adjusted based on the local scene information of the local area to obtain the local motion degree value of the local area, thereby obtaining the first intermediate motion information.

[0112] It should be noted that the first intermediate motion information in this embodiment may be used to indicate the degree of local motion of a local area in the image to be processed.

[0113] After obtaining the first intermediate motion information, the first intermediate motion information can be used alone as the motion information of the image to be processed, or the first intermediate motion information can be combined with other aspects of motion information (such as the second intermediate motion information) to determine the motion information of the image to be processed.

[0114] In one implementation, the determining of the motion information of the image to be processed based on the first intermediate motion information may include the following steps:

[0115] determining second intermediate motion information according to a difference between the image to be processed and the reference image;

[0116] The first intermediate motion information and the second intermediate motion information are fused to obtain motion information of the image to be processed.

[0117] Since the local scene information of the local area has been used when determining the first intermediate motion information, the local scene information of the local area may not be used when determining the second intermediate motion information, and the second intermediate motion information can be determined only based on the difference between the image to be processed and the reference image.

[0118] Exemplarily, fixed local motion discrimination information may be preset, and after obtaining the difference between the image to be processed and the reference image, the second intermediate motion information is determined according to the difference and the local motion discrimination information.

[0119] Furthermore, the first intermediate motion information and the second intermediate motion information may be fused, and the fused motion information may be used as the motion information of the image to be processed.

[0120] After determining the motion information of the image to be processed, continue to refer to Figure 3 In step S330, the image to be processed is fused with the reference image based on the motion information of the image to be processed to obtain a denoised image corresponding to the image to be processed.

[0121] The motion information of the image to be processed includes the motion degree value of each pixel in the image to be processed. Therefore, the fusion ratio corresponding to each pixel can be determined according to the motion degree value of each pixel, and then the image to be processed and the reference image are fused at the pixel level based on the fusion ratio to obtain a denoised image, thereby completing the image time domain denoising process.

[0122] In one implementation, image spatial denoising may be performed before or after image temporal denoising. Exemplarily, filter parameters may be determined based on scene information of the image to be processed, and spatial denoising may be performed on the image to be processed based on the filter parameters. Then, the image after spatial denoising is used as a new image to be processed, and image temporal denoising is performed.

[0123] Figure 7 A schematic diagram of an image denoising method according to this exemplary embodiment is shown.

[0124] like Figure 7 As shown, the image denoising processing method may include:

[0125] Step S701, obtaining the original image of the i-th frame, recorded as the original image i, performing spatial domain denoising processing on the original image i to obtain the image to be processed i; and obtaining the corresponding inertial sensor data i;

[0126] Step S702, performing scene detection on the image to be processed i to obtain global scene information and local scene information;

[0127] Step S703, determining global motion discrimination information according to the global scene information;

[0128] Step S704, determining first intermediate motion information according to the relationship between the inertial sensor data i and the global motion discrimination information;

[0129] Step S705, determining image difference information according to the difference between the image to be processed i and the reference image i-1; the reference image may be one or more frames of images before the original image i, recorded as the reference image i-1;

[0130] Step S706, determining the second intermediate motion information according to the image difference information and the local scene information. For example, the local motion discrimination information may be determined according to the local scene information, and the second intermediate motion information may be determined according to the relationship between the image difference information and the local motion discrimination information.

[0131] Step S707, fusing the first intermediate motion information with the second intermediate motion information to obtain motion information of the image to be processed i, recorded as motion information i;

[0132] Step S708: Based on the motion information i, the image to be processed i is fused with the reference image i-1 to perform time domain denoising processing to obtain a denoised image, which can be used for output, thereby completing the image denoising processing flow.

[0133] The exemplary embodiment of the present disclosure also provides an image denoising processing device. Figure 8 As shown, the image denoising processing device 800 may include:

[0134] The image acquisition module 810 is configured to acquire the image to be processed, and acquire the inertial sensing data corresponding to the image to be processed and the scene information of the image to be processed;

[0135] The motion information determination module 820 is configured to determine the motion information of the image to be processed according to the inertial sensing data and the scene information;

[0136] The image fusion module 830 is configured to fuse the image to be processed with the reference image based on motion information to obtain a denoised image corresponding to the image to be processed; the reference image is an image shot at a different time for the same shooting scene as the image to be processed.

[0137] In one embodiment, the scene information includes global scene information of the image to be processed. The above-mentioned determining the motion information of the image to be processed based on the inertial sensor data and the scene information includes:

[0138] Determine corresponding global motion discrimination information according to the global scene information;

[0139] Determining first intermediate motion information according to a relationship between the inertial sensor data and the global motion discrimination information;

[0140] The motion information of the image to be processed is determined based on the first intermediate motion information.

[0141] In one implementation, the global motion discrimination information includes n global motion discrimination value intervals, where n is a positive integer not less than 2; the first intermediate motion information is determined based on the relationship between the inertial sensor data and the global motion discrimination information, including:

[0142] Fuse inertial sensor data of multiple dimensions to obtain quantified values ​​of inertial sensor data;

[0143] The first intermediate motion information is determined according to the global motion discrimination value interval in which the quantized value of the inertial sensor data is located.

[0144] In one embodiment, the scene information further includes local scene information of at least one local area in the image to be processed. The above-mentioned determining the motion information of the image to be processed based on the first intermediate motion information includes:

[0145] determining second intermediate motion information according to a difference between the image to be processed and the reference image and local scene information of the local area;

[0146] The first intermediate motion information and the second intermediate motion information are fused to obtain motion information of the image to be processed.

[0147] In one embodiment, the second intermediate motion information is determined according to the difference between the image to be processed and the reference image and the local scene information of the local area, including:

[0148] Determine corresponding local motion discrimination information according to local scene information of the local area;

[0149] The second motion information is determined according to the difference between the image to be processed and the reference image and the local motion discrimination information.

[0150] In one embodiment, the scene information includes local scene information of at least one local area in the image to be processed. The above-mentioned determining the motion information of the image to be processed based on the inertial sensor data and the scene information includes:

[0151] Determining global motion information of an image to be processed based on inertial sensor data;

[0152] Adjusting the global motion information based on the local scene information of the local area to obtain first intermediate motion information;

[0153] The motion information of the image to be processed is determined based on the first intermediate motion information.

[0154] In one implementation, the determining the motion information of the image to be processed based on the first intermediate motion information further includes:

[0155] determining second intermediate motion information according to a difference between the image to be processed and the reference image;

[0156] The first intermediate motion information and the second intermediate motion information are fused to obtain motion information of the image to be processed.

[0157] The specific details of each part of the above-mentioned device have been described in detail in the implementation method of the method part. The undisclosed details can be found in the implementation method of the method part, so they will not be repeated here.

[0158] The exemplary embodiments of the present disclosure also provide a computer-readable storage medium, which can be implemented in the form of a program product, which includes a program code, and when the program product is run on an electronic device, the program code is used to cause the electronic device to perform the steps described in the above "Exemplary Method" section of this specification according to various exemplary embodiments of the present disclosure. In an optional embodiment, the program product can be implemented as a portable compact disk read-only memory (CD-ROM) and includes program code, and can be run on an electronic device, such as a personal computer. However, the program product of the present disclosure is not limited to this, and in this document, the readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, device or device.

[0159] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0160] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0161] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0162] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0163] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the exemplary embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0164] It will be appreciated by those skilled in the art that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or implementations combining hardware and software aspects, which may be collectively referred to herein as "circuit", "module" or "system". Those skilled in the art will readily think of other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any variations, uses or adaptive changes of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and implementation are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0165] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image denoising method, characterized in that: include: Acquire an image to be processed, and acquire inertial sensor data corresponding to the image to be processed and scene information of the image to be processed; Determining motion information of the image to be processed according to the inertial sensing data and the scene information; Based on the motion information of the image to be processed, the image to be processed is fused with a reference image to obtain a denoised image corresponding to the image to be processed; the reference image is an image shot at a different time for the same shooting scene as the image to be processed; The scene information includes global scene information of the image to be processed and / or local scene information of at least one local area in the image to be processed; and determining the motion information of the image to be processed based on the inertial sensor data and the scene information includes at least one of the following methods: Determine corresponding global motion discrimination information according to the global scene information; determine first intermediate motion information according to the relationship between the inertial sensor data and the global motion discrimination information; determine motion information of the image to be processed based on the first intermediate motion information; Determine the global motion information of the image to be processed according to the inertial sensor data; adjust the global motion information based on the local scene information of the local area to obtain first intermediate motion information; determine the motion information of the image to be processed based on the first intermediate motion information.

2. The method according to claim 1, characterized in that The global motion discrimination information includes a numerical mapping parameter between the inertial sensor data and the global motion degree; and determining the first intermediate motion information according to the relationship between the inertial sensor data and the global motion discrimination information includes: The inertial sensor data is mapped into a global motion degree value using the numerical mapping parameter to obtain first intermediate motion information.

3. The method according to claim 1, characterized in that The global motion discrimination information includes n global motion discrimination value intervals, where n is a positive integer not less than 2; The determining the first intermediate motion information according to the relationship between the inertial sensor data and the global motion discrimination information comprises: Fusing the inertial sensor data of multiple dimensions to obtain a quantized value of the inertial sensor data; The first intermediate motion information is determined according to the global motion discrimination value interval in which the quantized value of the inertial sensor data is located.

4. The method according to claim 1, characterized in that: The scene information further includes local scene information of at least one local area in the image to be processed; and determining the motion information of the image to be processed based on the first intermediate motion information includes: determining second intermediate motion information according to a difference between the image to be processed and the reference image and local scene information of the local area; The first intermediate motion information and the second intermediate motion information are fused to obtain motion information of the image to be processed.

5. The method according to claim 4, characterized in that The determining the second intermediate motion information according to the difference between the image to be processed and the reference image and the local scene information of the local area includes: Determining corresponding local motion discrimination information according to the local scene information of the local area; The second intermediate motion information is determined according to the difference between the image to be processed and the reference image and the local motion discrimination information.

6. The method according to claim 1, characterized in that The scene information includes local scene information of at least one local area in the image to be processed; and determining the motion information of the image to be processed according to the inertial sensor data and the scene information includes: Determining global motion information of the image to be processed according to the inertial sensing data; Adjusting the global motion information based on the local scene information of the local area to obtain first intermediate motion information; The motion information of the image to be processed is determined based on the first intermediate motion information.

7. The method according to claim 1, characterized in that The motion information of the image to be processed includes the motion degree value of each pixel in the image to be processed; the fusing the image to be processed with the reference image based on the motion information of the image to be processed to obtain a denoised image corresponding to the image to be processed includes: According to the motion degree value of each pixel point, the fusion ratio corresponding to each pixel point is determined; and based on the fusion ratio, the image to be processed and the reference image are fused at the pixel level to obtain a denoised image.

8. An image denoising processing device, characterized in that: include: An image acquisition module is configured to acquire an image to be processed, and acquire inertial sensor data corresponding to the image to be processed and scene information of the image to be processed; a motion information determination module, configured to determine the motion information of the image to be processed according to the inertial sensing data and the scene information; An image fusion module is configured to fuse the image to be processed with a reference image based on the motion information to obtain a denoised image corresponding to the image to be processed; the reference image is an image captured at a different time and with respect to the same shooting scene as the image to be processed; The scene information includes global scene information of the image to be processed and / or local scene information of at least one local area in the image to be processed; and determining the motion information of the image to be processed based on the inertial sensor data and the scene information includes at least one of the following methods: Determine corresponding global motion discrimination information according to the global scene information; determine first intermediate motion information according to the relationship between the inertial sensor data and the global motion discrimination information; determine motion information of the image to be processed based on the first intermediate motion information; Determine the global motion information of the image to be processed according to the inertial sensor data; adjust the global motion information based on the local scene information of the local area to obtain first intermediate motion information; determine the motion information of the image to be processed based on the first intermediate motion information.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 7 by executing the executable instructions.

Images