Image processing method, apparatus and electronic device

Abstract

The present disclosure provides an image processing method, device and electronic equipment, wherein the method comprises: obtaining a current frame image and a first filtered image corresponding to a previous frame image of the current frame image; based on a trained motion estimation model, generating a first motion estimation image according to the current frame image and the first filtered image; performing difference operation on the current frame image and the first filtered image to obtain a second motion estimation image; generating a third motion estimation image according to the first motion estimation image and the second motion estimation image; and performing time domain filtering processing on the current frame image according to the third motion estimation image to obtain a second filtered image corresponding to the current frame image. Thus, the first motion estimation image and the second motion estimation image are combined to generate the third motion estimation image, which improves the accuracy of motion estimation, and the time domain filtering based on the third motion estimation image can improve the image quality of the time domain filtering.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to an image processing method, apparatus and electronic device. Background Technology

[0002] In image processing, temporal filtering (temporal denoising) is a commonly used technique. Temporal filtering involves using multiple images (usually past frames) obtained at different times to denoise the current frame. Because temporal filtering has certain application conditions—that is, the multiple images (or parts of images) used for denoising must remain relatively still compared to the current frame (or part of images)—if the temporal images used for filtering do not meet this requirement, noticeable ghosting or ghosting will appear in the filtered result.

[0003] Currently, related technologies use motion estimation algorithms from Image Signal Processing (ISP) for temporal filtering. However, due to limitations such as small receptive field and low processing complexity, motion estimation algorithms in image signal processing cannot accurately estimate motion. This may result in image quality issues such as ghosting or ghosting in moving areas, or excessive noise in non-moving areas due to the inability to apply temporal filtering. Summary of the Invention

[0004] This disclosure aims to at least partially address one of the technical problems in the related art.

[0005] To this end, this disclosure proposes an image processing method, apparatus, and electronic device that can combine a first motion estimation map generated by a motion estimation model with a second motion estimation map generated by differential operation between the current frame image and the first filtered image to generate a third motion estimation map, thereby improving the accuracy of motion estimation. Based on the third motion estimation map, temporal filtering can be performed to improve the image quality of temporal filtering.

[0006] According to a first aspect of this disclosure, an image processing method is provided, comprising: acquiring a current frame image and a first filtered image corresponding to a previous frame image of the current frame image; generating a first motion estimation map based on a trained motion estimation model, according to the current frame image and the first filtered image; performing a difference operation on the current frame image and the first filtered image to obtain a second motion estimation map; generating a third motion estimation map based on the first motion estimation map and the second motion estimation map; and performing temporal filtering processing on the current frame image based on the third motion estimation map to obtain a second filtered image corresponding to the current frame image.

[0007] According to a second aspect of the present disclosure, an image processing apparatus is provided, comprising: an acquisition module, configured to acquire a current frame image and a first filtered image corresponding to a previous frame image of the current frame image; a first generation module, configured to generate a first motion estimation map based on a trained motion estimation model, according to the current frame image and the first filtered image; a second generation module, configured to perform a difference operation on the current frame image and the first filtered image to obtain a second motion estimation map; a third generation module, configured to generate a third motion estimation map based on the first motion estimation map and the second motion estimation map; and a processing module, configured to perform temporal filtering processing on the current frame image according to the third motion estimation map to obtain a second filtered image corresponding to the current frame image.

[0008] According to a third aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform an image processing method according to an embodiment of the first aspect of this disclosure.

[0009] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided that, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the image processing method proposed in the first aspect of this disclosure.

[0010] According to a fifth aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor of an electronic device, enables the electronic device to perform the image processing method proposed in the first aspect embodiment.

[0011] The technical solution disclosed herein obtains the current frame image and a first filtered image corresponding to the previous frame image of the current frame image; generates a first motion estimation map based on a trained motion estimation model, according to the current frame image and the first filtered image; performs a difference operation on the current frame image and the first filtered image to obtain a second motion estimation map; generates a third motion estimation map based on the first motion estimation map and the second motion estimation map; and performs temporal filtering on the current frame image based on the third motion estimation map to obtain a second filtered image corresponding to the current frame image. Thus, by combining the first motion estimation map generated by the motion estimation model and the second motion estimation map generated by the difference operation between the current frame image and the first filtered image, a third motion estimation map is generated, improving the accuracy of motion estimation. Therefore, temporal filtering based on the third motion estimation map can improve the image quality of temporal filtering.

[0012] Additional aspects and advantages of this disclosure will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this disclosure. Attached Figure Description

[0013] The above and / or additional aspects and advantages of this disclosure will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, in which:

[0014] Figure 1 This is a flowchart illustrating an image processing method according to an exemplary embodiment;

[0015] Figure 2 This is a flowchart illustrating an image processing method according to an exemplary embodiment;

[0016] Figure 3 This is a flowchart illustrating an image processing method according to an exemplary embodiment;

[0017] Figure 4 This is a flowchart illustrating an image processing method according to an exemplary embodiment;

[0018] Figure 5 This is a flowchart illustrating an image processing method according to an exemplary embodiment;

[0019] Figure 6 This is a schematic diagram illustrating an image processing system implementing an image processing method according to an exemplary embodiment;

[0020] Figure 7 This is a schematic diagram of the structure of an image processing apparatus according to an exemplary embodiment;

[0021] Figure 8 This is a block diagram illustrating an electronic device for image processing according to an exemplary embodiment. Detailed Implementation

[0022] Embodiments of this disclosure are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this disclosure, and should not be construed as limiting this disclosure.

[0023] It's important to understand that temporal filtering requires motion estimation. This involves comparing the image information of the current frame with that of other frames to obtain a motion estimation map. The pixel values ​​(grayscale values) in the motion estimation map represent whether motion exists at that location in the current frame and other frames. If motion exists, temporal filtering cannot be applied; if no motion exists, temporal filtering can be applied.

[0024] However, due to limitations in hardware cost, design complexity, and difficulty, motion estimation in traditional ISPs can hardly guarantee accurate motion detection. Especially in low-light, high-noise environments, traditional algorithms, limited by small receptive fields and low processing complexity, cannot accurately estimate motion. This can ultimately lead to image quality issues such as ghosting in moving areas or excessive noise in non-moving areas due to the inability to apply temporal filtering.

[0025] To address the aforementioned problems, this disclosure proposes an image processing method, apparatus, and electronic device.

[0026] The following detailed description of an image processing method, apparatus, and electronic device provided in this disclosure is provided in conjunction with the accompanying drawings.

[0027] Figure 1 This is a schematic flowchart illustrating an image processing method according to an exemplary embodiment. It should be noted that this image processing method can be applied to an image processing apparatus. This image processing apparatus can be configured in an electronic device. The electronic device can be a mobile terminal, such as a mobile phone, tablet computer, personal digital assistant, or other hardware device with various operating systems.

[0028] like Figure 1 As shown, the image processing method includes the following steps:

[0029] Step 101: Obtain the current frame image and the first filtered image corresponding to the previous frame image of the current frame image.

[0030] In this embodiment of the disclosure, the current frame image may be the image captured by the image acquisition device at the current moment, and the previous frame image may be the image captured by the image acquisition device at the previous moment.

[0031] As an example, when the previous frame is the first frame, any temporal filtering algorithm in the relevant technology can be used to perform temporal filtering on the previous frame to obtain the first filtered image of the previous frame. When the previous frame is not the first frame, for example, the previous frame is the (t-1)th frame, a first motion estimation map can be generated based on the trained motion estimation model and the filtered images of the (t-1)th frame and the (t-2)th frame. A difference operation is performed on the filtered images of the (t-1)th frame and the (t-2)th frame to obtain a second motion estimation map. Then, a third motion estimation map is generated based on the first and second motion estimation maps. Based on the third motion estimation map, temporal filtering is performed on the (t-1)th frame to obtain the filtered image corresponding to the (t-1)th frame.

[0032] Step 102: Based on the trained motion estimation model, generate a first motion estimation map according to the current frame image and the first filtered image.

[0033] In this embodiment of the disclosure, in order to obtain the motion estimation map of the current frame image, the motion estimation map corresponding to the current image can be generated based on the current frame image and the first filtered image.

[0034] As an example, the current frame image and the first filtered image can be input into a trained motion estimation model, which can output a first motion estimation map. The trained motion estimation model has learned the correspondence between the current frame image, the first filtered image, and the motion estimation map.

[0035] It should be noted that before inputting the current frame image and the first filtered image into the trained motion estimation model, the motion estimation model can be trained so that the trained motion estimation model has learned the correspondence between the current frame image, the first filtered image and the motion estimation map.

[0036] As an example, the following steps are taken: obtaining a sample frame image, a first sample filtered image corresponding to the previous sample image, and a sample motion estimation map of the sample frame image; inputting the sample frame image and the first sample filtered image into an initial motion estimation model to obtain a predicted motion estimation map; generating a loss function value based on the difference between the predicted motion estimation map and the sample estimation map; and training the initial motion estimation model based on the loss function value to obtain a trained motion estimation model.

[0037] In this embodiment of the disclosure, a sample frame image, a first sample filtered image corresponding to the previous sample image of the sample frame image, and a sample motion estimation map of the sample frame image can be obtained from the database. Then, the sample frame image and the first sample filtered image are input into an initial motion estimation model. The initial motion estimation model can output a predicted motion estimation map. Then, based on the difference between the predicted estimation map and the sample estimation map, a loss function value is generated. Based on the loss function value, the coefficients of the initial motion estimation model are adjusted to obtain a trained motion estimation model.

[0038] Step 103: Perform a difference operation on the current frame image and the first filtered image to obtain the second motion estimation map.

[0039] As another example, the current frame image and the first filtered image are differentially processed pixel by pixel to obtain the pixel value difference between each pixel in the current frame image and the corresponding pixel in the first filtered image. Then, based on the pixel value difference, the second motion estimation map is obtained.

[0040] Step 104: Generate a third motion estimation map based on the first motion estimation map and the second motion estimation map.

[0041] To improve the accuracy of motion estimation, the first and second motion estimation maps can be combined to obtain a third motion estimation map.

[0042] Step 105: Based on the third motion estimation map, perform temporal filtering on the current frame image to obtain the second filtered image corresponding to the current frame image.

[0043] To improve the image quality of temporal filtering, the current frame image can be processed by temporal filtering based on the third motion estimation map to obtain the second filtered image corresponding to the current frame image.

[0044] In summary, by acquiring the current frame image and the first filtered image corresponding to the previous frame image, and generating a first motion estimation map based on the trained motion estimation model, a second motion estimation map is obtained by performing a difference operation on the current frame image and the first filtered image. A third motion estimation map is then generated based on the first and second motion estimation maps. Finally, a second filtered image is obtained by performing temporal filtering on the current frame image based on the third motion estimation map. Thus, by combining the first motion estimation map generated by the motion estimation model with the second motion estimation map generated by the difference operation on the first filtered image, the accuracy of motion estimation is improved. Therefore, temporal filtering based on the third motion estimation map can improve the image quality of temporal filtering.

[0045] To improve the accuracy of motion estimation, such as Figure 2 As shown, Figure 2 This is a schematic flowchart illustrating an image processing method according to an exemplary embodiment. In this embodiment, a first motion estimation map and a second motion estimation map can be combined to generate a third motion estimation map. Figure 2 The illustrated embodiment may include the following steps:

[0046] Step 201: Obtain the current frame image and the first filtered image corresponding to the previous frame image of the current frame image.

[0047] Step 202: Based on the trained motion estimation model, generate a first motion estimation map according to the current frame image and the first filtered image.

[0048] Step 203: Perform a difference operation on the current frame image and the first filtered image to obtain the second motion estimation map.

[0049] Step 204: Time-align the first motion estimation map and the second motion estimation map to obtain the time-aligned first motion map and the time-aligned second motion map.

[0050] In this embodiment of the disclosure, since the computational logic complexity and computational amount of the first motion estimation map obtained according to the motion estimation model and the second motion estimation map obtained by performing differential operation on the current frame image and the first filtered image are quite different, it may lead to different computational delays between the first motion estimation map obtained according to the motion estimation model and the second motion estimation map obtained by performing differential operation on the current frame image and the first filtered image. Therefore, in order to improve the accuracy of motion estimation, the temporal relationship between the first motion estimation map and the second motion estimation map can be aligned, that is, the first motion estimation map and the second motion estimation map are temporally aligned to obtain the temporally aligned first motion estimation map and the temporally aligned second motion estimation map.

[0051] Step 205: For any pixel in the first motion map after temporal alignment, determine the gray value of the corresponding pixel in the third motion estimation map based on the average gray value of the corresponding pixel in the second motion map after temporal alignment.

[0052] As an example, the grayscale value of any pixel in the time-aligned first motion map and the mean of the grayscale values ​​of the corresponding pixel in the time-aligned second motion map can be obtained, and the mean of the grayscale values ​​of any pixel can be used as the grayscale value of the corresponding pixel in the third motion estimation map.

[0053] For example, taking the first motion estimation map after time alignment as motion estimation map A, the second motion estimation map after time alignment as motion estimation map B, and the third motion estimation map as motion estimation map C, the gray value of the corresponding pixel in the third motion estimation map can be determined based on the average gray value of any pixel in the first motion estimation map after time alignment and the gray value of the corresponding pixel in the second motion estimation map after time alignment. This can be expressed by the following formula:

[0054] Motion estimation map C = (motion estimation map A + motion estimation map B) / 2.

[0055] Step 206: Based on the third motion estimation map, perform temporal filtering on the current frame image to obtain the second filtered image corresponding to the current frame image.

[0056] It should be noted that the execution process of steps 201 to 203 and step 206 can be implemented in any of the embodiments of this disclosure. This disclosure does not limit this and will not elaborate further.

[0057] In summary, by temporally aligning the first and second motion estimation maps, we obtain a temporally aligned first motion map and a temporally aligned second motion map. For any pixel in the temporally aligned first motion map, the gray value of the corresponding pixel in the third motion estimation map is determined based on the average gray value of the corresponding pixel in the temporally aligned second motion map. Thus, by combining the first and second motion estimation maps, we obtain the third motion estimation map, which improves the accuracy of motion estimation. Consequently, temporal filtering based on the third motion estimation map can improve the image quality of temporal filtering.

[0058] To improve the accuracy of motion estimation, such as Figure 3 As shown, Figure 3 This is a schematic flowchart illustrating an image processing method according to an exemplary embodiment. In this embodiment, a first motion estimation map and a second motion estimation map can be combined to generate a third motion estimation map. Figure 3 The illustrated embodiment may include the following steps:

[0059] Step 301: Obtain the current frame image and the first filtered image corresponding to the previous frame image of the current frame image.

[0060] Step 302: Based on the trained motion estimation model, generate a first motion estimation map according to the current frame image and the first filtered image.

[0061] Step 303: Perform a difference operation on the current frame image and the first filtered image to obtain the second motion estimation map.

[0062] Step 304: Time-align the first motion estimation map and the second motion estimation map to obtain the time-aligned first motion map and the time-aligned second motion map.

[0063] In this embodiment of the disclosure, since the computational logic complexity and computational amount of the first motion estimation map obtained according to the motion estimation model and the second motion estimation map obtained by performing differential operation on the current frame image and the first filtered image are quite different, it may lead to different computational delays between the first motion estimation map obtained according to the motion estimation model and the second motion estimation map obtained by performing differential operation on the current frame image and the first filtered image. Therefore, in order to improve the accuracy of motion estimation, the temporal relationship between the first motion estimation map and the second motion estimation map can be aligned, that is, the first motion estimation map and the second motion estimation map are temporally aligned to obtain the temporally aligned first motion estimation map and the temporally aligned second motion estimation map.

[0064] Step 305: For any pixel in the first motion map after temporal alignment, determine the gray value of the corresponding pixel in the third motion estimation map based on the maximum value of the gray value of the pixel and the corresponding pixel in the second motion map after temporal alignment.

[0065] As an example, the maximum value of the grayscale value of any pixel in the first motion map after temporal alignment and the corresponding grayscale value in the second motion map after temporal alignment can be obtained, and the maximum value of the grayscale value corresponding to any pixel can be used as the grayscale value of the corresponding pixel in the third motion estimation map.

[0066] For example, taking the first motion estimation map after time alignment as motion estimation map A, the second motion estimation map after time alignment as motion estimation map B, and the third motion estimation map as motion estimation map C, the gray value of the corresponding pixel in the third motion estimation map is determined based on the maximum gray value of the corresponding pixel in the second motion estimation map after time alignment. This can be expressed by the following formula:

[0067] Motion estimation map C = Max(motion estimation map A, motion estimation map B).

[0068] Step 306: Based on the third motion estimation map, perform temporal filtering on the current frame image to obtain the second filtered image corresponding to the current frame image.

[0069] It should be noted that the execution process of steps 301 to 303 and step 306 can be implemented in any of the embodiments of this disclosure. This disclosure does not limit this and will not elaborate further.

[0070] In summary, by temporally aligning the first and second motion estimation maps, we obtain a temporally aligned first motion map and a temporally aligned second motion map. For any pixel in the temporally aligned first motion map, the gray value of the corresponding pixel in the third motion estimation map is determined based on the maximum gray value of the corresponding pixel in the temporally aligned second motion map. Thus, by combining the first and second motion estimation maps, we obtain the third motion estimation map, which improves the accuracy of motion estimation. Consequently, temporal filtering based on the third motion estimation map can improve the image quality of temporal filtering.

[0071] To improve the image quality of temporal filtering, such as Figure 4 As shown, Figure 4 This is a schematic flowchart illustrating an image processing method according to an exemplary embodiment. In this embodiment, temporal filtering processing can be performed on the current frame image based on a third motion estimation map. Figure 4 The illustrated embodiment may include the following steps:

[0072] Step 401: Obtain the current frame image and the first filtered image corresponding to the previous frame image of the current frame image.

[0073] Step 402: Based on the trained motion estimation model, generate a first motion estimation map according to the current frame image and the first filtered image.

[0074] Step 403: Perform a difference operation on the current frame image and the first filtered image to obtain the second motion estimation map.

[0075] Step 404: Generate a third motion estimation map based on the first motion estimation map and the second motion estimation map.

[0076] Step 405: For any target pixel in the current frame image, determine the motion weight of any target pixel based on the gray value of the corresponding pixel in the third motion estimation map.

[0077] In this embodiment of the present disclosure, the gray value of any target pixel in the current frame image corresponding to the pixel in the third motion estimation map can be in the range of [0.0, 1.0]. Here, 0.0 can indicate that the target pixel is not moving, and 1.0 can indicate that the target pixel is moving. That is, the gray value of the pixel in the third motion estimation map can represent the motion probability of the corresponding pixel in the current frame image. In other words, the gray value of any target pixel in the current frame image corresponding to the pixel in the third motion estimation map can represent the motion weight of any target pixel.

[0078] Step 406: Based on the set temporal filtering coefficients, the pixel value of any target pixel and the pixel value of the corresponding pixel in the first filtered image are weighted and summed to obtain the fused pixel value.

[0079] To improve the accuracy of temporal filtering, the current frame image can be fused with the first filtered image. As an example, the pixel value of any target pixel and the pixel value of the corresponding pixel in the first filtered image can be weighted and summed according to the set temporal filtering coefficients to obtain the fused pixel value.

[0080] Step 407: Based on the motion weights, the pixel value of any target pixel and the fused pixel value are weighted and summed to obtain the pixel value of the corresponding pixel in the second filtered image.

[0081] To improve the accuracy of temporal filtering, the current frame image is fused with the first filtered image. Simultaneously, considering the motion information from the third motion estimation map, temporal filtering is applied to the current frame image (excluding motion regions within the current frame image), generating a second filtered image of the current frame. This can be expressed as the following formula:

[0082] O_N=I_N*Mot+(I_N*P+O_N_1*(1.0-P))*(1.0-Mot);

[0083] Wherein, O_N represents the second filtered image, I_N represents the current frame image, O_N_1 represents the first filtered image, P represents the set temporal filtering coefficient, and Mot represents the third motion estimation image. The gray value of the pixel in Mot ranges from 0.0 to 1.0, where 0.0 can represent no motion and 1.0 can represent motion.

[0084] It should be noted that the execution process of steps 401 to 404 can be implemented in any of the embodiments of this disclosure. This disclosure does not limit this and will not elaborate further.

[0085] In summary, for any target pixel in the current frame image, the motion weight of the target pixel is determined based on the grayscale value of the corresponding pixel in the third motion estimation map. Based on the set temporal filtering coefficients, the pixel value of the target pixel and the pixel values ​​of the corresponding pixels in the first filtered image are weighted and summed to obtain the fused pixel value. Based on the motion weight, the pixel value of the target pixel and the fused pixel value are weighted and summed to obtain the pixel value of the corresponding pixel in the second filtered image. Thus, the current frame image is fused with the first filtered image, and considering the motion information of the third motion estimation map, temporal filtering of the current frame image can improve the image quality.

[0086] To clearly illustrate how the current frame image and the first filtered image are differentially processed in the above embodiments to obtain the second motion estimation map, this disclosure proposes another image processing method.

[0087] Figure 5 This is a flowchart illustrating an image processing method according to an exemplary embodiment, such as... Figure 5 As shown, the image processing method may include the following steps:

[0088] Step 501: Obtain the current frame image and the first filtered image corresponding to the previous frame image of the current frame image.

[0089] Step 502: Based on the trained motion estimation model, generate a first motion estimation map according to the current frame image and the first filtered image.

[0090] Step 503: For any target pixel in the current frame image, perform a difference operation between the target pixel and the corresponding pixel in the first filtered image to obtain the pixel value difference between the target pixel and the corresponding pixel in the first filtered image.

[0091] In this embodiment of the disclosure, the current frame image and the first filtered image can be compared pixel by pixel to determine the pixel value difference between each pixel in the current frame image and the corresponding pixel in the first filtered image. As an example, for any target pixel in the current frame image, a difference operation can be performed between the target pixel and the corresponding pixel in the first filtered image to obtain the pixel value difference between the target pixel and the corresponding pixel in the first filtered image.

[0092] Step 504: When the pixel value difference corresponding to any target pixel is greater than the set difference threshold, query the set motion estimation list according to the pixel value difference corresponding to any target pixel to obtain the gray value of the corresponding pixel in the second motion estimation map.

[0093] The motion estimation list includes the correspondence between pixel value differences and the grayscale values ​​of the corresponding pixels in the motion estimation map.

[0094] Next, the pixel value difference corresponding to any target pixel is compared with a set difference threshold. When the pixel value difference corresponding to any target pixel is greater than the set difference threshold, the pixel value difference corresponding to any target pixel can be queried to a set motion estimation list to obtain the gray value corresponding to each pixel in the second motion estimation image. The motion estimation list includes the correspondence between the pixel value difference and the gray value of the corresponding pixel in the motion estimation image. The gray value corresponding to each pixel in the second motion estimation image can represent the motion probability of the corresponding pixel in the current frame image.

[0095] Step 505: Generate a third motion estimation map based on the first motion estimation map and the second motion estimation map.

[0096] Step 506: Based on the third motion estimation map, perform temporal filtering on the current frame image to obtain the second filtered image corresponding to the current frame image.

[0097] It should be noted that the execution processes of steps 501 to 502 and steps 505 to 506 can be implemented in any of the embodiments of this disclosure. This disclosure does not limit these processes and will not elaborate further.

[0098] In summary, by performing a difference operation on any target pixel in the current frame image and its corresponding pixel in the first filtered image, the pixel value difference between the target pixel and its corresponding pixel in the first filtered image is obtained. When the pixel value difference corresponding to any target pixel is greater than a set difference threshold, a set motion estimation list is queried based on the pixel value difference to obtain the grayscale value of the corresponding pixel in the second motion estimation image. The motion estimation list includes the correspondence between the pixel value difference and the grayscale value of the corresponding pixel in the motion estimation image. Thus, by performing a difference operation on the current frame image and the first filtered image, the second motion estimation image can be effectively obtained.

[0099] To illustrate the above embodiments more clearly, examples are given below.

[0100] For example, such as Figure 6 As shown, the image processing method can be implemented by an image processing system, which may include: an image sensor M01, a neural network motion estimation module M02, a traditional ISP motion estimation module M03, a motion estimation map integration module M04, a traditional ISP temporal filtering module M05, a storage module M06, and a storage module M07.

[0101] The image sensor M01 is typically a semiconductor device, commonly including Complementary Metal Oxide Semiconductor (CMOS) image sensors and Charge Coupled Device (CCD) image sensors. Its function is to convert the light signal passing through the lens into a digital signal, which is then input to the Image Sensor (ISP) for imaging.

[0102] The neural network motion estimation module M02 is typically a semiconductor device. Common examples include integrated circuits with a fixed design for a specific neural network, or more general-purpose network processors (NPUs), central processing units (CPUs), digital signal processors (DSPs), and graphics processing units (GPUs). The function of M02 is to obtain a motion estimation map M0 from the motion estimation network. The pixel values ​​in this image represent the probability that a pixel is moving.

[0103] A traditional ISP motion estimation module M03 is typically a semiconductor integrated circuit. Its function is to obtain a motion estimation map M1 through embedded motion estimation logic. The pixel values ​​of this image represent the probability that the pixel is in motion.

[0104] The motion estimation map integration module M04 is typically a semiconductor integrated circuit. Its function is to integrate the motion estimation maps obtained from the neural network motion estimation module M02 and the traditional ISP motion estimation module M03 to obtain an integrated motion estimation map.

[0105] The traditional ISP temporal filtering module M05 is typically a semiconductor integrated circuit. Its function is to read the output image of the previous frame and the input image of the current frame, and then blend the two images to achieve temporal filtering. Simultaneously, this module also reads the motion estimation map obtained from the motion estimation map integration module M04. In areas of motion within the image, the temporal filtering effect is not applied; in areas without motion, the temporal filtering effect is applied.

[0106] Storage unit M06 is typically a static random-access memory (SRAM), which serves to buffer the time difference (timing alignment) between the output of the neural network motion estimation module M02 and the output of the traditional ISP motion estimation module M03.

[0107] An image processing system implementing an image processing method may include the following steps:

[0108] 1. Image sensor M01 converts optical signals into digital signals, generating the currently input image signal. Image sensor M01 stores I_N (the currently input image) into storage unit M05;

[0109] 2. The neural network motion estimation module M02 reads the currently input image I_N and the result of temporal filtering of the previous frame O_N_1 from the storage unit M05. Based on these two images (I_N and O_N_1), the motion estimation map A is obtained;

[0110] 3. The traditional ISP motion estimation module M03 reads the currently input image I_N and the temporal filtering record O_N_1 of the previous frame from the storage unit M05. Based on these two images, a motion estimation image B is obtained. The motion estimation algorithm can calculate the difference between the corresponding pixels of I_N and O_N_1 and compare it with a pre-set difference threshold to obtain an estimate of whether it is motion.

[0111] 4. Due to the significant difference in computational complexity between steps 2 and 3, the computational load may vary considerably, resulting in different computational delays. To align the timing relationships between motion estimation graph A and motion estimation graph B, storage unit M06 stores motion estimation graph A and motion estimation graph B, and outputs them to motion estimation graph integration module M04 after timing alignment.

[0112] 5. The motion estimation map integration module M04 integrates motion estimation map A and motion estimation map B to obtain the integrated motion estimation map C. Common integration methods include the following:

[0113] Motion estimation map C = (Motion estimation map A + Motion estimation map B) / 2;

[0114] Motion estimation graph C = Max(motion estimation graph A, motion estimation graph B);

[0115] 6. The traditional ISP temporal filtering module M05 reads images I_N and O_N_1 and fuses them to obtain the temporal filtering result O_N. Simultaneously considering the information from the motion estimation map, the temporal filtering result is not applied to motion regions. Assuming the value range of the motion estimation map Mot is 0.0 to 1.0, where 0.0 represents no motion and 1.0 represents motion, the following formula can be obtained:

[0116] O_N=I_N*Mot+(I_N*P+O_N_1*(1.0-P))*(1.0-Mot);

[0117] In the above formula, P represents the coefficients of the time-domain filter, which can be configured in advance.

[0118] 7. The traditional ISP time-domain filtering module M05 stores O_N into the storage unit M07.

[0119] Repeat steps 1 through 7 to generate the output image for each frame.

[0120] The image processing method of this disclosure acquires a current frame image and a first filtered image corresponding to the previous frame image of the current frame image; generates a first motion estimation map based on a trained motion estimation model, according to the current frame image and the first filtered image; performs a difference operation on the current frame image and the first filtered image to obtain a second motion estimation map; generates a third motion estimation map based on the first motion estimation map and the second motion estimation map; and performs temporal filtering on the current frame image based on the third motion estimation map to obtain a second filtered image corresponding to the current frame image. Thus, by combining the first motion estimation map generated by the motion estimation model with the second motion estimation map generated by the difference operation on the first filtered image to generate the third motion estimation map, the accuracy of motion estimation is improved. Therefore, temporal filtering based on the third motion estimation map can improve the image quality of temporal filtering.

[0121] To implement the above embodiments, this disclosure also proposes an image processing apparatus.

[0122] Figure 7 This is a schematic diagram of the structure of an image processing apparatus according to an exemplary embodiment.

[0123] like Figure 7 As shown, the image processing device 700 includes: an acquisition module 710, a first generation module 720, a second generation module 730, a third generation module 740, and a processing module 750.

[0124] The acquisition module 710 is used to acquire the current frame image and the first filtered image corresponding to the previous frame image of the current frame image; the first generation module 720 is used to generate a first motion estimation map based on the trained motion estimation model, the current frame image, and the first filtered image; the second generation module 730 is used to perform a difference operation on the current frame image and the first filtered image to obtain a second motion estimation map; the third generation module 740 is used to generate a third motion estimation map based on the first motion estimation map and the second motion estimation map; and the processing module 750 is used to perform temporal filtering processing on the current frame image based on the third motion estimation map to obtain the second filtered image corresponding to the current frame image.

[0125] As one possible implementation of this disclosure, the third generation module 740 is configured to: perform temporal alignment on the first motion estimation map and the second motion estimation map to obtain a temporally aligned first motion map and a temporally aligned second motion map; for any pixel in the temporally aligned first motion map, determine the gray value of the corresponding pixel in the third motion estimation map based on the average gray value of the corresponding pixel in the temporally aligned second motion map.

[0126] As one possible implementation of this disclosure, the third generation module 740 is configured to: perform temporal alignment on the first motion estimation map and the second motion estimation map to obtain a temporally aligned first motion map and a temporally aligned second motion map; for any pixel in the temporally aligned first motion map, determine the gray value of the corresponding pixel in the third motion estimation map based on the maximum value of the gray value of the corresponding pixel in the temporally aligned second motion map.

[0127] As one possible implementation of this disclosure, the processing module 750 is configured to: for any target pixel in the current frame image, determine the motion weight of any target pixel based on the grayscale value of the corresponding pixel in the third motion estimation image; perform a weighted summation of the pixel value of any target pixel and the pixel value of the corresponding pixel in the first filtered image based on a set temporal filtering coefficient to obtain a fused pixel value; and perform a weighted summation of the pixel value of any target pixel and the fused pixel value based on the motion weight to obtain the pixel value of the corresponding pixel in the second filtered image.

[0128] As one possible implementation of this disclosure, the second generation module 730 is configured to: perform a difference operation between any target pixel in the current frame image and the corresponding pixel in the first filtered image to obtain the pixel value difference between the pixel value of the target pixel and the pixel value of the corresponding pixel in the first filtered image; and query a set motion estimation list based on the pixel value difference corresponding to any target pixel to obtain the grayscale value of the corresponding pixel in the second motion estimation image, wherein the motion estimation list includes the correspondence between the pixel value difference and the grayscale value of the corresponding pixel in the motion estimation image.

[0129] As one possible implementation of this disclosure, the motion estimation model is trained using the following modules: a sample acquisition module, a prediction module, a generation module, and a training module.

[0130] The system includes a sample acquisition module for acquiring a sample frame image, a first sample filtered image corresponding to the previous sample frame image, and a sample motion estimation map of the sample frame image; a prediction module for inputting the sample frame image and the first sample filtered image into an initial motion estimation model to obtain a predicted motion estimation map; a generation module for generating a loss function value based on the difference between the predicted motion estimation map and the sample estimation map; and a training module for training the initial motion estimation model based on the loss function value to obtain the trained motion estimation model.

[0131] The image processing apparatus of this embodiment acquires a current frame image and a first filtered image corresponding to the previous frame image of the current frame image; generates a first motion estimation map based on a trained motion estimation model, according to the current frame image and the first filtered image; performs a difference operation on the current frame image and the first filtered image to obtain a second motion estimation map; generates a third motion estimation map based on the first motion estimation map and the second motion estimation map; and performs temporal filtering on the current frame image based on the third motion estimation map to obtain a second filtered image corresponding to the current frame image. Thus, by combining the first motion estimation map generated by the motion estimation model with the second motion estimation map generated by the difference operation on the first filtered image to generate the third motion estimation map, the accuracy of motion estimation is improved. Therefore, temporal filtering based on the third motion estimation map can improve the image quality of temporal filtering.

[0132] To implement the above embodiments, this disclosure also proposes an electronic device, such as... Figure 8 As shown, Figure 8 This is a block diagram illustrating an image processing electronic device according to an exemplary embodiment. Figure 8 As shown, the above-mentioned electronic device 800 may include:

[0133] The system includes a memory 810 and a processor 820, and a bus 830 connecting different components (including the memory 810 and the processor 820). The memory 810 stores a computer program, and when the processor 820 executes the program, it implements the image processing method described in the embodiments of this disclosure.

[0134] Bus 830 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. Examples of these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0135] Electronic device 800 typically includes a variety of computer-readable media. These media can be any available media that can be accessed by electronic device 800, including volatile and non-volatile media, removable and non-removable media.

[0136] The memory 810 may also include computer system readable media in the form of volatile memory, such as random access memory (RAM) 840 and / or cache memory 850. The electronic device 800 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 860 may be used to read and write non-removable, non-volatile magnetic media (…). Figure 8 Not shown; usually referred to as a "hard drive"). Although Figure 8 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 830 via one or more data media interfaces. Memory 810 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this disclosure.

[0137] A program / utility 880 having a set (at least one) of program modules 870 may be stored, for example, in memory 810. Such program modules 870 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 870 typically perform the functions and / or methods described in the embodiments of this disclosure.

[0138] Electronic device 800 can also communicate with one or more external devices 890 (e.g., keyboard, pointing device, display 891, etc.), and with one or more devices that enable a user to interact with the electronic device 800, and / or with any device that enables the electronic device 800 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed through input / output (I / O) interface 892. Furthermore, electronic device 800 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) through network adapter 893. Figure 8 As shown, network adapter 893 communicates with other modules of electronic device 800 via bus 830. It should be understood that, although... Figure 8 As not shown in the diagram, other hardware and / or software modules may be used in conjunction with the electronic device 800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0139] The processor 820 performs various functional applications and data processing by running programs stored in the memory 810.

[0140] It should be noted that the implementation process and technical principles of the electronic device in this embodiment are explained in the foregoing description of the image processing method of the present disclosure embodiment, and will not be repeated here.

[0141] The electronic device provided in this embodiment acquires a current frame image and a first filtered image corresponding to the previous frame image of the current frame image; generates a first motion estimation map based on a trained motion estimation model, according to the current frame image and the first filtered image; performs a difference operation on the current frame image and the first filtered image to obtain a second motion estimation map; generates a third motion estimation map based on the first motion estimation map and the second motion estimation map; and performs temporal filtering on the current frame image based on the third motion estimation map to obtain a second filtered image corresponding to the current frame image. Thus, by combining the first motion estimation map generated by the motion estimation model with the second motion estimation map generated by the difference operation on the first filtered image to generate the third motion estimation map, the accuracy of motion estimation is improved. Therefore, temporal filtering based on the third motion estimation map can improve the image quality of temporal filtering.

[0142] To implement the above embodiments, this disclosure also proposes a computer-readable storage medium.

[0143] When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the image processing method as described above.

[0144] To implement the above embodiments, this disclosure also provides a computer program product that, when executed by the processor of an electronic device, enables the electronic device to perform the image processing method as described above.

[0145] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0146] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that, include: Obtain the current frame image and the first filtered image corresponding to the previous frame image of the current frame image; Based on the trained motion estimation model, a first motion estimation map is generated according to the current frame image and the first filtered image; Perform a difference operation between the current frame image and the first filtered image to obtain a second motion estimation map; A third motion estimation map is generated based on the first motion estimation map and the second motion estimation map; Based on the third motion estimation map, the current frame image is subjected to temporal filtering to obtain the second filtered image corresponding to the current frame image.

2. The method according to claim 1, characterized in that, The step of generating a third motion estimation map based on the first motion estimation map and the second motion estimation map includes: The first motion estimation map and the second motion estimation map are time-aligned to obtain the time-aligned first motion map and the time-aligned second motion map; For any pixel in the first motion map after time alignment, the gray value of the corresponding pixel in the third motion estimation map is determined based on the average gray value of the pixel and the gray value of the corresponding pixel in the second motion map after time alignment.

3. The method according to claim 1, characterized in that, The step of generating a third motion estimation map based on the first motion estimation map and the second motion estimation map includes: The first motion estimation map and the second motion estimation map are time-aligned to obtain the time-aligned first motion map and the time-aligned second motion map; For any pixel in the first motion map after time alignment, the gray value of the corresponding pixel in the third motion estimation map is determined based on the maximum value of the gray value of the pixel and the gray value of the corresponding pixel in the second motion map after time alignment.

4. The method according to claim 1, characterized in that, The step of performing temporal filtering on the current frame image based on the third motion estimation map to obtain the second filtered image corresponding to the current frame image includes: For any target pixel in the current frame image, the motion weight of the target pixel is determined based on the gray value of the corresponding pixel in the third motion estimation map. Based on the set temporal filtering coefficients, the pixel value of any target pixel and the pixel value of the corresponding pixel in the first filtered image are weighted and summed to obtain the fused pixel value. Based on the motion weights, the pixel value of any target pixel is weighted and summed with the fused pixel value to obtain the pixel value of the corresponding pixel in the second filtered image.

5. The method according to any one of claims 1-4, characterized in that, The step of performing a difference operation between the current frame image and the first filtered image to obtain a second motion estimation map includes: For any target pixel in the current frame image, a difference operation is performed between the target pixel and the corresponding pixel in the first filtered image to obtain the pixel value difference between the target pixel and the corresponding pixel in the first filtered image. When the pixel value difference corresponding to any target pixel is greater than a set difference threshold, a set motion estimation list is queried based on the pixel value difference corresponding to any target pixel to obtain the gray value of the corresponding pixel in the second motion estimation map. The motion estimation list includes the correspondence between the pixel value difference and the gray value of the corresponding pixel in the motion estimation map.

6. The method according to any one of claims 1-4, characterized in that, The motion estimation model is trained using the following steps: Acquire a sample frame image, a first sample filtered image corresponding to the previous sample frame image of the sample frame image, and a sample motion estimation map of the sample frame image; The sample frame image and the first sample filtered image are input into the initial motion estimation model to obtain the predicted motion estimation map; Based on the difference between the predicted motion estimation map and the sample estimation map, a loss function value is generated; The initial motion estimation model is trained based on the loss function value to obtain the trained motion estimation model.

7. An image processing apparatus, characterized in that, include: The acquisition module is used to acquire the current frame image and the first filtered image corresponding to the previous frame image of the current frame image; The first generation module is used to generate a first motion estimation map based on the trained motion estimation model, the current frame image, and the first filtered image. The second generation module is used to perform a difference operation on the current frame image and the first filtered image to obtain a second motion estimation map; The third generation module is used to generate a third motion estimation map based on the first motion estimation map and the second motion estimation map; The processing module is used to perform temporal filtering on the current frame image based on the third motion estimation map to obtain a second filtered image corresponding to the current frame image.

8. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the image processing method according to any one of claims 1-6.

9. A computer-readable storage medium, wherein instructions in the computer-readable storage medium, when executed by a processor of an electronic device, enable the electronic device to perform the image processing method as described in any one of claims 1-6.

10. A computer program product comprising a computer program that, when executed by a processor of an electronic device, enables the electronic device to perform the image processing method as described in any one of claims 1-6.

Images