Image processing methods and apparatus

Abstract

This application discloses an image processing method and apparatus, belonging to the field of image processing technology. The image processing method includes: acquiring a first image frame and a second image frame of a target video; wherein the first image frame and the second image frame are adjacent image frames, and the first image frame is the preceding image frame of the second image frame; acquiring target information; wherein the target information includes: first high-frequency information at a first moment and second high-frequency information at a second moment in the original image of the target video; and performing noise reduction processing on the second image frame based on the first image frame and the target information.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to an image processing method and device thereof. BACKGROUND

[0002] With the continuous development of terminal technology, video shooting has become an essential function of terminal equipment, and image shooting quality is a key indicator for evaluating the shooting function of the terminal.

[0003] To improve the image shooting quality, in a shooting scene with high image noise, the terminal equipment can automatically perform noise reduction processing during shooting. For example, in a night video shooting scene, due to low ambient brightness, the image sensor noise of the terminal equipment is large, and the image signal processor (ISP) can perform online noise reduction processing on the collected image through a noise reduction algorithm.

[0004] However, the noise reduction processing during shooting is limited by factors such as device performance, power consumption, and heat generation, and cannot use artificial intelligence models with large computing power, so the noise reduction effect needs to be improved. Although the offline noise reduction method can use artificial intelligence models with large computing power, the input of the model is generally a video processed by the image signal processor, and the high-frequency information of the image is lost, and the definition is insufficient, so the offline noise reduction effect is limited by the quality of the input image. SUMMARY

[0005] The purpose of the embodiments of the present application is to provide an image processing method and device, which can solve the problem of improving the noise reduction effect of the video image in the prior art.

[0006] In a first aspect, the embodiments of the present application provide an image processing method, which comprises:

[0007] obtaining a first image frame and a second image frame of a target video; wherein the first image frame and the second image frame are adjacent image frames, and the first image frame is a previous image frame of the second image frame;

[0008] obtaining target information; wherein the target information includes first high-frequency information at a first time and second high-frequency information at a second time in an original image of the target video, the first time is a time corresponding to the first image frame, and the second time is a time corresponding to the second image frame;

[0009] performing noise reduction processing on the second image frame according to the first image frame and the target information.

[0010] In a second aspect, the embodiments of the present application provide an image processing device, which comprises:

[0011] The first obtaining module is configured to obtain a first image frame and a second image frame of a target video; wherein the first image frame and the second image frame are adjacent image frames, and the first image frame is a previous image frame of the second image frame.

[0012] The second obtaining module is configured to obtain target information; wherein the target information comprises first high-frequency information at a first time and second high-frequency information at a second time in an original image of the target video, the first time is a time corresponding to the first image frame, and the second time is a time corresponding to the second image frame.

[0013] The first noise processing module is configured to perform noise reduction processing on the second image frame according to the first image frame and the target information.

[0014] In a third aspect, an electronic device is provided, which comprises a processor and a memory, the memory stores programs or instructions executable on the processor, and the programs or instructions are executed by the processor to implement the steps of the image processing method according to the first aspect.

[0015] In a fourth aspect, a readable storage medium is provided, the readable storage medium stores programs or instructions, and the programs or instructions are executed by a processor to implement the steps of the image processing method according to the first aspect.

[0016] In a fifth aspect, a chip is provided, the chip comprises a processor and a communication interface, the communication interface and the processor are coupled, the processor is configured to run programs or instructions to implement the image processing method according to the first aspect.

[0017] In a sixth aspect, a computer program product is provided, the program product is stored in a storage medium, and the program product is executed by at least one processor to implement the image processing method according to the first aspect.

[0018] In the embodiments of the present application, in the video shooting process, the lossless high-frequency information can be obtained based on the original image without noise reduction processing and saved. When the video image is offline noise reduction processing, it is applied to the noise reduction processing process, which can make up for the loss of high-frequency information of the image to be noise reduction, improve the image definition, and improve the noise reduction effect. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 is a flowchart of the image processing method provided by the embodiments of the present application;

[0020] Figure 2 is a schematic diagram of a camera main interface provided by the embodiments of the present application;

[0021] Figure 3 is a schematic diagram of a video editing main interface provided by an embodiment of the present application;

[0022] Figure 4 is a schematic diagram of a custom noise reduction editing interface provided by an embodiment of the present application;

[0023] Figure 5 is an example schematic diagram of noise reduction processing provided by an embodiment of the present application;

[0024] Figure 6 is an example schematic diagram of obtaining high-frequency information and motion vectors provided by an embodiment of the present application;

[0025] Figure 7 is a schematic block diagram of an image processing apparatus provided by an embodiment of the present application;

[0026] Figure 8 is a schematic block diagram of an electronic device provided by an embodiment of the present application;

[0027] Figure 9 is a hardware structure schematic of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of the present application will be clearly described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art belong to the scope of protection of the present application.

[0029] The terms "first", "second", and the like in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than that illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally a category and do not limit the number of objects, for example, the first object can be one or more. In addition, "and / or" in the specification and claims indicates at least one of the connected objects, and the character " / ", generally indicates that the front and rear associated objects are in a "or" relationship.

[0030] The image processing method provided by the embodiments of the present application will be described in detail below with reference to the drawings, specific embodiments and application scenarios.

[0031] Figure 1 is a flowchart of an image processing method provided by an embodiment of the present application, which is applied to an electronic device, i.e., the steps in the image processing method are performed by the electronic device.

[0032] The image processing method can include:

[0033] Step 101: Obtain a first image frame and a second image frame of a target video.

[0034] The first image frame and the second image frame are adjacent image frames in the target video, and the first image frame is a previous image frame of the second image frame.

[0035] In the embodiments of the present application, offline noise reduction processing can be performed on the recorded video frame by frame. When performing frame-by-frame noise reduction processing, in addition to decoding the to-be-reduced-noise image frame from the video file, the previous image frame of the to-be-reduced-noise image frame also needs to be decoded for noise reduction processing of the to-be-reduced-noise image frame. In the embodiments of the present application, the second image frame is the to-be-reduced-noise image frame, and the first image frame is the previous image frame of the to-be-reduced-noise image frame. Offline noise reduction processing refers to noise reduction processing after video shooting is completed.

[0036] Optionally, during video shooting, H.264 or H.265 standard in MPEG-4 standard (ISO / IEC 14496) can be used to encode video images, and the shot video, i.e., the target video, can be saved in MP4 format. Of course, other encoding methods can also be used for image encoding, and the shot video can be saved in other formats, which can be set according to actual needs.

[0037] Step 102: Obtain target information.

[0038] The target information can include: first high-frequency information at a first time and second high-frequency information at a second time in an original image of the target video, the first time is a time corresponding to the first image frame, and the second time is a time corresponding to the second image frame. The original image of the target video refers to an image that has not been subjected to noise reduction processing during video shooting, such as a Raw image that has not been subjected to noise reduction processing.

[0039] The first time is a time corresponding to the first image frame, and the second time is a time corresponding to the second image frame. Assuming that the first image frame is an image frame at time t-1 in the target video, and the second image frame is an image frame at time t in the target video, the first time is time t-1, and the second time is time t.

[0040] In the embodiments of the present application, during video shooting, the high-frequency information of the image that has not been subjected to noise reduction processing is retained. The high-frequency information of the image is the information of the image position where the image intensity (such as brightness, gray scale, etc.) changes sharply, that is, the information of the commonly known edge and contour position, and the human eye is more sensitive to these image positions. When performing offline noise reduction processing on the target video, the original high-frequency information retained during shooting can provide more abundant image contour information, which is conducive to improving the definition of the image after noise reduction processing.

[0041] Step 103: performing noise reduction processing on the second image frame according to the first image frame and the target information.

[0042] According to the first image frame and the first high-frequency information, the first image frame with more abundant high-frequency information can be obtained, and according to the second image frame and the second high-frequency information, the second image frame with more abundant high-frequency information can be obtained. Based on the first image frame and the second image frame with more abundant high-frequency information, noise reduction processing can be performed to make up for the loss of high-frequency information of the image to be reduced, improve the image definition, and improve the noise reduction effect.

[0043] As an optional embodiment, before step 101: obtaining the first image frame and the second image frame of the target image frame, the image processing method can further include:

[0044] Step A1: receiving a noise reduction intensity value input by a user.

[0045] In the embodiments of the present application, for offline noise reduction processing, a custom noise reduction function is provided, through which the user can customize the noise reduction intensity, including reducing image noise and increasing image noise, to meet different needs of the user. The noise reduction intensity value range can be [-a, a], and a is greater than 0. When the noise reduction intensity is greater than 0, it means reducing image noise; when the noise reduction intensity is less than 0, it means increasing image noise.

[0046] As shown in FIG. 1, Figure 2 The main interface 201 of the album application is shown in FIG. 1, and a plurality of video files are displayed. The user long-presses a video to be edited, such as the “night scene a” video, and then selects the edit control below the interface, and then accesses the edit main interface 301 of the “night scene a” video, as shown in FIG. 2. Figure 3 After that, the user clicks the custom noise reduction control, and enters the custom noise reduction edit interface 401, and adjusts the image noise reduction intensity by sliding the noise reduction intensity adjustment control 4011 left and right. Sliding left from the noise reduction intensity 0 value increases the image noise, which can make the video effect more grainy. Sliding right from the noise reduction intensity 0 value reduces the image noise, which can make the video effect smoother. The image display area 4012 displays the processing preview result of a single frame of video image. After the user determines the adjustment effect, the terminal device can process the night scene a video based on the noise reduction intensity customized by the user, and save the processed video to the local album.

[0047] In the embodiments of the present application, by increasing the custom noise reduction function, different image processing needs of the user can be met.

[0048] Step A2: in the case where the noise reduction intensity value is greater than 0, performing the step of obtaining the first image frame and the second image frame of the target image frame.

[0049] When the noise reduction intensity value set by the user is greater than 0, it means that the image needs to be denoised, and then steps 101 to 103 can be executed.

[0050] Step A3: When the noise reduction intensity value is less than 0, acquire the second image frame and the second high-frequency information at the second moment in the target video, and adjust the noise intensity of the second image frame according to the noise reduction intensity value and the second high-frequency information.

[0051] When the noise reduction intensity value set by the user is less than 0, it indicates that image noise needs to be increased. Based on the noise reduction intensity value set by the user and the second high frequency information, noise increase processing can be performed on the second image frame.

[0052] Optionally, step A3, "adjusting the noise intensity of the second image frame based on the noise reduction intensity value and the second high-frequency information," may include:

[0053] Obtain the ratio of the absolute value of the noise reduction intensity to the maximum noise reduction intensity; obtain the product of the second high-frequency information and the ratio; superimpose the product onto the second image frame to obtain the third image frame after adjusting the noise intensity.

[0054] In this embodiment, since the second high-frequency information is high-frequency information separated from the original image, the noise morphology is not destroyed by the ISP module, the noise granularity is relatively fine, and some object edges and details are preserved. Therefore, adding image noise can directly superimpose the second image frame with high frequency, that is, superimpose the product of the second high-frequency information and the ratio of the absolute value of the noise reduction intensity value set by the user to the maximum noise reduction intensity value onto the second image frame to obtain the third image frame, which is the second image frame after adjusting the noise intensity.

[0055] Next, let's take... Figure 5 For example, the above scheme will be illustrated.

[0056] like Figure 5 As shown, this example may include:

[0057] Step 501: Enter custom noise reduction editing mode.

[0058] Step 502: Obtain the noise reduction intensity value β set by the user.

[0059] The noise reduction intensity value ranges from [-α, α], where -α ≤ β ≤ α.

[0060] Step 503: Determine the relationship between the noise reduction intensity value β and 0. If the noise reduction intensity value β is less than 0, proceed to step 504.

[0061] Step 504: Obtain the first image I1 at time t from the recorded video.

[0062] The first image I1 is a YUV image, and corresponds to the second image frame in step 101.

[0063] Step 505: Separate the Y channel image from the first image I1, denoted as the second image I2, and buffer the U and V channel images of the first image I1.

[0064] Step 506: Obtain the high frequency information Res t .

[0065] Here, the time t corresponds to the aforementioned second time, and the high frequency information Res t corresponds to the second high frequency information in the target information in step 102.

[0066] Step 507: Perform granularity processing on the first image I1, the second image I2, the noise reduction intensity value β, and the high frequency information Res t to obtain a third image I3.

[0067] Here, the third image I3 corresponds to the aforementioned third image frame.

[0068] The granularity processing process can be represented as:

[0069]

[0070] In formula (1), Concat means merging and arranging the image data of the Y, U, and V channels in the original YUV order, U represents the U channel image data of the first image I1, V represents the V channel image data of the first image I1, and α is the maximum noise reduction intensity value.

[0071] Step 508: Re-compress and encode the third image I3 and write it to a new video file.

[0072] As an optional embodiment, the target information can also include a target motion vector from the image content of the second image to the image content of the first image, which is used for pixel alignment between the first image frame and the second image frame in the noise reduction process. The first image is an image corresponding to the first image frame and after noise reduction processing, such as a Raw image after noise reduction processing. The second image is an original image corresponding to the second image frame and without noise reduction processing, such as a Raw image without noise reduction processing.

[0073] Step 103: Perform noise reduction processing on the second image frame according to the first image frame and the target information, which can include:

[0074] Step B1: Superimpose the first high frequency information onto the first image frame to obtain a fourth image frame.

[0075] Step B2: superimpose the second high-frequency information on the second image frame to obtain a fifth image frame.

[0076] Step B3: based on the target motion vector, perform pixel alignment on the fourth image frame and the fifth image frame to obtain a sixth image frame.

[0077] Step B4: input the fifth image frame, the sixth image frame, and the denoising intensity value into the target denoising model to obtain a seventh image frame after denoising processing.

[0078] The target denoising model described herein is a pre-trained model with relatively large computing power. Using this model for denoising processing can improve the denoising effect.

[0079] In the embodiments of the present application, based on the first image frame and the first high-frequency information, a fourth image frame with more rich high-frequency information can be obtained. Similarly, based on the second image frame and the second high-frequency information, a fifth image frame with more rich high-frequency information can be obtained. Based on the fourth image frame and the fifth image frame with more rich high-frequency information, denoising processing can compensate for the loss of high-frequency information of the second image frame, improve the image clarity of the seventh image frame after denoising processing, and further improve the denoising effect.

[0080] The following will continue to take Figure 5 as an example to illustrate the scheme described in steps B1 to B4.

[0081] As shown in Figure 5 , the example can include:

[0082] Step 501: enter the custom denoising editing mode.

[0083] Step 502: obtain the denoising intensity value β set by the user.

[0084] Wherein, the value range of the denoising intensity value is [-α, α], -α≤β≤α.

[0085] Step 503: determine the relationship between the denoising intensity value β and 0. In the case where the denoising intensity value β is greater than 0, go to step 509.

[0086] Step 509: obtain the first image I1 at time t and the fourth image I4 at time t-1 from the recorded video.

[0087] The fourth image I4 described herein is a YUV image, corresponding to the second image frame in step 101. The t-1 time described herein corresponds to the first time described above.

[0088] Step 510: separate the Y channel image from the first image I1, denoted as the second image I2; separate the Y channel image from the fourth image I4, denoted as the fifth image I5.

[0089] Step 511: Obtain high-frequency information Res at time t t High-frequency information Res at time t-1 t-1 and the motion vector MV from time t to time t-1 t And cache the U and V channel images of the first image I1.

[0090] The high-frequency information Res mentioned here t-1 The first high-frequency information in the target information described in step 102 corresponds to this. The motion vector MV mentioned here... t The target motion vector in the corresponding target information.

[0091] Step 512: Based on the first image I1, the second image I2, the fifth image I5, and the high-frequency information Res t High-frequency information Res t-1 and motion vector MV t The first AI model is combined with noise reduction processing to obtain the sixth image I6.

[0092] The first AI model described here corresponds to the target denoising model in step B4. This first AI model is an offline denoising model, which consumes more power and requires more computation, but can achieve better denoising results.

[0093] The sixth image I6 mentioned here corresponds to the seventh image frame in step B4.

[0094] The offline noise reduction process can be represented as follows:

[0095] I′2=I2+resize(Res t (2)

[0096] I′5=I5+resize(Res t-1 (3)

[0097] I″5=warp(I′5,MV t (4)

[0098] I6=Concat(f2(I′2,I″5,β / α;θ),U,V) (5)

[0099] In formula (2), resize(Res) t ) indicates that the high-frequency information Res t The image size is adjusted to be the same as the second image I2. Since the size of the original image may differ from the size of the final generated image (such as the first image frame I1), the image size of high-frequency information can be adjusted. This can be achieved using a nearest neighbor interpolation algorithm to adjust the image size of the high-frequency information Res. tto the same image size as the second image I2. Equation (2) represents superimposing the second image I2 and the high-frequency information Res t to obtain the image I'2 of the restored high-frequency information.

[0100] In equation (3), resize(Res t-1 ) represents adjusting the image size of the high-frequency information Res t-1 to the same image size as the fifth image I5. Since the size of the original image can be different from the size of the finally generated image (such as the fourth image frame I4), the image size adjustment of the high-frequency information can be performed. In which, the nearest neighbor interpolation algorithm can be used to adjust the high-frequency information Res t-1 to the same image size as the fifth image I5. Equation (3) represents superimposing the fifth image I5 and the high-frequency information Res t-1 adjusted in image size to obtain the image I'5 of the restored high-frequency information.

[0101] Equation (4) represents aligning the image I'5 and the image I'2 in pixels according to the offset vector of each pixel point of the motion vector MV t . Before that, the MV t can be up-sampled to the same image size as the fifth image I5 by the bilinear interpolation method.

[0102] Equation (5) represents inputting I'2, I"5, and β into the first AI model, and then merging and arranging the output result of the first AI model with the U and V channel images of the first image I1 to obtain the sixth image I6 after noise reduction processing. Wherein, f2 represents the function of the first AI model, θ is the training parameter of the model, and Concat means merging and arranging the output result of the first AI model, that is, the Y channel image after noise reduction processing, with the U and V channel images of the first image frame I1 according to the original YUV order.

[0103] Wherein, the first AI model can be a convolutional neural network (CNN) model, and the network structure can be a UNet structure.

[0104] Step 513: Re-compressing and encoding the sixth image I6 and writing it into a new video file.

[0105] As an optional embodiment, how to obtain the target motion vector and the second high-frequency information is described below.

[0106] Before step 101: obtaining the first image frame and the second image frame of the target video, the image processing method can further include:

[0107] Step C1: Obtain the first image and the second image.

[0108] Step C2: Based on the first image and the second image, perform motion estimation to obtain the target motion vector of the image content of the second image relative to the image content of the first image.

[0109] It should be noted that motion estimation involves dividing each frame of the image sequence into many non-overlapping image blocks. Then, for each image block in the current image frame within a given search range to the previous or next image frame, the most similar image block (i.e., the matching image block) is found according to a certain matching criterion. The motion displacement is calculated from the relative position of the matching image block and the current image block. The resulting motion position is the motion vector of the current image block. Finally, the motion vectors of all image blocks in the current image frame are the motion vectors of the image content of the current image frame relative to the previous or next image frame. In the embodiments of this application, the second image is the current image frame, and the first image is the image frame preceding the current image frame.

[0110] Step C3: Obtain the third image.

[0111] The third image is the image obtained after denoising the second image, and can also be understood as the first image after denoising.

[0112] Step C4: Perform noise separation on the second and third images to obtain the second high-frequency information at the second time point.

[0113] The following is based on Figure 6 For example, the solutions described in steps C1 to C4 will be illustrated.

[0114] like Figure 6 As shown, this example may include:

[0115] Step 601: Start recording video.

[0116] Step 602: Acquire the seventh image I7 at time t from the image sensor.

[0117] The seventh image I7 mentioned here corresponds to the second image described in step C1. In this example, the seventh image I7 is a raw image without noise reduction processing.

[0118] Step 603: Retrieve the eighth image I8 at time t-1 from the cache.

[0119] The eighth image I8 mentioned here corresponds to the first image mentioned in step C1. In this example, the eighth image I8 is the Raw image cached after Raw domain noise reduction processing at time t-1.

[0120] Step 604: Perform motion estimation on the seventh image I7 and the eighth image I8 to obtain the motion vector MV of the image content of the seventh image I7 relative to the image content of the eighth image I8. t and the calculated MV t Write it to an intermediate file.

[0121] The motion vector MV mentioned here t This corresponds to the target motion vector described in step C2. Motion vector MV t Specifically, this can be achieved through the following process:

[0122] First, divide images I7 and I8 into m×n blocks of equal size, where 1≤n≤H, 1≤m≤W, and H and W are the height and width of the two images, respectively. The image blocks at the same location in the two images are represented as follows: Where i∈[1,m],j∈[1,n].

[0123] Then, using the grayscale projection method, calculate... Spatial position offset vector MV in the x and y directions t Specifically, when m, n = 1, it is equivalent to calculating the offset vector of each pixel position.

[0124] Specifically, the calculation process for the grayscale projection method can be as follows:

[0125] First, calculate using formulas (6) and (7) respectively. Projection in the vertical direction and projection in the horizontal direction.

[0126]

[0127]

[0128] Among them, in formula (6) express In the vertical projection, H B express The height. In formula (7) express In the horizontal projection, W B express The width.

[0129] Then, using formula (8), we calculate u and v in the x and y directions respectively to minimize the absolute value of the projection deviation, and use them as MV. t .

[0130]

[0131] In formula (8), λ and γ are algorithm parameters, representing the search space of u and v.

[0132] Alternatively, a block matching algorithm can be used for vector estimation. The block matching algorithm can refer to the Non Local Means filtering algorithm, which iterates through the sum of absolute differences (SAD) of all image blocks in the search window and takes the minimum value of u and v as the actual offset vector MV.

[0133] Alternatively, the sparse optical flow tracking KLT optical flow method can be used to calculate the actual offset vector MV.

[0134] Alternatively, the results provided by other modules that can detect motion vectors, such as the phone's gyroscope, can be used as the actual offset vector MV.

[0135] Step 605: Based on the MV at time t t Then, the eighth image I8 and the seventh image I7 are pixel-aligned to obtain the ninth image I9.

[0136] The ninth image I9 mentioned here corresponds to the third image in step C4.

[0137] Alternatively, MV can be first calculated using bilinear interpolation. t Upsample to the same width and height as the eighth image I8, then according to MV t The (u,v) offset vector of each pixel is used to align the eighth image I8 and the seventh image I7, so that the eighth image I8 is transformed to the same viewpoint position as the seventh image I7, resulting in the ninth image I9.

[0138] When performing pixel alignment, the moving image can use a bicubic interpolation algorithm.

[0139] Step 606: Input the seventh image I7 and the ninth image I9 into the second AI model simultaneously to obtain the tenth image I. 10 .

[0140] Among them, the tenth image I 10 Let I be the denoised image at time t, corresponding to the third image described in step C3. For the tenth image I... 10 It can be cached and used at time t+1.

[0141] The tenth image I can be obtained through formula (9). 10 .

[0142] I 10 =f1(Concat(I7,I9);θ) (9)

[0143] In formula (9), Concat means stacking the seventh image I7 and the ninth image I9, f1 represents the first AI model function, and θ is the training parameter of the model.

[0144] Optionally, the second AI model can be a CNN network model with pre-trained parameters, and the network structure can be a UNet structure, whose computing power and power consumption can meet the requirements for real-time operation on the device.

[0145] Step 607: Combine the seventh image I7 and the tenth image I 10 Noise separation is performed to obtain the high-frequency information Res at time t. t and the calculated Res t Write it to an intermediate file.

[0146] Among them, Res t =I1-I4.

[0147] Because the noise reduction effect of I4 is limited by the power consumption and computing power of the second AI model, there may be a loss of detail. t Some of these technologies can address the problem of irreversible loss of high-frequency information, making them suitable for use in the offline noise reduction stage and thus improving the offline noise reduction effect.

[0148] Step 608: Transfer the tenth image I 10 The video is fed into the ISP pipeline, processed in the RGB and YUV domains, and then output as the final video.

[0149] The intermediate files and the final video file exist in pairs. The intermediate files can be binary files with any suffix such as .bin, .meta, .data, etc., and this embodiment of the application does not limit this.

[0150] The above is a description of the image processing method provided in the embodiments of this application.

[0151] In summary, in this embodiment, lossless high-frequency information of the image is preserved during video recording. When performing offline noise reduction on the video image, this information is applied to the noise reduction process to compensate for the loss of high-frequency information in the image to be denoised, improving image clarity and thus enhancing the noise reduction effect. By combining online and offline noise reduction methods, a better noise reduction effect can be achieved. Furthermore, this embodiment also provides a custom noise reduction adjustment function, allowing users to adjust the noise reduction effect more freely and conveniently.

[0152] The image processing method provided in this application can be executed by an image processing device. This application uses an image processing device executing the image processing method as an example to illustrate the image processing device provided in this application.

[0153] Figure 7 This is a schematic block diagram of an image processing apparatus provided in an embodiment of this application, which is applied to an electronic device.

[0154] like Figure 7 As shown, the image processing apparatus may include:

[0155] The first acquisition module 701 is used to acquire the first image frame and the second image frame of the target video.

[0156] Wherein, the first image frame and the second image frame are adjacent image frames, and the first image frame is the image frame preceding the second image frame.

[0157] The second acquisition module 702 is used to acquire target information.

[0158] The target information includes: first high-frequency information at a first moment and second high-frequency information at a second moment in the original image of the target video, wherein the first moment is the moment corresponding to the first image frame and the second moment is the moment corresponding to the second image frame.

[0159] The first noise processing module 703 is used to perform noise reduction processing on the second image frame based on the first image frame and the target information.

[0160] Optionally, the device may further include:

[0161] The receiving module is used to receive the noise reduction intensity value input by the user.

[0162] The control module is used to control the execution of the steps of acquiring the first image frame and the second image frame of the target image frame when the noise reduction intensity value is greater than 0.

[0163] The second noise processing module is used to acquire the second image frame and the second high-frequency information at the second moment in the target video when the noise reduction intensity value is less than 0; and to adjust the noise intensity of the second image frame according to the noise reduction intensity value and the second high-frequency information.

[0164] Optionally, the second noise processing module may include:

[0165] The first acquisition unit is used to acquire the ratio of the absolute value of the noise reduction intensity value to the maximum noise reduction intensity value.

[0166] The second acquisition unit is used to acquire the product of the second high-frequency information and the ratio.

[0167] The first superposition processing unit is used to superimpose the product onto the second image frame to obtain a third image frame with adjusted noise intensity.

[0168] Optionally, the target information may further include: the target motion vector from the second image at the second time to the first image at the first time; the first image is an image corresponding to the first image frame and after noise reduction processing, and the second image is the original image corresponding to the second image frame and without noise reduction processing.

[0169] The first noise processing module 703 may include:

[0170] The second overlay processing unit is used to overlay the first high-frequency information onto the first image frame to obtain a fourth image frame.

[0171] The third overlay processing unit is used to overlay the second high-frequency information onto the second image frame to obtain the fifth image frame.

[0172] A pixel alignment unit is used to align the fourth image frame with the fifth image frame based on the target motion vector to obtain a sixth image frame.

[0173] The noise reduction processing unit is used to input the fifth image frame, the sixth image frame and the noise reduction intensity value into the target noise reduction model to obtain the seventh image frame after noise reduction processing.

[0174] Optionally, the device may further include:

[0175] The third acquisition module is used to acquire the first image and the second image.

[0176] The motion estimation module is used to perform motion estimation based on the first image and the second image to obtain the target motion vector of the image content of the second image relative to the image content of the first image.

[0177] The fourth acquisition module is used to acquire a third image; wherein the third image is an image obtained after denoising the second image.

[0178] The noise separation module is used to separate noise from the second image and the third image to obtain the second high-frequency information at the second time.

[0179] In summary, in this embodiment of the application, during video recording, lossless high-frequency information is obtained from the original image without noise reduction processing and saved. When performing offline noise reduction processing on video images, applying this information to the noise reduction process can compensate for the loss of high-frequency information in the image to be denoised, improve image clarity, and enhance the noise reduction effect.

[0180] The image processing device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television set (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.

[0181] The image processing device in this application embodiment can be a device with an operating system. The operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system.

[0182] The image processing apparatus provided in this application embodiment can achieve... Figure 1 The various processes implemented in the image processing method embodiment shown will not be described again here to avoid repetition.

[0183] Optionally, such as Figure 8 As shown, this application embodiment also provides an electronic device 800, including: a processor 801 and a memory 802. The memory 802 stores a program or instructions that can be executed by the processor 801. When the program or instructions are executed by the processor 801, they implement the various steps of the above-described image processing method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0184] It should be noted that the electronic device 800 in this application embodiment includes the aforementioned mobile electronic device and non-mobile electronic device.

[0185] Figure 9 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0186] The electronic device 900 includes, but is not limited to, components such as: radio frequency unit 901, network module 902, audio output unit 903, input unit 904, sensor 905, display unit 906, user input unit 907, interface unit 908, memory 909, and processor 910.

[0187] Those skilled in the art will understand that the electronic device 900 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 910 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 9 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0188] The processor 910 can be used to: acquire a first image frame and a second image frame of the target video; wherein the first image frame and the second image frame are adjacent image frames, and the first image frame is the image frame preceding the second image frame; acquire target information; wherein the target information includes: a first high-frequency information at a first time and a second high-frequency information at a second time in the original image of the target video, the first time being the time corresponding to the first image frame and the second time being the time corresponding to the second image frame; and perform noise reduction processing on the second image frame based on the first image frame and the target information.

[0189] Optionally, the processor 910 can also be used to: receive a noise reduction intensity value input by a user; if the noise reduction intensity value is greater than 0, execute the step of acquiring the first image frame and the second image frame of the target image frame; if the noise reduction intensity value is less than 0, acquire the second image frame in the target video and the second high-frequency information at the second time, and adjust the noise intensity of the second image frame according to the noise reduction intensity value and the second high-frequency information.

[0190] Optionally, the processor 910 can also be used to: obtain the ratio of the absolute value of the noise reduction intensity value to the maximum noise reduction intensity value; obtain the product of the second high-frequency information and the ratio; and superimpose the product onto the second image frame to obtain a third image frame with adjusted noise intensity.

[0191] Optionally, the target information may further include: the target motion vector from the second image at the second time to the first image at the first time, wherein the first image is an image corresponding to the first image frame and denoised, and the second image is the original image corresponding to the second image frame and not denoised.

[0192] The processor 910 can also be used to: superimpose the first high-frequency information onto the first image frame to obtain a fourth image frame; superimpose the second high-frequency information onto the second image frame to obtain a fifth image frame; align the fourth image frame and the fifth image frame pixel-wise based on the target motion vector to obtain a sixth image frame; and input the fifth image frame, the sixth image frame, and the noise reduction intensity value into the target noise reduction model to obtain a seventh image frame after noise reduction processing.

[0193] Optionally, the processor 910 can also be used to: acquire the first image and the second image; perform motion estimation based on the first image and the second image to obtain the target motion vector of the image content of the second image relative to the image content of the first image; acquire a third image; wherein the third image is an image obtained after denoising the second image; and perform noise separation on the second image and the third image to obtain the second high-frequency information at the second time.

[0194] In this embodiment of the invention, during video recording, lossless high-frequency information is obtained from the original image without noise reduction processing and saved. When performing offline noise reduction processing on the video image, applying this information can compensate for the loss of high-frequency information in the image to be denoised, improve image clarity, and enhance the noise reduction effect.

[0195] It should be understood that, in this embodiment, the input unit 904 may include a graphics processing unit (GPU) 9041 and a microphone 9042. The GPU 9041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 906 may include a display panel 9061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 907 includes at least one of a touch panel 9071 and other input devices 9072. The touch panel 9071 is also called a touch screen. The touch panel 9071 may include a touch detection device and a touch controller. Other input devices 9072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.

[0196] The memory 909 can be used to store software programs and various data. The memory 909 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 909 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be ROM, Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), or flash memory. Volatile memory can be RAM, Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The memory 909 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0197] Processor 910 may include one or more processing units; optionally, processor 910 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 910.

[0198] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described image processing method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.

[0199] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0200] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described image processing method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0201] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the above-described image processing method embodiments, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0202] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0203] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0204] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM, RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0205] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. An image processing method, characterized by, The method comprises: obtaining a first image frame and a second image frame of a target video; wherein the first image frame and the second image frame are adjacent image frames, and the first image frame is a previous image frame of the second image frame; obtaining target information; wherein the target information comprises first high-frequency information at a first time and second high-frequency information at a second time in an original image of the target video, the first time being a time corresponding to the first image frame, and the second time being a time corresponding to the second image frame; performing noise reduction processing on the second image frame according to the first image frame and the target information; Before the step of obtaining the first image frame and the second image frame of the target video, the method further comprises: receiving a noise reduction intensity value input by a user; in a case where the noise reduction intensity value is greater than 0, performing the step of obtaining the first image frame and the second image frame of the target video; in a case where the noise reduction intensity value is less than 0, obtaining the second image frame and the second high-frequency information at the second time in the target video, and adjusting a noise intensity of the second image frame according to the noise reduction intensity value and the second high-frequency information.

2. The image processing method of claim 1, wherein, The step of adjusting the noise intensity of the second image frame according to the noise reduction intensity value and the second high-frequency information comprises: obtaining a ratio of an absolute value of the noise reduction intensity value to a maximum noise reduction intensity value; obtaining a product of the second high-frequency information and the ratio; superimposing the product on the second image frame to obtain a third image frame with an adjusted noise intensity.

3. The image processing method of claim 1, wherein, The target information further comprises a target motion vector of image content of a second image relative to image content of a first image, the first image being an original image corresponding to the first image frame and having been subjected to noise reduction processing, and the second image being an original image corresponding to the second image frame and not having been subjected to noise reduction processing; The step of performing noise reduction processing on the second image frame according to the first image frame and the target information comprises: superimposing the first high-frequency information on the first image frame to obtain a fourth image frame; superimposing the second high-frequency information on the second image frame to obtain a fifth image frame; performing pixel alignment on the fourth image frame and the fifth image frame based on the target motion vector to obtain a sixth image frame; inputting the fifth image frame, the sixth image frame, and the noise reduction intensity value into a target noise reduction model to obtain a seventh image frame that has been subjected to noise reduction processing.

4. The image processing method of claim 3, wherein, The method further comprises: obtaining the first image and the second image; performing motion estimation according to the first image and the second image to obtain the target motion vector of image content of the second image relative to image content of the first image; obtaining a third image; wherein the third image is an image obtained by performing noise reduction processing on the second image; performing noise separation on the second image and the third image to obtain the second high-frequency information at the second time.

5. An image processing apparatus characterized by comprising: The device comprises: The first obtaining module is configured to obtain a first image frame and a second image frame of a target video; wherein the first image frame and the second image frame are adjacent image frames, and the first image frame is a previous image frame of the second image frame. The second obtaining module is configured to obtain target information; wherein the target information comprises first high-frequency information at a first time and second high-frequency information at a second time in an original image of the target video, the first time is a time corresponding to the first image frame, and the second time is a time corresponding to the second image frame. The first noise processing module is configured to perform noise reduction processing on the second image frame according to the first image frame and the target information. The device further comprises: The receiving module is configured to receive a noise reduction intensity value input by a user. The control module is configured to, in a case where the noise reduction intensity value is greater than 0, control execution of the step of obtaining the first image frame and the second image frame of the target video. The second noise processing module is configured to, in a case where the noise reduction intensity value is less than 0, obtain the second image frame in the target video and the second high-frequency information at the second time, and adjust a noise intensity of the second image frame according to the noise reduction intensity value and the second high-frequency information.

6. The image processing apparatus according to claim 5, characterized by The second noise processing module comprises: The first obtaining unit is configured to obtain a ratio of an absolute value of the noise reduction intensity value to a maximum noise reduction intensity value. The second obtaining unit is configured to obtain a product of the second high-frequency information and the ratio. The first superposition processing unit is configured to superimpose the product on the second image frame to obtain a third image frame with an adjusted noise intensity.

7. The image processing apparatus according to claim 5, characterized by, The target information further comprises a target motion vector of image content of a second image relative to image content of a first image, the first image being an original image corresponding to the first image frame and subjected to noise reduction processing, and the second image being an original image corresponding to the second image frame and not subjected to noise reduction processing. The first noise processing module comprises: The second superposition processing unit is configured to superimpose the first high-frequency information on the first image frame to obtain a fourth image frame. The third superposition processing unit is configured to superimpose the second high-frequency information on the second image frame to obtain a fifth image frame. The pixel alignment unit is configured to perform pixel alignment between the fourth image frame and the fifth image frame based on the target motion vector to obtain a sixth image frame. The noise reduction processing unit is configured to input the fifth image frame, the sixth image frame, and the noise reduction intensity value into a target noise reduction model to obtain a seventh image frame subjected to noise reduction processing.

8. The image processing apparatus according to claim 7, characterized by The device further comprises: The third obtaining module is configured to obtain the first image and the second image. The motion estimation module is configured to perform motion estimation according to the first image and the second image to obtain the target motion vector of image content of the second image relative to image content of the first image. The fourth obtaining module is configured to obtain a third image; wherein the third image is an image obtained by performing noise reduction processing on the second image. A noise separation module is configured to separate noise from the second image and the third image to obtain the second high-frequency information at the second time.

Images