An image processing method and apparatus

Abstract

The embodiment of the present application provides a kind of image processing method and device, it is related to image processing technical field, method includes: sequentially obtaining each video frame to be aligned, and carry out caching;After caching each video frame to be aligned, if the number of current video frame to be aligned that has been cached is not less than first number, then the mapping matrix between the video frame to be aligned and corresponding second number of adjacent video frames is calculated respectively;According to the order of caching, for the last cached first number of video frames to be aligned except center video frame, based on the mapping matrix between the video frame to be aligned and heart video frame, the video frame to be aligned is processed, and the video frame to be fused is obtained;Each video frame to be fused and the center video frame are fused and handled, and target video frame is obtained.Such, it can improve the image quality of video image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to an image processing method and device. BACKGROUND

[0002] With the rapid development of computer technology, a video image can be acquired by a video acquisition device, and then subsequent processing can be performed based on the acquired video image. For example, target detection can be performed on the video image.

[0003] However, when the ambient light brightness is insufficient, the video image acquired by the video acquisition device contains a large amount of noise, resulting in low image quality of the video image. SUMMARY

[0004] The purpose of the embodiments of the present application is to provide an image processing method and device to improve the image quality of a video image.

[0005] The specific technical solutions are as follows:

[0006] In a first aspect, the embodiments of the present application provide an image processing method, which comprises:

[0007] Each to-be-aligned video frame is acquired in sequence and buffered;

[0008] After each to-be-aligned video frame is buffered, if the number of the to-be-aligned video frames that have been buffered is not less than a first number, mapping matrices between the to-be-aligned video frame and a second number of adjacent video frames corresponding to the to-be-aligned video frame are calculated, respectively; wherein the second number of adjacent video frames comprises: a center video frame in the first number of to-be-aligned video frames buffered last in the order of buffering, and video frames between the center video frame and the to-be-aligned video frame;

[0009] For each to-be-aligned video frame other than the center video frame in the first number of to-be-aligned video frames buffered last in the order of buffering, the to-be-aligned video frame is processed based on the mapping matrix between the to-be-aligned video frame and the center video frame, to obtain a to-be-fused video frame;

[0010] Each to-be-fused video frame and the center video frame are fused to obtain a target video frame.

[0011] In some embodiments, the first number is an odd number greater than 1, and the center video frame is a to-be-aligned video frame located at the center position in the first number of to-be-aligned video frames in the order of buffering.

[0012] In some embodiments, after the each to-be-fused video frame and the center video frame are fused to obtain a target video frame, the method further comprises:

[0013] delete, in the order of caching, the first cached video frame from the last cached first number of video frames to be aligned.

[0014] In some embodiments, after the calculating the mapping matrix between the video frame to be aligned and the corresponding second number of adjacent video frames respectively, the method further comprises:

[0015] cache the calculated mapping matrices in the order of caching between the corresponding second number of adjacent video frames;

[0016] after the fusing processing of each video frame to be fused and the center video frame to obtain the target video frame, the method further comprises:

[0017] delete, in the order of caching, the second number of first cached mapping matrices.

[0018] In some embodiments, the processing, in the order of caching, of each video frame to be aligned except the center video frame from the last cached first number of video frames to be aligned based on the mapping matrix between the video frame to be aligned and the center video frame to obtain the video frame to be fused comprises:

[0019] if the pre-calculated mapping matrix corresponding to the video frame to be aligned indicates the mapping relationship from the center video frame to the video frame to be aligned, calculating the inverse matrix of the mapping matrix corresponding to the video frame to be aligned, in the order of caching, for each video frame to be aligned except the center video frame from the last cached first number of video frames to be aligned.

[0020] processing, based on the calculated inverse matrix, of the video frame to be aligned to obtain the video frame to be fused.

[0021] In some embodiments, the fusing processing of each video frame to be fused and the center video frame to obtain the target video frame comprises:

[0022] determining, based on the weights of each video frame to be fused and the center video frame, the weight of each of a plurality of pixel points with the same pixel coordinates in each video frame to be fused and the center video frame; wherein the weights of each video frame to be fused and the center video frame conform to Gaussian distribution, the weight of the center video frame is the largest, and the distance between the video frame to be aligned corresponding to each video frame to be fused and the center video frame is negatively correlated with the weight of the video frame to be fused in the order of caching.

[0023] According to the weight of each of the plurality of pixel points, a weighted sum of pixel values of the plurality of pixel points is calculated to obtain a pixel value of a corresponding pixel coordinate in the target video frame.

[0024] In some embodiments, the weight of each of the plurality of pixel points at the same pixel coordinate in each of the to-be-fused video frames and the center video frame is determined based on the weight of each of the to-be-fused video frames and the center video frame, including:

[0025] For each pixel point at the same pixel coordinate in each of the to-be-fused video frames and the center video frame, if the pixel point belongs to the to-be-fused video frame, and the difference between the pixel point and the pixel point at the same pixel coordinate in the center video frame is less than a preset threshold, the weight of the pixel point is determined as the weight of the to-be-fused video frame to which the pixel point belongs.

[0026] If the pixel point belongs to the to-be-fused video frame, and the difference between the pixel point and the pixel point at the same pixel coordinate in the center video frame is not less than the preset threshold, the weight of the pixel point is determined as 0.

[0027] The weight of the pixel point at the same pixel coordinate in the center video frame is calculated based on the weight of the pixel point at the same pixel coordinate in each of the to-be-fused video frames.

[0028] In some embodiments, the step of sequentially obtaining each to-be-aligned video frame and buffering includes:

[0029] Sequentially obtaining each video frame from the to-be-processed video;

[0030] Processing the video frame based on a pre-trained denoising network model to obtain a to-be-aligned video frame and buffering.

[0031] In some embodiments, the step of calculating the mapping matrix between the to-be-aligned video frame and a corresponding second number of adjacent video frames includes:

[0032] For each adjacent video frame, based on a block matching algorithm, a pixel point in the adjacent video frame that matches a corner point in the to-be-aligned video frame is determined as an aligned pixel point.

[0033] Based on the pixel coordinates of the corner point in the to-be-aligned video frame and the pixel coordinates of the aligned pixel point, a mapping matrix between the to-be-aligned video frame and the adjacent video frame is calculated according to a RANSAC (RANdom SAmple Consensus) algorithm.

[0034] In some embodiments, the step of processing the to-be-aligned video frame based on the mapping matrix between the to-be-aligned video frame and the center video frame to obtain a to-be-fused video frame includes:

[0035] mapping, based on the mapping matrix between the video frame to be aligned and the center video frame, each pixel coordinate in the video frame to be aligned, to obtain a corresponding pixel coordinate in the video frame to be aligned as a mapped pixel coordinate;

[0036] determining a pixel value at the mapped pixel coordinate as a pixel value at the pixel coordinate in the video frame to be fused.

[0037] In some embodiments, before the determining of the pixel value at the mapped pixel coordinate as the pixel value at the pixel coordinate in the video frame to be fused, the method further comprises:

[0038] if the mapped pixel coordinate is not an integer, determining a neighborhood range containing the mapped pixel coordinate in the video frame to be aligned;

[0039] calculating the pixel value at the mapped pixel coordinate according to a bilinear interpolation algorithm based on pixel values of a plurality of pixel points in the neighborhood range.

[0040] A second aspect of the embodiment of the application provides an image processing device, which comprises:

[0041] a video frame to be aligned buffer module configured to sequentially acquire each video frame to be aligned and buffer the video frame to be aligned;

[0042] a mapping matrix calculation module configured to, after buffering each video frame to be aligned, if a number of currently buffered video frames to be aligned is not less than a first number, calculate a mapping matrix between the video frame to be aligned and a corresponding second number of adjacent video frames; wherein the second number of adjacent video frames comprises a center video frame in the first number of video frames to be aligned buffered last in the order of buffering and video frames between the center video frame and the video frame to be aligned.

[0043] a video frame to be fused acquisition module configured to, according to the order of buffering, for each video frame to be aligned except the center video frame in the first number of video frames to be aligned buffered last, process the video frame to be aligned based on the mapping matrix between the video frame to be aligned and the center video frame to obtain a video frame to be fused.

[0044] a target video frame acquisition module configured to fuse each video frame to be fused and the center video frame to obtain a target video frame.

[0045] In some embodiments, the first number is an odd number greater than 1, and according to the order of buffering, the center video frame is a video frame to be aligned located at a center position in the first number of video frames to be aligned.

[0046] In some embodiments, the apparatus further comprises:

[0047] a to-be-aligned video frame deleting module configured to delete, after the fusion processing of each to-be-fused video frame and the center video frame to obtain a target video frame, the first to-be-aligned video frame from the last to-be-aligned video frame in the first number of to-be-aligned video frames according to the order of caching.

[0048] In some embodiments, the apparatus further comprises:

[0049] a mapping matrix caching module configured to cache, after the calculation of the mapping matrix between the to-be-aligned video frame and the corresponding second number of adjacent video frames, each mapping matrix according to the order of caching between the corresponding second number of adjacent video frames;

[0050] a mapping matrix deleting module configured to delete, after the fusion processing of each to-be-fused video frame and the center video frame to obtain a target video frame, the second number of first-cached mapping matrices according to the order of caching.

[0051] In some embodiments, the to-be-fused video frame obtaining module comprises:

[0052] an inverse matrix calculating submodule configured to, for each to-be-aligned video frame except the center video frame in the last-cached first number of to-be-aligned video frames, calculate the inverse matrix of the mapping matrix corresponding to the to-be-aligned video frame if the mapping matrix corresponding to the to-be-aligned video frame represents the mapping relationship from the center video frame to the to-be-aligned video frame according to the order of caching.

[0053] a to-be-fused video frame obtaining submodule configured to obtain a to-be-fused video frame by processing the to-be-aligned video frame based on the calculated inverse matrix.

[0054] In some embodiments, the target video frame obtaining module comprises:

[0055] a weight obtaining submodule configured to determine the weight of each of a plurality of pixel points with the same pixel coordinates in each to-be-fused video frame and the center video frame based on the weight of each to-be-fused video frame and the center video frame, wherein the weight of each to-be-fused video frame and the center video frame conforms to a Gaussian distribution, the weight of the center video frame is the largest, and the distance between the to-be-aligned video frame corresponding to each to-be-fused video frame and the center video frame is negatively correlated with the weight of the to-be-fused video frame according to the order of caching.

[0056] The pixel value obtaining submodule is configured to calculate a weighted sum of pixel values of the plurality of pixel points according to respective weights of the plurality of pixel points, and obtain a pixel value of a corresponding pixel coordinate in the target video frame.

[0057] In some embodiments, the weight obtaining submodule is specifically configured to:

[0058] For each pixel point of the same pixel coordinate in each to-be-fused video frame and the center video frame, if the pixel point belongs to the to-be-fused video frame, and a difference between the pixel point and a pixel point of the same pixel coordinate in the center video frame is less than a preset threshold, the weight of the pixel point is determined as a weight of the to-be-fused video frame to which the pixel point belongs.

[0059] If the pixel point belongs to the to-be-fused video frame, and the difference between the pixel point and the pixel point of the same pixel coordinate in the center video frame is not less than the preset threshold, the weight of the pixel point is determined as 0.

[0060] The weight of the pixel point of the same pixel coordinate in the center video frame is calculated based on the weights of the pixel points of the same pixel coordinate in each to-be-fused video frame.

[0061] In some embodiments, the to-be-aligned video frame caching module comprises:

[0062] The video frame obtaining submodule is configured to sequentially obtain each video frame from a to-be-processed video.

[0063] The to-be-aligned video frame caching submodule is configured to process the video frame based on a pre-trained noise reduction network model, obtain a to-be-aligned video frame, and cache the to-be-aligned video frame.

[0064] In some embodiments, the mapping matrix calculation module comprises:

[0065] The aligned pixel point obtaining submodule is configured to, for each adjacent video frame, determine, based on a block matching algorithm, a pixel point matched with a corner point in the to-be-aligned video frame from the adjacent video frame as an aligned pixel point.

[0066] The mapping matrix calculation submodule is configured to calculate, based on pixel coordinates of the corner point in the to-be-aligned video frame and pixel coordinates of the aligned pixel point, a mapping matrix between the to-be-aligned video frame and the adjacent video frame according to a random sample consensus (RANSAC) algorithm.

[0067] In some embodiments, the to-be-fused video frame obtaining module comprises:

[0068] The mapping pixel coordinate obtaining submodule is configured to, for each pixel coordinate in the video frame to be aligned, perform mapping processing on the pixel coordinate based on a mapping matrix between the video frame to be aligned and the center video frame, to obtain a corresponding pixel coordinate in the video frame to be aligned as a mapping pixel coordinate.

[0069] The video frame to be fused determining submodule is configured to determine a pixel value at the mapping pixel coordinate in the video frame to be fused as a pixel value at the pixel coordinate in the video frame to be fused.

[0070] In some embodiments, the apparatus further comprises:

[0071] The neighborhood range determining module is configured to, before the determination of the pixel value at the mapping pixel coordinate in the video frame to be fused as the pixel value at the pixel coordinate in the video frame to be fused, determine a neighborhood range containing the mapping pixel coordinate in the video frame to be aligned if the mapping pixel coordinate is not an integer.

[0072] The pixel value calculating module is configured to calculate the pixel value at the mapping pixel coordinate based on pixel values of a plurality of pixel points in the neighborhood range according to a bilinear interpolation algorithm.

[0073] A third aspect of the embodiment of the present application provides an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus.

[0074] The memory is configured to store a computer program.

[0075] The processor is configured to execute the program stored on the memory, and implement the image processing method described above.

[0076] A fourth aspect of the embodiment of the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the image processing method described above.

[0077] The embodiment of the present application further provides a computer program product comprising instructions which, when executed on a computer, cause the computer to perform the image processing method described above.

[0078] The embodiment of the present application has the following beneficial effects:

[0079] The image processing method provided by the embodiment of the present application comprises sequentially acquiring each to-be-aligned video frame and buffering; after buffering each to-be-aligned video frame, if the number of the to-be-aligned video frames currently buffered is not less than a first number, mapping matrices between the to-be-aligned video frame and a corresponding second number of adjacent video frames are calculated respectively; wherein the second number of adjacent video frames comprises: a center video frame in the last buffered first number of to-be-aligned video frames and video frames between the center video frame and the to-be-aligned video frame according to the buffering order; for each to-be-aligned video frame except the center video frame in the last buffered first number of to-be-aligned video frames, the to-be-aligned video frame is processed based on the mapping matrix between the to-be-aligned video frame and the center video frame, to obtain a to-be-fused video frame; and the to-be-fused video frames and the center video frame are fused to obtain a target video frame.

[0080] Based on the above processing, after buffering each to-be-aligned video frame, if the number of the to-be-aligned video frames currently buffered reaches a certain number, the certain number of to-be-aligned video frames can be fused, which can effectively reduce the noise in the fused target video frame and improve the image quality of the video image. In addition, the video frames can be fused according to the mapping matrix between each to-be-aligned video frame and the center video frame, that is, each to-be-aligned video frame and the center video frame can be aligned and fused, which can reduce the difference between the to-be-fused video frame and the center video frame, and further improve the image quality of the fused target video frame, thereby improving the image quality of the video image.

[0081] Of course, implementing any product or method of the present application does not necessarily require achieving all the advantages described above at the same time. BRIEF DESCRIPTION OF DRAWINGS

[0082] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description only constitute some embodiments of the present application, and other embodiments can also be obtained by those skilled in the art based on these drawings.

[0083] Figure 1 A flowchart of an image processing method provided by an embodiment of the present application;

[0084] Figure 2 A flowchart of another image processing method provided by an embodiment of the present application;

[0085] Figure 3 An effect comparison schematic diagram of a video frame processing based on a noise reduction network model provided by an embodiment of the present application;

[0086] Figure 4 A flowchart illustrating yet another image processing method provided in an embodiment of the present invention;

[0087] Figure 5 A flowchart illustrating another image processing method provided in an embodiment of the present invention;

[0088] Figure 6 This is a schematic diagram illustrating cached information of an electronic device at different times, provided by an embodiment of the present invention.

[0089] Figure 7 This is a schematic diagram of a process for fusing video frames according to an embodiment of the present invention;

[0090] Figure 8 This is a schematic diagram illustrating the determination of the optimal matching point in adjacent frames, provided by an embodiment of the present invention.

[0091] Figure 9 This is a schematic diagram of a matching structure using a sliding block, provided by an embodiment of the present invention;

[0092] Figure 10 This is a structural diagram of an image processing apparatus provided in an embodiment of the present invention;

[0093] Figure 11 This is a structural diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0094] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of the present invention.

[0095] In related technologies, when the ambient light is insufficient, the video images captured by the video acquisition device contain a lot of noise, resulting in low image quality.

[0096] To address the aforementioned problems, embodiments of the present invention provide an image processing method applicable to electronic devices, such as cameras. The electronic device can determine the video frames to be aligned based on acquired video images. Furthermore, the electronic device can process these images using the image processing method provided in this embodiment to obtain the target video frames, thus obtaining a video image containing all the target video frames—that is, a fused video image.

[0097] See Figure 1 , Figure 1 A flowchart of an image processing method provided in an embodiment of the present invention, the method may include the following steps:

[0098] S101: sequentially acquire each to-be-aligned video frame and buffer.

[0099] S102: after buffering each to-be-aligned video frame, if the number of currently buffered to-be-aligned video frames is not less than a first number, calculate mapping matrices between the to-be-aligned video frames and a corresponding second number of adjacent video frames respectively.

[0100] The second number of adjacent video frames include: a center video frame in the last buffered first number of to-be-aligned video frames in the order of buffering, and video frames between the center video frame and the to-be-aligned video frames.

[0101] S103: for each to-be-aligned video frame except the center video frame in the last buffered first number of to-be-aligned video frames in the order of buffering, process the to-be-aligned video frame based on the mapping matrix between the to-be-aligned video frame and the center video frame to obtain a to-be-fused video frame.

[0102] S104: fuse each to-be-fused video frame and the center video frame to obtain a target video frame.

[0103] Based on the above processing, after buffering each to-be-aligned video frame, if the number of currently buffered to-be-aligned video frames reaches a certain number, the certain number of to-be-aligned video frames can be fused, which can effectively reduce the noise in the fused target video frame and improve the image quality of the video image. In addition, the video frames can be fused according to the mapping matrix between each to-be-aligned video frame and the center video frame, that is, the to-be-aligned video frames and the center video frame can be aligned and fused, which can reduce the difference between the to-be-fused video frames and the center video frame, and further improve the image quality of the fused target video frame, thereby improving the image quality of the video image.

[0104] For step S101, the to-be-aligned video frames can be determined based on a pre-acquired video image (which can be referred to as a to-be-processed video image). For example, the to-be-processed video image can be a video image collected by an image collection device in a low-illumination condition. The to-be-processed video image can be in RAW format.

[0105] In an implementation manner, the to-be-aligned video frames are video frames in a to-be-processed video. That is, each video frame in the to-be-processed video can be sequentially acquired in the order of timestamps as a to-be-aligned video frame and buffered.

[0106] In another implementation manner, referring to Figure 2 , in the basis of Figure 1 , Figure 2A flowchart of another image processing method provided by the embodiment of the present application is shown. Step S101 includes:

[0107] S1011: sequentially acquire each video frame from the to-be-processed video.

[0108] S1012: based on the pre-trained denoising network model, process the video frame to obtain a to-be-aligned video frame and cache the to-be-aligned video frame.

[0109] The denoising network model can be a Unet network model or a Vnet network model.

[0110] As shown in Figure 3 , a comparison diagram of effects of processing a video frame based on a denoising network model is provided by the embodiment of the present application. Figure 3 In the comparison diagram, the left image is a video frame before processing, and the right image is a video frame after processing based on the denoising network model. The denoising network model can effectively remove noise in the video frame, and can further improve the image quality of the fused target video frame to improve the image quality of the video image. Figure 3

[0111] After acquiring each to-be-aligned video frame, the electronic device can cache the to-be-aligned video frame in a specified storage space. In addition, the electronic device can also record the caching order of each to-be-aligned video frame. For example, the to-be-aligned video frame can be cached in a queue (which can be referred to as a to-be-aligned video frame queue), and accordingly, based on the to-be-aligned video frames stored in the queue, the caching order of each to-be-aligned video frame can be determined.

[0112] For step S102, after caching each to-be-aligned video frame, the electronic device can fuse a first number of to-be-aligned video frames including the to-be-aligned video frame, that is, the first number represents the number of to-be-aligned video frames that need to be fused each time. The larger the first number, the more to-be-aligned video frames that need to be fused each time, which increases the calculation amount each time, and accordingly, improves the image quality of the fused target video frame. The smaller the first number, the fewer to-be-aligned video frames that need to be fused each time, which reduces the calculation amount each time, and accordingly, reduces the image quality of the fused target video frame. Therefore, the first number can be determined by the technical personnel in combination with the calculation amount and the image quality. For example, the first number can be 5.

[0113] The second number is the number of other cached to-be-aligned video frames that need to calculate the mapping matrix with the to-be-aligned video frame after caching one to-be-aligned video frame each time. It can be understood that the second number is not greater than the first number.

[0114] ​The mapping matrix between two video frames can represent the mapping relationship of pixel coordinates between pixel points of the two video frames. The mapping matrix can also be referred to as the homography matrix between the two video frames. For example, for two video frames adjacent in time stamp, there is a difference between the image contents of the two video frames. Therefore, for a pixel point in a video frame, the pixel coordinates of the pixel point can be processed based on the mapping matrix to determine the corresponding pixel coordinates of the pixel point in another video frame. The above process is to align the video frame to another video frame.

[0115] The mapping matrix between the to-be-aligned video frame and the adjacent video frame can represent the mapping relationship from the adjacent video frame to the to-be-aligned video frame (which can be referred to as a first mapping relationship), or can also represent the mapping relationship from the to-be-aligned video frame to the adjacent video frame (which can be referred to as a second mapping relationship). It can be understood that the first mapping relationship and the second mapping relationship are reciprocal. For example, the first mapping matrix representing the mapping relationship from video frame A to video frame B is the inverse matrix of the second mapping matrix representing the mapping relationship from video frame B to video frame A.

[0116] There is a center video frame in the first number of to-be-aligned video frames that need to be fused each time. According to the order of caching, the center video frame can be a to-be-aligned video frame at the center position in the first number of to-be-aligned video frames. For example, the center video frame can be any one of the video frames other than the last one and the first one in the first number of to-be-aligned video frames.

[0117] For example, the first number is an odd number greater than 1, and according to the order of caching, the center video frame is a to-be-aligned video frame at the center position in the first number of to-be-aligned video frames. For example, the first number is 5, and the center video frame is the third video frame in the five to-be-aligned video frames; the first number is 7, and the center video frame is the fourth video frame in the seven to-be-aligned video frames. Correspondingly, the relationship between the first number M and the second number N can be represented by formula (1):

[0118]

[0119] For another example, the first number can also be an even number greater than 1, and correspondingly, according to the order of caching, the center video frame is a video frame at the center position on both sides of the first number of to-be-aligned video frames. For example, the first number is 4, and the center video frame is the second video frame or the third video frame in the four to-be-aligned video frames; the first number is 6, and the center video frame is the third video frame or the fourth video frame in the six to-be-aligned video frames.

[0120] Based on the above processing, since the center video frame is a video frame in a central position in the first number of to-be-aligned video frames, the difference between the center video frame and other video frames in the first number of to-be-aligned video frames is relatively small as a whole, and thus, when the other video frames are aligned to the center video frame, the calculation amount can be reduced, the accuracy of the aligned to-be-fused video frame can be improved, and accordingly, the image quality of the fused target video frame can be further improved, so as to improve the stability of the video and improve the image quality of the video image.

[0121] In an implementation manner, when the to-be-aligned video frames are buffered, since the number of to-be-aligned video frames that start to be buffered is small, as the number of buffered to-be-aligned video frames increases, when the number of buffered to-be-aligned video frames is greater than the second number, the mapping matrix between the last buffered to-be-aligned video frame and the corresponding second number of adjacent video frames can be directly calculated. That is, when the number of buffered to-be-aligned video frames is less than the first number and greater than the second number, the mapping matrix between the video frames can also be calculated.

[0122] For example, the first number is 5 and the second number is 2. The electronic device can acquire the to-be-aligned video frame Q1 and buffer it into the to-be-aligned video frame queue Q, at this time, the number of currently buffered to-be-aligned video frames is 1, which is less than the second number and does not reach the first number, and thus, the mapping matrix does not need to be calculated and the fusion processing does not need to be performed; then, the to-be-aligned video frame Q2 can be acquired and buffered into the to-be-aligned video frame queue Q, at this time, the number of currently buffered to-be-aligned video frames is 2, which is less than the second number and does not reach the first number, and thus, the mapping matrix does not need to be calculated and the fusion processing does not need to be performed.

[0123] Then, the electronic device acquires the to-be-aligned video frame Q3 and buffers it into the to-be-aligned video frame queue Q, at this time, the number of currently buffered to-be-aligned video frames is 3, which is greater than the second number. Therefore, the mapping matrix H 13 between the to-be-aligned video frame Q3 and the to-be-aligned video frame Q1 and the mapping matrix H 23 between the to-be-aligned video frame Q3 and the to-be-aligned video frame Q2 can be calculated. The mapping matrix H 13 represents the mapping relationship from the to-be-aligned video frame Q1 to the to-be-aligned video frame Q3, and the mapping matrix H 23 represents the mapping relationship from the to-be-aligned video frame Q2 to the to-be-aligned video frame Q3. Since the number of currently buffered to-be-aligned video frames is 3, which does not reach the first number, the mapping matrix does not need to be calculated and the fusion processing does not need to be performed.

[0124] Then, the electronic device acquires the video frame Q4 to be aligned and caches it in the video frame queue Q. At this point, the number of cached video frames to be aligned is 4, which is greater than the second number. Therefore, the mapping matrix H between the video frame Q4 and the video frame Q2 to be aligned can be calculated. 24 The mapping matrix H between video frame Q4 and video frame Q3 to be aligned 34 Wherein, the mapping matrix H 24 The mapping matrix H represents the mapping relationship from video frame Q2 to video frame Q4 to be aligned. 34 This represents the mapping relationship from video frame Q3 to video frame Q4 to be aligned. Since the number of cached video frames to be aligned is currently 4, which is less than the first number, there is no need to calculate the mapping matrix and no need to perform fusion processing.

[0125] Then, the electronic device acquires the video frame Q5 to be aligned and caches it in the video frame queue Q. At this point, the number of cached video frames to be aligned is 5, which is greater than the second number. Therefore, the mapping matrix H between video frame Q5 and video frame Q3 to be aligned can be calculated. 35 The mapping matrix H between video frame Q5 and video frame Q4 to be aligned 45 Wherein, the mapping matrix H 35 The mapping matrix H represents the mapping relationship from video frame Q3 to video frame Q5 to be aligned. 45 This indicates the mapping relationship from video frame Q4 to video frame Q5 to be aligned. Since the number of cached video frames to be aligned is 5, reaching the first number, fusion processing can be performed based on step S103, that is, the first number of cached video frames to be aligned (i.e., video frames Q1-Q5 to be aligned) are fused.

[0126] Regarding step S103, each of the first number of video frames to be aligned, except for the center video frame, can be aligned to the center video frame. In one implementation, if the mapping matrix calculated in step S102 represents the mapping relationship from other video frames to be aligned to the center video frame, then the other video frames to be aligned can be processed directly based on the mapping matrix.

[0127] In another implementation, if the mapping matrix calculated in step S102 represents the mapping relationship from the center video frame to other video frames to be aligned, then step S103 includes:

[0128] Step 1: According to the order of caching, for each of the first number of last-cached to-be-aligned video frames except the center video frame, if the pre-calculated mapping matrix of the to-be-aligned video frame represents the mapping relationship from the center video frame to the to-be-aligned video frame, the inverse matrix of the mapping matrix corresponding to the to-be-aligned video frame is calculated.

[0129] Step 2: The to-be-aligned video frame is processed based on the calculated inverse matrix to obtain a to-be-fused video frame.

[0130] Since the mapping matrix calculated in step S102 represents the mapping relationship from the center video frame to other to-be-aligned video frames, if the to-be-aligned video frame is to be aligned to the center video frame, the inverse matrix of the mapping matrix can be calculated, and then the to-be-aligned video frame can be processed based on the calculated inverse matrix, so that the video frame after the to-be-aligned video frame is aligned to the center video frame (i.e., the to-be-fused video frame) can be obtained.

[0131] Since the to-be-aligned video frames in the present application are fused, and the video frames can be aligned in the fusion process. Therefore, for two video frames, each video frame needs to be aligned to the other video frame in two different fusion processes. When aligning in the two fusion processes, only one mapping matrix needs to be recorded to obtain two mapping relationships by using the inverse matrix of the mapping matrix, that is, the mapping relationship between the two video frames is represented by one mapping matrix and the inverse matrix of the mapping matrix. Further, it is not necessary to calculate the mapping matrix multiple times, the mapping matrix can be reused, the calculation amount is reduced, and the efficiency of image processing is improved.

[0132] For example, the first number is 5 and the second number is 2. After each to-be-aligned video frame is cached, the mapping matrix between the to-be-aligned video frame and the adjacent video frame can be calculated, and the mapping matrix represents the mapping relationship from the adjacent video frame to the to-be-aligned video frame. Subsequently, the inverse matrix of the mapping matrix can also be calculated, and the inverse matrix of the mapping matrix represents the mapping relationship from the to-be-aligned video frame to the adjacent video frame.

[0133] After the to-be-aligned video frame Q5 is cached, the first number of last-cached to-be-aligned video frames in the to-be-aligned video frame queue Q include: to-be-aligned video frames Q1, Q2, Q3, Q4 and Q5, at this time, the center video frame is to-be-aligned video frame Q3, and the calculated mapping matrix includes: mapping matrix H 13 , H 23 , H 24 , H 34 , H 35 , H 45 . Further, the electronic device can calculate the inverse matrix of the mapping matrix H 13The video frame Q1 to be aligned is processed to obtain the video frame Q1 to be merged. * Based on mapping matrix H 23 The video frame Q2 to be aligned is processed to obtain the video frame Q2 to be merged. * ; Calculate the mapping matrix H 34 inverse matrix And based on this inverse matrix The video frame Q4 to be aligned is processed to obtain the video frame Q4 to be merged. * ; Calculate the mapping matrix H 35 inverse matrix Based on the video frame Q5 to be aligned, the video frame Q5 to be merged is obtained. * Among them, the mapping matrix This represents the mapping relationship from video frame Q4 to video frame Q3 to be aligned; the mapping matrix is ​​used. This represents the mapping relationship from video frame Q5 to video frame Q3. Then, video frames Q1, Q2, Q3, Q4, and Q5 can be merged.

[0134] After caching video frame Q5, continue caching video frames Q6 and Q7. At this point, the last cached video frames in the alignment queue Q are Q3, Q4, Q5, Q6, and Q7, with the center video frame being video frame Q5. The calculated mapping matrix includes: mapping matrix H. 13 H 23 H 24 H 34 H 35 H 45 H 46 H 56 H 57 H 67 Furthermore, electronic devices can be based on the mapping matrix H 35 The video frame Q3 to be aligned is processed to obtain the video frame Q3 to be merged. * Based on mapping matrix H 45 The video frame Q4 to be aligned is processed to obtain the video frame Q4 to be merged. * ; Calculate the mapping matrix H 56 inverse matrix And based on this inverse matrix The video frame Q6 to be aligned is processed to obtain the video frame Q6 to be merged. * ; Calculate the mapping matrix H 57 inverse matrix Based on the video frame Q7 to be aligned, the video frame Q7 to be merged is obtained. * Among them, the mapping matrix This represents the mapping relationship from video frame Q6 to video frame Q5 to be aligned; the mapping matrix is ​​used. This represents the mapping relationship from video frame Q7 to video frame Q5. Then, video frames Q3, Q4, Q5, Q6, and Q7 can be merged.

[0135] It is evident that in one of the two fusion processes described above, the mapping matrix H is used. 35 inverse matrix Another use of the mapping matrix H 35 In other words, the mapping matrix H can be achieved during the two fusion processes. 35 The mapping matrix can be reused, eliminating the need to calculate it multiple times. This reduces computational load and improves image processing efficiency.

[0136] For step S104, the weights of each video frame to be merged and the center video frame can be the same or different.

[0137] In one embodiment, see Figure 4 ,exist Figure 2 On this basis, Figure 4 A flowchart of another image processing method provided in an embodiment of the present invention, wherein step S104 includes:

[0138] S1041: For multiple pixels with the same pixel coordinates in each video frame to be fused and the center video frame, determine the weight of each of the multiple pixels based on the weights of each video frame to be fused and the center video frame.

[0139] Among them, the weights of each video frame to be merged and the center video frame follow a Gaussian distribution, with the center video frame having the largest weight. Furthermore, according to the order of caching, the distance between the video frame to be aligned and the center video frame corresponding to each video frame to be merged is negatively correlated with the weight of that video frame to be merged.

[0140] S1042: Calculate the weighted sum of the pixel values ​​of the multiple pixels according to their respective weights, and obtain the pixel value of the corresponding pixel coordinate in the target video frame.

[0141] Since the weights of each video frame to be merged and the center video frame conform to a Gaussian distribution, the weights of each video frame to be merged and the center video frame can be obtained based on a Gaussian function.

[0142] The distance between the to-be-aligned video frame corresponding to each to-be-blended video frame and the center video frame is negatively correlated with the weight of the to-be-blended video frame in the order of the cache. That is, for each to-be-blended video frame, if the distance between the to-be-aligned video frame corresponding to the to-be-blended video frame and the center video frame is greater, the weight of the to-be-blended video frame is smaller; if the distance between the to-be-aligned video frame corresponding to the to-be-blended video frame and the center video frame is smaller, the weight of the to-be-blended video frame is greater. The distance between the to-be-aligned video frame and the center video frame represents the distance between the to-be-aligned video frame and the center video frame in the order of the cache.

[0143] In an implementation manner, for each to-be-blended video frame, the weight of the to-be-blended video frame can be determined as the weight of each pixel point in the to-be-blended video frame, that is, the weights of the pixel points in the same to-be-blended video frame are the same; and the weight of the center video frame can be determined as the weight of each pixel point in the center video frame.

[0144] For example, the first number is 5, and the second number is 2. The weight of the center video frame is 0.5, and the weights of the four to-be-blended video frames are 0.1, 0.15, 0.15, and 0.1 respectively.

[0145] In another implementation manner, the step S1041 can include:

[0146] Step one: for each pixel point of the same pixel coordinates in each to-be-blended video frame and the center video frame, if the pixel point belongs to the to-be-blended video frame, and the difference between the pixel point and the pixel point of the same pixel coordinates in the center video frame is less than a preset threshold, the weight of the pixel point is determined as the weight of the to-be-blended video frame to which the pixel point belongs.

[0147] Step two: if the pixel point belongs to the to-be-blended video frame, and the difference between the pixel point and the pixel point of the same pixel coordinates in the center video frame is not less than the preset threshold, the weight of the pixel point is determined as 0.

[0148] Step three: based on the weights of the pixel points of the same pixel coordinates in each to-be-blended video frame, the weight of the pixel point of the same pixel coordinates in the center video frame is calculated.

[0149] For each pixel point of each to-be-blended video frame, the difference between the pixel point and the pixel point of the same pixel coordinates in the center video frame (which can be referred to as a center pixel point) can be represented by the absolute value of the difference between the pixel values of the pixel point and the corresponding center pixel point.

[0150] If the difference between the pixel point and the corresponding center pixel point is not less than the preset threshold, it indicates that the difference between the image information embodied by the pixel point in the to-be-fused video frame and the image information embodied by the corresponding center pixel point is large, i.e., the accuracy of the pixel point obtained by aligning the to-be-aligned video frames is low. Therefore, in order to improve the accuracy of fusion, the weight of the pixel point can be determined as 0. Conversely, if the difference between the pixel point and the corresponding center pixel point is less than the preset threshold, it indicates that the difference between the image information embodied by the pixel point in the to-be-fused video frame and the image information embodied by the corresponding center pixel point is small, i.e., the accuracy of the pixel point obtained by aligning the to-be-aligned video frames is high. Therefore, the weight of the to-be-fused video frame to which the pixel point belongs can be determined as the weight of the pixel point.

[0151] For example, for each pixel point of the same pixel coordinates in each to-be-fused video frame and the center video frame Q3, the weight of each pixel point can be determined based on formula (2), which is as follows:

[0152]

[0153] wherein, α1(x,y) represents the weight of the pixel point at the pixel coordinates (x,y) in the to-be-fused video frame i , α2(x,y) represents the weight of the pixel point at the pixel coordinates (x,y) in the to-be-fused video frame i , α3(x,y) represents the weight of the pixel point at the pixel coordinates (x,y) in the center video frame Q3, and α4(x,y) and α5(x,y) represent the weight of the pixel point at the pixel coordinates (x,y) in the to-be-fused video frames and . represents the difference between the pixel point at the pixel coordinates (x,y) in the to-be-fused video frame and the pixel point at the pixel coordinates (x,y) in the center video frame Q3. The value of i can be 1, 2, 4, and 5.

[0154] For example, the sum of the weight of the pixel point of the same pixel coordinates in each to-be-fused video frame and the weight of the pixel point of the same pixel coordinates in the center video frame can be 1. Therefore, the sum of the weight of the pixel point of the same pixel coordinates in each to-be-fused video frame can be calculated, and the difference between 1 and the sum can be calculated as the weight of the pixel point of the same pixel coordinates in the center video frame.

[0155] After the weight of the pixel point at the pixel coordinates (x,y) in each to-be-fused video frame is determined, the weight of the pixel point at the pixel coordinates (x,y) in the center video frame Q3 can be calculated based on the weight of the pixel point at the pixel coordinates (x,y) in each to-be-fused video frame , as shown in formula (3):

[0156] α3(x,y)=1-α1(x,y)-α2(x,y)-α4(x,y)-α5(x,y) (3)

[0157] wherein a3(x, y) represents the weight of the pixel point at pixel coordinate (x, y) in the center video frame Q3.

[0158] After determining the weight of each pixel point at the same pixel coordinate in each to-be-fused video frame and the center video frame, the electronic device can calculate the weighted sum of the pixel values of the plurality of pixel points to obtain the pixel value of the corresponding pixel coordinate in the target video frame.

[0159] For example, based on the to-be-fused video frame to-be-fused video frame center video frame Q3, to-be-fused video frame and to-be-fused video frame The process of calculating the pixel value of the pixel coordinate in the target video frame M is shown in formula (4):

[0160]

[0161] wherein M(x, y) represents the pixel value of the pixel point at pixel coordinate (x, y) in the target video frame M, represents the pixel value of the pixel point at pixel coordinate (x, y) in the to-be-fused video frame , a1 represents the weight of the pixel point at pixel coordinate (x, y) in the to-be-fused video frame , represents the pixel value of the pixel point at pixel coordinate (x, y) in the to-be-fused video frame , a2 represents the weight of the pixel point at pixel coordinate (x, y) in the to-be-fused video frame , Q3(x, y) represents the pixel value of the pixel point at pixel coordinate (x, y) in the center video frame Q3, and a3 represents the weight of the pixel point at pixel coordinate (x, y) in the center video frame Q3, represents the pixel value of the pixel point at pixel coordinate (x, y) in the to-be-fused video frame , a4 represents the weight of the pixel point at pixel coordinate (x, y) in the to-be-fused video frame , represents the pixel value of the pixel point at pixel coordinate (x, y) in the to-be-fused video frame , a5 represents the weight of the pixel point at pixel coordinate (x, y) in the to-be-fused video frame .

[0162] Based on the above processing, when fusing each to-be-fused video frame and the center video frame, if the difference between the pixel value of a pixel point in the to-be-fused video frame and the pixel point at the same pixel coordinate in the center video frame is large, the weight of the pixel point in the to-be-fused video frame can be set to 0, so that the problem of ghosting in the target video frame caused by fusion can be avoided, and the image quality of the video image can be improved.

[0163] In one embodiment, referring to Figure 5 , Figure 5 A flowchart of another image processing method provided by an embodiment of the present application is shown in FIG. 10. Based on the method shown in FIG. 9, after step S1042, the method further includes: Figure 4

[0164] S105: deleting the first to-be-aligned video frame that is cached earliest from the last cached first number of to-be-aligned video frames in the order of caching.

[0165] In one implementation, the to-be-aligned video frame queue can have a fixed length, for example, the fixed length can be the first number. Therefore, after the target video frame is determined based on the last cached first number of to-be-aligned video frames, the to-be-aligned video frame at the head of the to-be-aligned video frame queue has been fused, and the to-be-aligned video frame is not needed in the subsequent fusion process, so the to-be-aligned video frame can be deleted. Subsequently, the next to-be-aligned video frame can be continuously cached at the tail of the to-be-aligned video frame queue.

[0166] Based on the above processing, the redundant video frames in the cached to-be-aligned video frames can be deleted, that is, in the cached to-be-aligned video frames, the to-be-aligned video frames that will not be used in the subsequent image processing process are deleted. Further, the storage space required for storing the to-be-aligned video frames when processing the video frames in the to-be-processed video can be reduced, and the waste of storage space can be avoided.

[0167] In one embodiment, after step S102, the method further includes:

[0168] Step 3: caching each mapping matrix calculated in the order of caching between the corresponding second number of adjacent video frames.

[0169] After step S104, the method further includes:

[0170] Step 4: deleting the second number of mapping matrices that are cached earliest in the order of caching.

[0171] ​In an implementation, for each video frame to be aligned, the electronic device can calculate a mapping matrix between the video frame to be aligned and a corresponding second number of adjacent video frames, and cache each calculated mapping matrix in a specified storage space in the order of the corresponding second number of adjacent video frames. In addition, the electronic device can also record the caching order of each mapping matrix. For example, the mapping matrix can be cached in a queue (which can be referred to as a mapping matrix queue), and accordingly, the caching order of each mapping matrix can be determined based on the mapping matrices stored in the queue.

[0172] In an implementation, the mapping matrix queue can have a fixed length, for example, the fixed length can be the product of the second number and a third number, and the third number is the second number plus 1. Therefore, after the target video frame is determined based on the last cached first number of video frames to be aligned, since the second number of mapping matrices at the head of the mapping matrix queue have been used to process the video frames to be aligned and obtain the video frames to be fused, and the second number of mapping matrices are not needed in the subsequent process of obtaining the video frames to be fused, the second number of mapping matrices can be deleted. Subsequently, the mapping matrix between the next video frame to be aligned and the corresponding second number of adjacent video frames can be continuously cached at the tail of the mapping matrix queue.

[0173] Based on the above processing, redundant mapping matrices in the cached mapping matrices can be deleted, that is, in the cached mapping matrices, the mapping matrices that will not be used in the subsequent image processing are deleted. Further, the storage space required for storing the mapping matrices when processing the video frames in the video to be processed can be reduced, and the waste of storage space can be avoided.

[0174] For example, as shown in FIG. 2, Figure 6 Figure 6 FIG. 2 is a schematic diagram of the cached information of the electronic device at different times provided by an embodiment of the present application. In the figure, the first number is 5, and the second number is 2.

[0175] At time 1, the electronic device obtains the video frame to be aligned Q1 and caches it in the video frame queue Q. At this time, the number of currently cached video frames to be aligned is 1, which is less than the second number and does not reach the first number, so it is not necessary to calculate the mapping matrix and perform the fusion processing. That is, at time 1, the video frame to be aligned Q1 is cached in the video frame queue Q.

[0176] ​At time 2, the electronic device acquires the to-be-aligned video frame Q2 and caches it into the to-be-aligned video frame queue Q. At this time, the number of to-be-aligned video frames that have been cached is 2, which is not greater than the second number, and has not reached the first number, so there is no need to calculate the mapping matrix and there is no need to perform the fusion processing. That is, at time 2, the to-be-aligned video frame queue Q caches the to-be-aligned video frames Q1 and Q2.

[0177] At time 3, the electronic device acquires the to-be-aligned video frame Q3 and caches it into the to-be-aligned video frame queue Q. At this time, the number of to-be-aligned video frames that have been cached is 3, which is greater than the second number. Therefore, the mapping matrix H 13 between the to-be-aligned video frame Q3 and the to-be-aligned video frame Q1, the mapping matrix H 23 between the to-be-aligned video frame Q3 and the to-be-aligned video frame Q2 can be calculated and cached into the mapping matrix queue H. That is, at time 3, the to-be-aligned video frame queue Q caches the to-be-aligned video frames Q1, Q2 and Q3, and the mapping matrix queue H caches the mapping matrices H 13 , H 23 . Since the number of to-be-aligned video frames that have been cached is 3, which has not reached the first number, there is no need to perform the fusion processing.

[0178] At time 4, the electronic device acquires the to-be-aligned video frame Q4 and caches it into the to-be-aligned video frame queue Q. At this time, the number of to-be-aligned video frames that have been cached is 4, which is greater than the second number. Therefore, the mapping matrix H 24 between the to-be-aligned video frame Q4 and the to-be-aligned video frame Q2, the mapping matrix H 34 between the to-be-aligned video frame Q4 and the to-be-aligned video frame Q3 can be calculated and cached into the mapping matrix queue H. That is, at time 4, the to-be-aligned video frame queue Q caches the to-be-aligned video frames Q1, Q2, Q3 and Q4, and the mapping matrix queue H caches the mapping matrices H 13 , H 23 , H 24 , H 34 . Since the number of to-be-aligned video frames that have been cached is 4, which has not reached the first number, there is no need to perform the fusion processing.

[0179] At time 5, the electronic device acquires the to-be-aligned video frame Q5 and caches it into the to-be-aligned video frame queue Q. At this time, the number of to-be-aligned video frames that have been cached is 5, which is greater than the second number. Therefore, the mapping matrix H 35 between the to-be-aligned video frame Q5 and the to-be-aligned video frame Q3, the mapping matrix H 45and cached into the mapping matrix queue H. That is, at time 5, the to-be-aligned video frame queue Q caches to-be-aligned video frames Q1, Q2, Q3, Q4, Q5, and the mapping matrix queue H caches mapping matrices H 13 , 23 , 24 , 34 , 35 , 45 . Since the number of currently cached to-be-aligned video frames is 5, reaching the first number, at this time, the last cached first number of to-be-aligned video frames (i.e., to-be-aligned video frames Q1-Q5) can be fused to obtain the target video frame when the to-be-aligned video frame center video frame Q3 is the center video frame. And delete to-be-aligned video frame Q1 from the to-be-aligned video frame queue Q, and delete mapping matrix H 13 from the mapping matrix queue H. 23 .

[0180] At time 6, the electronic device acquires to-be-aligned video frame Q6 and caches it into the to-be-aligned video frame queue Q. At this time, the number of currently cached to-be-aligned video frames is 5, which is greater than the second number. Therefore, the mapping matrix H 46 between to-be-aligned video frame Q6 and to-be-aligned video frame Q4, and the mapping matrix H 45 between to-be-aligned video frame Q6 and to-be-aligned video frame Q5 can be calculated, and cached into the mapping matrix queue H. That is, at time 6, the to-be-aligned video frame queue Q caches to-be-aligned video frames Q2, Q3, Q4, Q5, Q6, and the mapping matrix queue H caches mapping matrices H 24 , 34 , 35 , 45 , 46 , 56 . Since the number of currently cached to-be-aligned video frames is 5, reaching the first number, at this time, the last cached first number of to-be-aligned video frames (i.e., to-be-aligned video frames Q2-Q6) can be fused to obtain the target video frame when the to-be-aligned video frame center video frame Q4 is the center video frame. And delete to-be-aligned video frame Q2 from the to-be-aligned video frame queue Q, and delete mapping matrix H 24 from the mapping matrix queue H. 34 .

[0181] In an implementation manner, as shown in FIG. 8, Figure 7 , a flowchart for fusing video frames is provided in an embodiment of the present application. Figure 7 Figure 7 ​In the specific implementation, the first number is 5 and the second number is 2. After the electronic device caches each original frame (i.e., a video frame in the to-be-processed video), the electronic device can denoise the video frame, i.e., process the video frame based on the pre-trained denoising network model to obtain a denoised frame (i.e., a to-be-aligned video frame), and cache the to-be-aligned video frame in the to-be-aligned video frame queue. Next, the electronic device calculates the alignment relationship (i.e., the mapping matrix H t-2,t t-1,t ) between the to-be-aligned video frame and the corresponding second number of adjacent video frames, and caches the mapping matrix in the mapping matrix queue H, to realize alignment relationship enqueueing. In the specific implementation, t represents the last cached to-be-aligned video frame in the order of caching; t-1 represents the first adjacent video frame before the to-be-aligned video frame in the order of caching; t-2 represents the second adjacent video frame before the to-be-aligned video frame in the order of caching; H t-1,t represents the mapping relationship between the first adjacent video frame before the to-be-aligned video frame and the to-be-aligned video frame; and H t-2,t represents the mapping relationship between the second adjacent video frame before the to-be-aligned video frame and the to-be-aligned video frame.

[0182] At this time, the to-be-aligned video frame queue caches 5 to-be-aligned video frames, and the mapping matrix queue H caches the mapping matrices H 13 23 24 34 35 and H 45 . The number of to-be-aligned video frames reaches the first number, and therefore, the 5 to-be-aligned video frames can be fused. The third to-be-aligned video frame is a center video frame. For each to-be-aligned video frame in the first number of to-be-aligned video frames except the third to-be-aligned video frame, the electronic device processes the to-be-aligned video frame based on the mapping matrix between the to-be-aligned video frame and the center video frame to obtain an aligned frame (i.e., a to-be-fused video frame). The electronic device can also delete the first to-be-aligned video frame from the head of the to-be-aligned video frame queue, and delete the first and second mapping matrices from the head of the mapping matrix queue H, to realize alignment relationship dequeueing. Subsequently, the electronic device fuses each to-be-fused video frame and the center video frame to output a target video frame.

[0183] In an embodiment, the above-mentioned step of respectively calculating the mapping matrix between the to-be-aligned video frame and the corresponding second number of adjacent video frames includes:

[0184] Step 5: For each adjacent video frame, the electronic device determines, as an aligned pixel point, a pixel point in the adjacent video frame that matches a corner point in the to-be-aligned video frame based on a block matching algorithm.

[0185] ​​​​​Step 6: Based on the pixel coordinates of the corner points and the pixel coordinates of the aligned pixels in the video frame to be aligned, calculate the mapping matrix between the video frame to be aligned and the adjacent video frame according to the Random Sample Consensus (RANSAC) algorithm.

[0186] Corner points can be the connection points of object outlines in a video frame, or the endpoints of line segments in an image.

[0187] In one implementation, the electronic device can obtain all corner points (which can be called first corner points) in the video frame to be aligned based on the Harris corner detection method. Alternatively, it can divide the video frame into several regions based on its aspect ratio. For a given region, if multiple first corner points exist, a corner point is randomly selected as a second corner point (which can be called a matching anchor point), and subsequently, the alignment pixel point corresponding to this second corner point is determined. If only one first corner point exists in the region, that first corner point can be used as the second corner point, and subsequently, the alignment pixel point corresponding to this second corner point is determined.

[0188] like Figure 8 As shown, Figure 8 This is a schematic diagram illustrating how to determine the optimal matching point in adjacent frames, as provided in an embodiment of the present invention. Figure 8 In the diagram, the left image represents adjacent frames (i.e., adjacent video frames), and the right image represents the center frame (i.e., the center video frame). For a corner point 'a' in the center video frame, the block to be aligned containing corner point 'a' can be determined. Furthermore, based on the SAD (Sum of Absolute Differences) algorithm, matching can be performed within the corresponding search range in adjacent frames to determine the optimal matching point, i.e., the aligned pixel point 'a' corresponding to corner point 'a' in the adjacent frames.

[0189] For example, if the size of the video frame to be aligned is 2000×1000, it can be divided into 20×10 regions of size 10×10. If corner points exist in each region, 200 second corner points can be determined from the video frame to be aligned.

[0190] For any two video frames, after determining all second corner points in one of the video frames, the pixel coordinates of each second corner point can be obtained. A block to be aligned, centered on that second corner point, can then be determined within the video frame to which that second corner point belongs. In the other video frame, a search range can be determined based on the pixel coordinates of the second corner point, and a sliding block can be used to perform matching within that search range. For example... Figure 9 As shown, Figure 9A structure diagram for matching by a sliding block is provided for an embodiment of the present application. The search range can be a range centered on the second corner point. Then, the sliding block can be slid by a preset step size. For each position of the sliding block in the search range, the sum of pixel values of all pixel points in the sliding block at the position is calculated, and the absolute value of the difference between the sum and the sum of pixel values of all pixel points in the to-be-aligned block is calculated. Then, the position corresponding to the minimum absolute value can be determined, and the pixel point at the center of the sliding block at the position is determined as the aligned pixel point.

[0191] After determining the aligned pixel points corresponding to each second corner point in the video frame, based on the pixel coordinates of each second corner point and the pixel coordinates of the aligned pixel points corresponding to each second corner point, a mapping matrix between the two video frames is calculated according to the RANSAC algorithm. For example, the obtained mapping matrix h can be as shown in matrix (5):

[0192]

[0193] Based on the above processing, the determined corner points can be uniformly distributed in the to-be-aligned video frame, so as to improve the accuracy of the obtained mapping matrix. Moreover, by removing redundant corner points in the same region, the calculation amount can be reduced, and the efficiency of video image processing can be improved.

[0194] In an embodiment, the step of processing the to-be-aligned video frame based on the mapping matrix between the to-be-aligned video frame and the center video frame to obtain each to-be-fused video frame includes:

[0195] Step 7: For each pixel coordinate in the to-be-fused video frame, the pixel coordinate is mapped based on the mapping matrix between the to-be-aligned video frame and the center video frame to obtain the corresponding pixel coordinate in the to-be-aligned video frame as a mapped pixel coordinate.

[0196] Step 8: The pixel value at the mapped pixel coordinate is determined as the pixel value at the pixel coordinate in the to-be-fused video frame.

[0197] For example, matrix (5) represents the mapping relationship of the center video frame mapped to the to-be-aligned video frame. After obtaining the mapping matrix between the to-be-aligned video frame and the center video frame, the pixel value of each pixel point in the to-be-fused video frame corresponding to the to-be-aligned video frame can be calculated based on formula (6), which is as follows:

[0198]

[0199] wherein Q * (x, y) represents the pixel value of the pixel point at pixel coordinate (x, y) in the to-be-fused video frame. pixel value of a pixel point at a pixel coordinate in the to-be-aligned video frame. pixel value of a pixel point at a pixel coordinate in the to-be-aligned video frame.

[0200] In addition, for each pixel coordinate in the to-be-fused video frame, if the to-be-aligned video frame does not contain a mapping pixel coordinate corresponding to the pixel coordinate in the to-be-fused video frame, the pixel value of the pixel point at the pixel coordinate in the to-be-fused video frame is set to 0. For example, the pixel coordinate (x, y) in the to-be-aligned video frame, x ranges from 0 to 499, and y ranges from 0 to 299. If the mapping pixel coordinate corresponding to the pixel coordinate (300, 299) in the to-be-fused video frame is (299, 302) based on the above formula (5), since the to-be-aligned video frame does not contain the pixel coordinate (299, 302), the pixel value of the pixel point with the pixel coordinate (300, 299) in the to-be-fused video frame can be set to 0.

[0201] In an embodiment, before step 8, the method further includes:

[0202] Step 9: If the mapping pixel coordinate is not an integer, determining a neighborhood range containing the mapping pixel coordinate in the to-be-aligned video frame.

[0203] Step 10: Based on the pixel values of the pixel points in the neighborhood range, calculating the pixel value at the mapping pixel coordinate according to a bilinear interpolation algorithm.

[0204] The mapping pixel coordinate calculated by the electronic device based on the above formula (6) can not be an integer, and therefore, the neighborhood range containing the mapping pixel coordinate in the to-be-aligned video frame can be determined to perform an interpolation operation. The interpolation operation can be implemented based on a sub-pixel coordinate bilinear interpolation algorithm.

[0205] For example, the mapping pixel coordinate corresponding to the pixel point with the pixel coordinate (2, 2) in the to-be-fused video frame is (2.5, 2.7), and the neighborhood range containing the pixel points with the pixel coordinates (2, 2), (2, 3), (3, 2), and (3, 3) in the to-be-aligned video frame can be determined. The pixel values of the pixel points with the pixel coordinates (2, 2), (2, 3), (3, 2), and (3, 3) are obtained, and the pixel value of the pixel point with the pixel coordinate (2, 2) in the to-be-fused video frame is calculated according to a bilinear interpolation algorithm.

[0206] In another implementation, after the neighborhood range containing the mapping pixel coordinate in the to-be-aligned video frame is determined, the average value of the pixel values of the pixel points in the neighborhood range can also be calculated to obtain the pixel value at the mapping pixel coordinate.

[0207] Based on the same inventive concept, the embodiment of the present application also provides an image processing device, which is described with reference to Figure 10 ,Figure 10 A structural diagram of an image processing device is provided for an embodiment of the present application, and the device comprises:

[0208] A to-be-aligned video frame buffer module 1001 is configured to sequentially acquire each to-be-aligned video frame and buffer the to-be-aligned video frame;

[0209] A mapping matrix calculation module 1002 is configured to, after buffering each to-be-aligned video frame, calculate a mapping matrix between the to-be-aligned video frame and a corresponding second number of adjacent video frames if the number of currently buffered to-be-aligned video frames is not less than a first number, wherein the second number of adjacent video frames comprises a center video frame in the first number of to-be-aligned video frames buffered last in the order of buffering and video frames between the center video frame and the to-be-aligned video frame;

[0210] A to-be-fused video frame acquisition module 1003 is configured to, for each to-be-aligned video frame other than the center video frame in the first number of to-be-aligned video frames buffered last in the order of buffering, process the to-be-aligned video frame based on a mapping matrix between the to-be-aligned video frame and the center video frame to obtain a to-be-fused video frame;

[0211] A target video frame acquisition module 1004 is configured to fuse each to-be-fused video frame and the center video frame to obtain a target video frame.

[0212] In some embodiments, the first number is an odd number greater than 1, and the center video frame is a to-be-aligned video frame located at a center position in the first number of to-be-aligned video frames in the order of buffering.

[0213] In some embodiments, the device further comprises:

[0214] A to-be-aligned video frame deletion module is configured to, after fusing each to-be-fused video frame and the center video frame to obtain a target video frame, delete a to-be-aligned video frame buffered first from the first number of to-be-aligned video frames buffered last in the order of buffering.

[0215] In some embodiments, the device further comprises:

[0216] A mapping matrix buffer module is configured to, after calculating the mapping matrix between the to-be-aligned video frame and the corresponding second number of adjacent video frames, buffer each mapping matrix calculated in the order of buffering between the corresponding second number of adjacent video frames.

[0217] The mapping matrix deleting module is configured to delete, after the target video frame is obtained by fusing each to-be-fused video frame and the center video frame, the second number of mapping matrices cached first according to the caching order.

[0218] In some embodiments, the to-be-fused video frame obtaining module 1003 comprises:

[0219] The inverse matrix calculating submodule is configured to, for each to-be-aligned video frame except the center video frame in the last cached first number of to-be-aligned video frames, calculate an inverse matrix of the mapping matrix corresponding to the to-be-aligned video frame if the mapping matrix corresponding to the to-be-aligned video frame represents a mapping relationship from the center video frame to the to-be-aligned video frame.

[0220] The to-be-fused video frame obtaining submodule is configured to process the to-be-aligned video frame based on the calculated inverse matrix to obtain a to-be-fused video frame.

[0221] In some embodiments, the target video frame obtaining module 1004 comprises:

[0222] The weight obtaining submodule is configured to determine, for each to-be-fused video frame and a plurality of pixel points at the same pixel coordinate in the center video frame, a weight of each of the plurality of pixel points based on the weight of each to-be-fused video frame and the center video frame; wherein the weight of each to-be-fused video frame and the center video frame conforms to a Gaussian distribution, the weight of the center video frame is the largest, and the distance between the to-be-aligned video frame corresponding to each to-be-fused video frame and the center video frame is negatively correlated with the weight of the to-be-fused video frame according to the caching order.

[0223] The pixel value obtaining submodule is configured to calculate a weighted sum of pixel values of the plurality of pixel points according to the weight of each of the plurality of pixel points to obtain a pixel value of a corresponding pixel coordinate in the target video frame.

[0224] In some embodiments, the weight obtaining submodule is specifically configured to:

[0225] For each pixel point at the same pixel coordinate in each to-be-fused video frame and the center video frame, if the pixel point belongs to the to-be-fused video frame and the difference between the pixel point and the pixel point at the same pixel coordinate in the center video frame is less than a preset threshold, the weight of the pixel point is determined as the weight of the to-be-fused video frame to which the pixel point belongs.

[0226] If the pixel point belongs to the to-be-fused video frame and the difference between the pixel point and the pixel point at the same pixel coordinate in the center video frame is not less than the preset threshold, the weight of the pixel point is determined as 0.

[0227] The weight of the pixel point with the same pixel coordinate in the center video frame is calculated based on the weight of the pixel point with the same pixel coordinate in each video frame to be fused.

[0228] In some embodiments, the video frame to be aligned buffer module 1001 comprises:

[0229] The video frame acquisition sub-module is configured to sequentially acquire each video frame from a video to be processed.

[0230] The video frame to be aligned buffer sub-module is configured to process the video frame based on a pre-trained noise reduction network model, obtain a video frame to be aligned, and buffer the video frame to be aligned.

[0231] In some embodiments, the mapping matrix calculation module 1002 comprises:

[0232] The aligned pixel point acquisition sub-module is configured to, for each adjacent video frame, determine, based on a block matching algorithm, a pixel point matched with a corner point in the video frame to be aligned from the adjacent video frame as an aligned pixel point.

[0233] The mapping matrix calculation sub-module is configured to calculate a mapping matrix between the video frame to be aligned and the adjacent video frame based on the pixel coordinates of the corner point in the video frame to be aligned and the pixel coordinates of the aligned pixel point according to a random sample consensus (RANSAC) algorithm.

[0234] In some embodiments, the video frame to be fused acquisition module 1003 comprises:

[0235] The mapping pixel coordinate acquisition sub-module is configured to, for each pixel coordinate in the video frame to be fused, perform mapping processing on the pixel coordinate based on the mapping matrix between the video frame to be aligned and the center video frame to obtain a corresponding pixel coordinate in the video frame to be aligned as a mapping pixel coordinate.

[0236] The video frame to be fused determination sub-module is configured to determine a pixel value at the mapping pixel coordinate in the video frame to be fused as a pixel value at the pixel coordinate in the video frame to be fused.

[0237] In some embodiments, the apparatus further comprises:

[0238] The neighborhood range determination module is configured to, before the pixel value at the mapping pixel coordinate is determined as the pixel value at the pixel coordinate in the video frame to be fused, determine a neighborhood range containing the mapping pixel coordinate in the video frame to be aligned if the mapping pixel coordinate is not an integer.

[0239] The pixel value calculation module is configured to calculate the pixel value at the mapping pixel coordinate based on pixel values of a plurality of pixel points in the neighborhood range according to a bilinear interpolation algorithm.

[0240] The embodiment of the present application further provides an electronic device, such as Figure 11 As shown in the figure, the electronic device comprises a processor 1101, a communication interface 1102, a memory 1103 and a communication bus 1104, wherein the processor 1101, the communication interface 1102 and the memory 1103 complete mutual communication through the communication bus 1104,

[0241] The memory 1103 is used for storing a computer program.

[0242] The processor 1101 is used for executing the program stored in the memory 1103, so as to realize the steps of the image processing method in any one of the above embodiments.

[0243] The communication bus mentioned in the above electronic device can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or only one type of bus.

[0244] The communication interface is used for communication between the above electronic device and other devices.

[0245] The memory can comprise a random access memory (RAM) and can also comprise a non-volatile memory (NVM), for example at least one disk memory. Optionally, the memory can also be at least one storage device located away from the aforementioned processor.

[0246] The processor mentioned above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0247] In yet another embodiment provided by the present application, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program. The computer program is executed by a processor to implement the steps of any of the image processing methods described above.

[0248] In yet another embodiment provided by the present application, a computer program product containing instructions which, when executed on a computer, cause the computer to perform any of the image processing methods described in the above embodiments.

[0249] In the above embodiments, the implementation can be wholly or partially in software, hardware, firmware, or any combination thereof. When implemented in software, the implementation can be in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed by a computer, the computer program instructions cause the implementation of the processes or functions described in the embodiments of the present application. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions can be stored in a computer readable storage medium or transferred from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transferred from one website site, computer, server, or data center to another website site, computer, server, or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) manner. The computer readable storage medium can be any available medium accessible by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available medium can be a magnetic medium (for example, floppy disk, hard disk, magnetic tape), optical medium (for example, DVD), or semiconductor medium (for example, solid state disk (SSD)), etc.

[0250] It should be noted that in this document, the terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of another identical element in the process, method, article or device including the element.

[0251] Various embodiments are described in related manner in the specification, and the same or similar parts among various embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, for the device, electronic device, computer-readable storage medium, and computer program product embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

[0252] The above descriptions are only the preferred embodiments of the present application, not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that, The method includes: Each video frame to be aligned is acquired sequentially and cached. After caching each video frame to be aligned, if the number of currently cached video frames to be aligned is not less than a first number, then calculate the mapping matrix between the video frame to be aligned and the corresponding second number of adjacent video frames; wherein, the second number of adjacent video frames include: the center video frame among the last cached first number of video frames to be aligned, according to the caching order, and the video frames between the center video frame and the video frame to be aligned. According to the order of caching, for each video frame to be aligned in the first number of video frames to be aligned in the last cache except for the center video frame, the video frame to be aligned is processed based on the mapping matrix between the video frame to be aligned and the center video frame to obtain the video frame to be fused. The target video frame is obtained by fusing each video frame to be fused and the central video frame.

2. The method according to claim 1, characterized in that, The first number is an odd number greater than 1, and according to the order of caching, the center video frame is the video frame to be aligned located at the center position among the first number of video frames to be aligned.

3. The method according to claim 1, characterized in that, After performing the fusion process on each video frame to be fused and the center video frame to obtain the target video frame, the method further includes: According to the order of caching, the earliest cached video frame to be aligned is deleted from the first number of cached video frames to be aligned last.

4. The method according to claim 1, characterized in that, After calculating the mapping matrix between the video frame to be aligned and the corresponding second number of adjacent video frames, the method further includes: The calculated mapping matrices are cached according to the order in which they are cached between the corresponding second number of adjacent video frames; After performing the fusion process on each video frame to be fused and the center video frame to obtain the target video frame, the method further includes: Delete the second number of the earliest cached mapping matrices according to the order of caching.

5. The method according to claim 1, characterized in that, The step involves processing each video frame to be aligned (excluding the center video frame) in the last cached first number of video frames to be aligned, according to the order of caching, based on the mapping matrix between the video frame to be aligned and the center video frame, to obtain the video frame to be fused, including: According to the order of caching, for each video frame to be aligned in the first number of video frames to be aligned in the last cached video frame other than the center video frame, if the pre-calculated mapping matrix corresponding to the video frame to be aligned represents the mapping relationship from the center video frame to the video frame to be aligned, then the inverse matrix of the mapping matrix corresponding to the video frame to be aligned is calculated. The video frame to be aligned is processed based on the calculated inverse matrix to obtain the video frame to be fused.

6. The method according to claim 1 or 2, characterized in that, The process of fusing each video frame to be fused and the center video frame to obtain the target video frame includes: For multiple pixels with the same pixel coordinates in each video frame to be merged and the central video frame, the weights of each of these pixels are determined based on the weights of each video frame to be merged and the central video frame. The weights of each video frame to be merged and the central video frame follow a Gaussian distribution, with the central video frame having the largest weight. Furthermore, according to the order of caching, the distance between the video frame to be aligned corresponding to each video frame to be merged and the central video frame is negatively correlated with the weight of that video frame to be merged. Based on the weights of each pixel, the weighted sum of the pixel values ​​of the multiple pixels is calculated to obtain the pixel value of the corresponding pixel coordinate in the target video frame.

7. The method according to claim 6, characterized in that, The step of determining the weight of each pixel with the same pixel coordinates in each video frame to be merged and the central video frame, based on the weights of each video frame to be merged and the central video frame, includes: For each pixel with the same pixel coordinates in each video frame to be merged and the central video frame, if the pixel belongs to the video frame to be merged and the difference between the pixel and the pixel with the same pixel coordinates in the central video frame is less than a preset threshold, then the weight of the pixel is determined to be the weight of the video frame to which the pixel belongs. If the pixel belongs to the video frame to be fused, and the difference between the pixel and the pixel with the same pixel coordinates in the center video frame is not less than the preset threshold, then the weight of the pixel is determined to be 0. The weight of the pixel with the same pixel coordinate in the central video frame is calculated based on the weight of the pixel with the same pixel coordinate in each video frame to be fused.

8. The method according to claim 1, characterized in that, The step of sequentially acquiring each video frame to be aligned and caching it includes: Each video frame is sequentially obtained from the video to be processed; Based on a pre-trained noise reduction network model, the video frame is processed to obtain the video frame to be aligned, and then cached.

9. The method according to claim 1, characterized in that, The step of calculating the mapping matrix between the video frame to be aligned and the corresponding second number of adjacent video frames includes: For each adjacent video frame, based on the block matching algorithm, the pixel point that matches the corner point in the video frame to be aligned is determined from the adjacent video frame and used as the alignment pixel point; Based on the pixel coordinates of the corner points in the video frame to be aligned and the pixel coordinates of the aligned pixels, the mapping matrix between the video frame to be aligned and the adjacent video frame is calculated according to the Random Sample Consensus (RANSAC) algorithm.

10. The method according to claim 1, characterized in that, The process of processing the video frame to be aligned based on the mapping matrix between the video frame to be aligned and the center video frame to obtain the video frame to be fused includes: For each pixel coordinate in the video frame to be fused, the pixel coordinate is mapped based on the mapping matrix between the video frame to be aligned and the center video frame to obtain the corresponding pixel coordinate in the video frame to be aligned, which is used as the mapped pixel coordinate. The pixel value at the mapped pixel coordinates is determined as the pixel value at the same pixel coordinates in the video frame to be fused.

11. The method according to claim 10, characterized in that, Before determining the pixel value at the mapped pixel coordinates as the pixel value at the same pixel coordinates in the video frame to be fused, the method further includes: If the mapped pixel coordinates are not integers, then the neighborhood range of the mapped pixel coordinates contained in the video frame to be aligned is determined. Based on the pixel values ​​of multiple pixels within the neighborhood, the pixel value at the mapped pixel coordinates is calculated using a bilinear interpolation algorithm.

12. An image processing apparatus, characterized in that, The device includes: The video frame caching module is used to sequentially obtain each video frame to be aligned and cache it. The mapping matrix calculation module is used to calculate the mapping matrix between each video frame to be aligned and the corresponding second number of adjacent video frames after caching each video frame to be aligned, if the number of currently cached video frames to be aligned is not less than a first number; wherein the second number of adjacent video frames includes: the center video frame among the last cached first number of video frames to be aligned, according to the caching order, and the video frames between the center video frame and the video frame to be aligned. The video frame to be merged acquisition module is used to process each video frame to be aligned, except for the center video frame, in the first number of video frames to be aligned in the last cached order, based on the mapping matrix between the video frame to be aligned and the center video frame, to obtain the video frame to be merged. The target video frame acquisition module is used to perform fusion processing on each video frame to be fused and the central video frame to obtain the target video frame.

13. The apparatus according to claim 12, characterized in that, The first number is an odd number greater than 1, and according to the order of caching, the center video frame is the video frame to be aligned located at the center position among the first number of video frames to be aligned.

14. The apparatus according to claim 12, characterized in that, The device further includes: The unaligned video frame deletion module is used to delete the earliest cached unaligned video frame from the last cached first number of unaligned video frames after the fusion processing of each unaligned video frame and the center video frame is performed to obtain the target video frame.

15. The apparatus according to claim 12, characterized in that, The device further includes: The mapping matrix caching module is used to cache the calculated mapping matrices according to the caching order among the corresponding second number of adjacent video frames after calculating the mapping matrix between the video frame to be aligned and the corresponding second number of adjacent video frames. The mapping matrix deletion module is used to delete the second number of the earliest cached mapping matrices according to the cache order after the fusion processing of each video frame to be fused and the center video frame is performed to obtain the target video frame.

16. The apparatus according to claim 12, characterized in that, The video frame acquisition module to be fused includes: The inverse matrix calculation submodule is used to calculate the inverse matrix of the mapping matrix corresponding to the video frame to be aligned for each video frame to be aligned in the first number of video frames to be aligned in the last cached video frames, excluding the center video frame, according to the order of caching. If the mapping matrix corresponding to the video frame to be aligned is pre-calculated to represent the mapping relationship from the center video frame to the video frame to be aligned, then the inverse matrix of the mapping matrix corresponding to the video frame to be aligned is calculated. The submodule for obtaining video frames to be merged is used to process the video frame to be aligned based on the calculated inverse matrix to obtain the video frame to be merged.

17. The apparatus according to claim 12 or 13, characterized in that, The target video frame acquisition module includes: The weight acquisition submodule is used to determine the weight of each pixel with the same pixel coordinates in each video frame to be merged and the center video frame, based on the weights of each video frame to be merged and the center video frame. The weights of each video frame to be merged and the center video frame follow a Gaussian distribution, with the center video frame having the largest weight. In addition, according to the order of caching, the distance between the video frame to be aligned corresponding to each video frame to be merged and the center video frame is negatively correlated with the weight of the video frame to be merged. The pixel value acquisition submodule is used to calculate the weighted sum of the pixel values ​​of the multiple pixels according to their respective weights, so as to obtain the pixel value of the corresponding pixel coordinate in the target video frame.

18. The apparatus according to claim 17, characterized in that, The weight acquisition submodule is specifically used for: For each pixel with the same pixel coordinates in each video frame to be merged and the central video frame, if the pixel belongs to the video frame to be merged and the difference between the pixel and the pixel with the same pixel coordinates in the central video frame is less than a preset threshold, then the weight of the pixel is determined to be the weight of the video frame to which the pixel belongs. If the pixel belongs to the video frame to be fused, and the difference between the pixel and the pixel with the same pixel coordinates in the center video frame is not less than the preset threshold, then the weight of the pixel is determined to be 0. The weight of the pixel with the same pixel coordinate in the central video frame is calculated based on the weight of the pixel with the same pixel coordinate in each video frame to be fused.

19. The apparatus according to claim 12, characterized in that, The video frame buffer module to be aligned includes: The video frame acquisition submodule is used to sequentially acquire each video frame from the video to be processed; The video frame caching submodule is used to process the video frame based on a pre-trained noise reduction network model to obtain the video frame to be aligned and cache it.

20. The apparatus according to claim 12, characterized in that, The mapping matrix calculation module includes: The alignment pixel acquisition submodule is used to determine, based on a block matching algorithm, the pixel that matches the corner point in the video frame to be aligned from the adjacent video frame for each adjacent video frame, and use it as the alignment pixel. The mapping matrix calculation submodule is used to calculate the mapping matrix between the video frame to be aligned and the adjacent video frame based on the pixel coordinates of the corner point in the video frame to be aligned and the pixel coordinates of the aligned pixel point, according to the Random Sample Consensus (RANSAC) algorithm.

21. The apparatus according to claim 12, characterized in that, The video frame acquisition module to be fused includes: The submodule for obtaining mapped pixel coordinates is used to map each pixel coordinate in the video frame to be fused based on the mapping matrix between the video frame to be aligned and the center video frame, so as to obtain the corresponding pixel coordinates in the video frame to be aligned, which are then used as mapped pixel coordinates. The video frame to be merged determination submodule is used to determine the pixel value at the mapped pixel coordinates as the pixel value at the same pixel coordinates in the video frame to be merged.

22. The apparatus according to claim 21, characterized in that, The device further includes: The neighborhood range determination module is used to determine the neighborhood range of the video frame to be aligned containing the mapped pixel coordinates before determining the pixel value at the mapped pixel coordinates as the pixel value at the pixel coordinates in the video frame to be merged. The pixel value calculation module is used to calculate the pixel value at the mapped pixel coordinates based on the pixel values ​​of multiple pixels within the neighborhood range, using a bilinear interpolation algorithm.

23. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the steps of the method described in any one of claims 1-11.

24. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 1-11.

Images