Multi-frame fusion method, device and equipment

Abstract

The invention provides a multi-frame fusion method, device and equipment, and relates to the technical field of image processing, and the method comprises the steps: dividing frames included in a video into N sets according to the light incoming amount; determining an optimal frame in the set according to the image information entropy and the image momentum of each frame in the set; selecting a plurality of fusion frames from the optimal frames according to the motion field of the optimal frames; dividing a fusion frame into a plurality of image blocks, and distributing a fusion weight for each fusion frame for the image block at each position; and according to the fusion weight, fusing the blocks of the plurality of fusion frames at the same position, and forming an image based on the fused blocks. Therefore, multiple frames of images in the video can be fused into one image, the information amount of the image is increased, and the problem that image information is incomplete due to factors such as illumination change or portrait shaking is solved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a multi-frame fusion method, device and equipment. Background Art

[0002] When taking photos, the picture information is often incomplete due to overexposure, insufficient light, portrait movement, etc.

[0003] At present, incomplete picture information greatly affects the perception of the image. In addition, in specific image application scenarios such as face recognition, incomplete picture information will also affect the accuracy of recognition. Summary of the invention

[0004] In view of this, the present application provides a multi-frame fusion method, device and equipment, aiming to solve the problem of incomplete picture information caused by problems such as lighting or shaking.

[0005] In a first aspect, a multi-frame fusion method includes:

[0006] Dividing the frames included in the video into N sets according to the amount of incoming light;

[0007] Determining an optimal frame in the set according to image information entropy and image momentum of each frame in the set;

[0008] According to the motion field of the optimal frame, multiple fusion frames are selected from the optimal frame;

[0009] The fused frame is divided into a plurality of blocks, and a fusion weight is assigned to each of the fused frames for the blocks at each position; according to the fusion weight, the blocks at the same position of the plurality of fused frames are fused, and an image is formed based on the fused blocks.

[0010] Optionally, dividing the frames included in the video into N sets according to the amount of incoming light includes:

[0011] Calculate the average grayscale value of each frame after converting each frame into a grayscale frame;

[0012] The grayscale values ​​in the range of 0 to 255 are divided into N grayscale value segments in turn, and the frames included in the video are allocated to corresponding grayscale value segments according to the average grayscale of each frame. The corresponding frames in each grayscale value segment form a set, and each grayscale value segment corresponds to at least one frame.

[0013] Optionally, determining the optimal frame in the set according to the image information entropy and image momentum of each frame in the set includes:

[0014] Calculating the image information entropy and image momentum of each frame in each set of the N sets;

[0015] The image information entropy of each frame is multiplied by the value of the first weight coefficient, and the result is added to the image momentum of the frame multiplied by the value of the second weight coefficient to obtain a preferred score of the frame;

[0016] The frame with the largest preference score in the set is taken as the optimal frame.

[0017] Optionally, the method for calculating the image momentum includes:

[0018] Calculate the motion vector of each pixel in each frame;

[0019] Calculating the image momentum of the entire image of the frame according to the motion vector of each pixel in each frame;

[0020] The motion vector calculation formula of each pixel is:

[0021] Where v is the motion vector of the pixel; I is the brightness of the pixel at time t, expressed as (x, y, t), x is the horizontal coordinate position of the pixel, and y is the vertical coordinate position of the pixel; I t represents the partial derivative of I with respect to t;

[0022] The image momentum calculation formula for the entire image is:

[0023] Where M(A) is the total momentum of frame A, v z Represents the motion vector of the z-th pixel.

[0024] Optionally, selecting a plurality of fused frames from the optimal frame according to the motion field of the optimal frame includes:

[0025] On the two-dimensional plane where the image of the optimal frame is located, a two-dimensional motion field is obtained according to the motion directions x and y, and the motion field between the two optimal frames is calculated;

[0026] Predicting the minimum motion field of the K+1th optimal frame according to the motion fields of the first K optimal frames, where K is a positive integer greater than one and less than the total number of the optimal frames;

[0027] A plurality of optimal frames with the smallest values ​​are selected as fused frames from among the minimum motion fields of the predicted plurality of optimal frames.

[0028] Optionally, the fusion weight is:

[0029]

[0030] Where j is the coefficient; D c b(w) is the block difference with frequency w after Fourier transformation for the block position c in frame b, and e is a constant; is the noise variance.

[0031] Optionally, the method further includes:

[0032] When the target face is facing the camera, the aperture of the camera is quickly adjusted to obtain a video with changes in brightness and darkness.

[0033] Optionally, the image is an image including a target person's face; the method further includes:

[0034] The image is matched with the portraits in the certificate library through the face recognition model to perform face recognition.

[0035] In a second aspect, the present application provides a multi-frame fusion device, comprising:

[0036] An allocating unit, configured to divide the frames included in the video into N sets according to the amount of incoming light;

[0037] A first calculation unit, configured to determine an optimal frame in the set according to image information entropy and image momentum of each frame in the set;

[0038] A second calculation unit, configured to select a plurality of fused frames from the optimal frame according to the motion field of the optimal frame;

[0039] The fusion unit is used to divide the fused frame into a plurality of blocks, and assign a fusion weight to each of the fused frames for the blocks at each position; according to the fusion weight, fuse the blocks at the same position of the plurality of fused frames, and form an image based on the fused blocks.

[0040] In a third aspect, the present application provides an electronic device, comprising: a memory and a processor, the memory being used to store instructions or codes, and the processor being used to execute the instructions or codes so that the device executes any one of the multi-frame fusion methods described above.

[0041] The present application provides a multi-frame fusion method, including: dividing the frames included in the video into N sets according to the amount of incoming light; determining the optimal frame in the set according to the image information entropy and image momentum of each frame in the set; selecting multiple fused frames from the optimal frame according to the motion field of the optimal frame; dividing the fused frame into multiple blocks, and assigning a fusion weight to each fused frame for the block at each position; according to the fusion weight, fusing the blocks of multiple fused frames at the same position, and forming an image based on the fused blocks. In this way, multiple frames of images in the video can be fused into one image, increasing the amount of information in the picture to compensate for the problem of incomplete image information caused by factors such as changes in lighting or shaking of the portrait. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in this embodiment or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 A schematic diagram of a multi-frame fusion method provided in an embodiment of the present application;

[0044] Figure 2 A schematic diagram of the change of image information entropy from dark to bright provided in an embodiment of the present application;

[0045] Figure 3 A schematic diagram of the distribution of pixels of each optimal frame in the grayscale value range of 0 to 255 in the nine optimal frames provided in an embodiment of the present application;

[0046] Figure 4 A schematic diagram of an image formed by fusing three fused frames provided in an embodiment of the present application;

[0047] Figure 5 A schematic diagram of a process of performing face recognition by multi-frame fusion provided in an embodiment of the present application;

[0048] Figure 6 A schematic diagram of the structure of a multi-frame fusion device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] When taking photos, the image information is often incomplete due to overexposure, insufficient light, motion of the portrait, etc. Especially in the increasingly widely used field of portrait recognition security, such as access control systems, border inspection systems, etc. Face recognition technology can achieve fast and efficient identity authentication, improve security and convenience. When capturing a face image, the image information may be incomplete due to overexposure, insufficient light, motion of the portrait, etc., resulting in a lower success rate when matching faces. In order to increase the success rate of face recognition in special scenes, it is very necessary to perform image enhancement processing on the face image of the scene.

[0050] Therefore, the present application provides a multi-frame fusion method, including: dividing the frames included in the video into N sets according to the amount of incoming light; determining the optimal frame in the set according to the image information entropy and image momentum of each frame in the set; selecting multiple fused frames from the optimal frame according to the motion field of the optimal frame; dividing the fused frame into multiple blocks, and assigning a fusion weight to each fused frame for the block at each position; according to the fusion weight, fusing the blocks of multiple fused frames at the same position, and forming an image based on the fused blocks. In this way, multiple frames of images in the video can be fused into one image, the amount of information in the picture can be increased, image enhancement processing can be achieved, the problem of imperfect image information caused by factors such as lighting changes or portrait shaking can be compensated, and the success rate of image matching in the field of face recognition can be improved.

[0051] In order to be able to understand the features and technical contents of the embodiments of the present disclosure in more detail, the implementation of the embodiments of the present disclosure is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present disclosure. In the following technical description, for the convenience of explanation, a full understanding of the disclosed embodiments is provided through multiple details. However, one or more embodiments can still be implemented without these details. In other cases, to simplify the drawings, well-known structures and devices can be simplified for display.

[0052] The terms "first", "second", etc. in the specification and claims of the embodiments of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged where appropriate, so that the embodiments of the embodiments of the present disclosure described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions.

[0053] Unless otherwise stated, the term "plurality" means two or more.

[0054] In the embodiment of the present disclosure, the character " / " indicates that the preceding and following objects are in an "or" relationship. For example, A / B indicates: A or B.

[0055] The term "and / or" is a description of the association relationship between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or, A and B.

[0056] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0057] See also Figure 1 , Figure 1 A flowchart of a multi-frame fusion method provided in an embodiment of the present application, a multi-frame fusion method comprising:

[0058] S101. Divide the frames included in the video into N sets according to the amount of incoming light.

[0059] Exemplarily, the sets to which the frames in the video belong are divided according to the grayscale values ​​of the images.

[0060] The video may be a video including a human face and may be used for face recognition.

[0061] S102: Determine the optimal frame in the set according to the image information entropy and image momentum of each frame in the set.

[0062] Image information entropy, also known as image entropy, represents the average number of bits of the image grayscale set, with the unit of bit / pixel, which describes the average amount of information in the image.

[0063] The above-mentioned image momentum is determined based on the motion vector of each pixel in each frame image to determine the image momentum of the entire image.

[0064] Optionally, the above-mentioned determination of the optimal frame in the set may be that each set includes at least one frame, and an optimal frame is selected from each set, and N optimal frames can be determined for N sets.

[0065] S103 . Select multiple fused frames from the optimal frame according to the motion field of the optimal frame.

[0066] The above motion field represents the instantaneous motion velocity vector information of each pixel. Specifically, on an image plane, the motion of an object is often reflected by the different grayscale distributions of different images in the image sequence. Therefore, the motion field in space is transferred to the image and represented as an optical flow field. The optical flow field is a two-dimensional vector field that reflects the trend of grayscale changes at each point on the image. It can be regarded as the instantaneous velocity field generated by the movement of grayscale pixels on the image plane.

[0067] S104, dividing the fused frame into a plurality of blocks, and assigning a fusion weight to each of the fused frames for the blocks at each position; according to the fusion weight, fusing the blocks at the same position of the plurality of fused frames, and forming an image based on the fused blocks.

[0068] A conventional image fusion algorithm may be used to form a plurality of fused frames into one image according to the fusion weights of the fused frames of each position block.

[0069] According to the above steps S101-S104, the present application fuses multiple frames of images in the video into one image, increasing the amount of information in the image to compensate for the problem of incomplete image information caused by factors such as lighting changes or portrait shaking.

[0070] In the embodiment of the present application, the above Figure 1 There are multiple possible implementations of step S101, which are described below. It should be noted that the implementations described below are only exemplary and do not represent all implementations of the embodiments of the present application.

[0071] In a possible implementation, frames included in the video are divided into N sets according to the amount of incoming light, including:

[0072] Calculate the average grayscale value of each frame after converting each frame into a grayscale frame;

[0073] The grayscale values ​​in the range of 0 to 255 are divided into N grayscale value segments in turn, and the frames included in the video are allocated to corresponding grayscale value segments according to the average grayscale of each frame. The corresponding frames in each grayscale value segment form a set, and each grayscale value segment corresponds to at least one frame.

[0074] The grayscale value ranges from 0 to 255.

[0075] Optionally, the video is a video that changes from dark to light.

[0076] In a specific implementation manner, each frame may be allocated to a corresponding set using the following formula.

[0077]

[0078] Among them, FrameSet g is the gth set; Average Frame is the average grayscale value of the Frame frame; N is the total number of sets to be allocated.

[0079] In the embodiment of the present application, the above Figure 1There are multiple possible implementations of step S102, which are described below. It should be noted that the implementations described below are only exemplary and do not represent all implementations of the embodiments of the present application.

[0080] Determining the optimal frame in the set according to the image information entropy and the image momentum of each frame in the set includes:

[0081] Step A1: Calculate the image information entropy and image momentum of each frame in each set of the N sets.

[0082] The specific formula for calculating the image information entropy of frame A can be: H(A) = -Σ(p(i)*log 2 (p(i)))

[0083] Where H(A) is the image information entropy of frame A; p(i) is the probability of pixel value i appearing in the frame. Figure 2 The diagram shows the change of image information entropy from dark to bright. If the image is too dark or too bright, p(i) will be concentrated in a smaller area, reducing the image information entropy.

[0084] The specific method for calculating the momentum of the A frame image can be:

[0085] First, the motion vector of each pixel in each frame is calculated.

[0086] Specifically, the motion vector calculation formula of each pixel is: T v=-I t

[0087] Where v is the motion vector of the pixel; I is the brightness of the pixel at time t, expressed as (x, y, t), x is the horizontal coordinate position of the pixel, and y is the vertical coordinate position of the pixel; I t represents the partial derivative of I with respect to t;

[0088] Then, according to the motion vector of each pixel point in each frame, the image momentum of the entire image of the frame is calculated.

[0089] Specifically, the image momentum calculation formula of the entire image is:

[0090] Where M(A) is the total momentum of frame A, v z Represents the motion vector of the z-th pixel.

[0091] Step A2: multiply the image information entropy of each frame by the value of the first weight coefficient, and add the result to the image momentum of the frame multiplied by the value of the second weight coefficient to obtain the optimal score of the frame. The specific formula may be:

[0092] T(A)=r 1 H(A)+r 2 M(A)

[0093] T(A) is the optimal score of frame A, H(A) is the image information entropy of frame A, M(A) is the total momentum value of frame A, r 1 is the first weight coefficient, r 2 is the second weight coefficient.

[0094] Step A3: taking the frame with the largest preference score in the set as the optimal frame.

[0095] According to the preferred score ranking of all frames in the set, the frame corresponding to the maximum value is selected as the optimal frame of the set. For example, see Figure 3 A schematic diagram of the distribution of pixels of each of the nine optimal frames in the grayscale value range of 0 to 255 is shown, wherein the abscissa is the grayscale value and the ordinate is the number of pixels.

[0096] In the embodiment of the present application, the above Figure 1 There are multiple possible implementations of step S103, which are described below. It should be noted that the implementations described below are only exemplary and do not represent all implementations of the embodiments of the present application.

[0097] The step of selecting a plurality of fused frames from the optimal frame according to the motion field of the optimal frame comprises:

[0098] Step B1: On the two-dimensional plane where the image of the optimal frame is located, a two-dimensional motion field is obtained according to the motion directions x and y, and the motion field between the two optimal frames is calculated.

[0099] The specific calculation method includes first calculating the motion field of each pixel point between two optimal frames, and then adding the motion fields of each pixel point to obtain the motion field between the two optimal frames. The calculation formula for calculating the motion field of each pixel point between two optimal frames is:

[0100]

[0101] In the above formula, ΔI t (x,y) is the brightness change of the pixel at coordinate (x,y), is the gradient value of the pixel at coordinate (x, y), is the motion field of the pixel at coordinate (x, y).

[0102] Step B2: predicting the minimum motion field of the K+1th optimal frame based on the motion fields of the first K optimal frames, where K is a positive integer greater than one and less than the total number of the optimal frames.

[0103] It can be understood that the first K optimal frames are in the time order of the video, the Kth optimal frame among multiple optimal frames, and the K+1th optimal frame are the same.

[0104] The specific prediction formula can be:

[0105] 1-Pr[V min <=V K ] K >=δ

[0106] In the above formula, V min To predict the K+1th optimal frame with the smallest motion field, Pr[V min <=V K ] indicates V min is less than the average probability of the previous K optimal frame modulus values, and δ is the set prediction success threshold.

[0107] Exemplarily, setting K to 3, the calculation method of the first three optimal frame modulus values ​​is to calculate the square of the motion field between the first and second optimal frames, plus the sum of the squares of the motion field between the second and third optimal frames; then taking the square root of the sum to obtain the first three optimal frame modulus values.

[0108] Step B3: Select multiple optimal frames with the smallest values ​​from the minimum motion fields of the predicted multiple optimal frames as fused frames.

[0109] Thus, through the above steps B1-B3, the optimized frame with the smallest motion field is selected from multiple optimized frames in order to minimize the blurring problem caused by shaking.

[0110] In the embodiment of the present application, the above Figure 1 The specific calculation formula of the fusion weight in step S104 may be:

[0111]

[0112] Where j is the coefficient; D c b(w) is the block difference with frequency w after Fourier transformation for the block position c in frame b, and e is a constant; is the noise variance.

[0113] In a specific implementation of step S104, first, the fused frame is divided into multiple tiles, and a fusion weight is assigned to each fused frame for each tile at each position. Exemplarily, multiple fused frames are divided in the same way, each fused frame includes multiple tiles, and then a fusion weight is assigned to the tiles at the same position of the multiple fused frames. Subsequently, the multiple tiles at the position are fused using existing image fusion technology in combination with the corresponding fusion weights to obtain a fused image at the position. Then, an image is formed based on the fused tiles. For details, see Figure 4 The schematic diagram of the three fused frames fused to form an image is shown, where the three fused frames are from Figure 3 The fused frame extracted from the nine optimized frames shown in Figure 4 The fusion pre-frame one, the fusion pre-frame two and the fusion pre-frame three are fused to obtain the fusion post-frame (image).

[0114] In summary, the present application performs segmentation based on grayscale values ​​to obtain a set of segments corresponding to different grayscale values, and extracts the optimal frame from each set by combining image information entropy and image momentum to avoid the image being too bright or too dark, and then extracts several fused frames from multiple optimal frames based on the minimum motion field as a benchmark to reduce the blur caused by the shaking of the target object in the image. In this way, several fused frames are fused to obtain the final image, thereby fusing multiple frames in the video into one image, increasing the amount of information in the picture, and compensating for the problem of incomplete image information caused by factors such as changes in lighting or shaking of the portrait.

[0115] Based on the above Figure 1 The steps S101-S104 shown in the figure can be further applied to the field of face recognition. Figure 5 A schematic diagram of the process of multi-frame fusion for face authentication is shown.

[0116] Optionally, before step S101, the method further includes acquiring a video including the target face. Specifically, the method for acquiring a video including the target face includes: when the target face is facing the camera, quickly adjusting the aperture of the camera to acquire a video with light and dark changes.

[0117] The above-mentioned target face aligning with the camera can keep the target face at the position corresponding to the camera.

[0118] The above-mentioned video with light-dark changes can be a video from dark to bright or from bright to dark. In this way, the multiple frames included in the video can be directly divided into multiple sets in sequence.

[0119] Optionally, after the above step S104, the method further includes: matching the image with a portrait in a certificate library by a face recognition model to perform face recognition authentication.

[0120] Specifically, similarity is used for facial feature matching, and the specific formula can be:

[0121]

[0122] In the above formula, dist is the similarity calculated by feature information. face is the feature of the image obtained by the above fusion, Feature lib Characteristics of the certificate library.

[0123] For further information, see Figure 5 The application flow diagram of a multi-frame fusion for face recognition is shown in FIG. 1 . The present application provides a method for multi-frame fusion for face recognition, including:

[0124] S501: When the target face is facing the camera, quickly adjust the aperture of the camera to obtain a video with changes in brightness.

[0125] S502, dividing the frames included in the video into N sets according to the amount of incoming light;

[0126] S503, determining the optimal frame in the set according to the image information entropy and image momentum of each frame in the set;

[0127] S504, selecting multiple fused frames from the optimal frame according to the motion field of the optimal frame;

[0128] S505, dividing the fused frame into a plurality of blocks, and assigning a fusion weight to each of the fused frames for the blocks at each position; according to the fusion weight, fusing the blocks at the same position of the plurality of fused frames, and forming an image based on the fused blocks.

[0129] S506: Match the image with the portrait in the certificate library through a face recognition model to perform face recognition.

[0130] Based on the above steps S501-S506, multiple frames of images in the video are fused into one image to increase the amount of information in the image, implement image enhancement processing, compensate for the problem of incomplete image information caused by factors such as lighting changes or portrait shaking, and improve the success rate of image matching in the field of face recognition.

[0131] The above are some specific implementations of a multi-frame fusion method provided by the embodiment of the present application. Based on this, the present application also provides a corresponding device. The device provided by the embodiment of the present application will be introduced from the perspective of functional modularization.

[0132] See also Figure 6 A structural schematic diagram of a multi-frame fusion device is shown, and a multi-frame fusion device includes:

[0133] An allocating unit 601 is used to divide the frames included in the video into N sets according to the amount of incoming light;

[0134] A first calculation unit 602, configured to determine an optimal frame in the set according to image information entropy and image momentum of each frame in the set;

[0135] A second calculation unit 603 is used to select a plurality of fused frames from the optimal frame according to the motion field of the optimal frame;

[0136] The fusion unit 604 is used to divide the fused frame into multiple blocks, and assign a fusion weight to each fused frame for the block at each position; according to the fusion weight, fuse the blocks at the same position of multiple fused frames, and form an image based on the fused blocks.

[0137] Based on the above device, the allocation unit 601 performs segmentation based on the amount of incoming light to obtain a set of corresponding segments of different grayscale values. The first calculation unit 602 extracts the optimal frame from each set by combining image information entropy and image momentum to avoid the image being too bright or too dark. Then, the second calculation unit 603 extracts several fused frames from multiple optimal frames based on the minimum motion field as a benchmark to reduce the blur caused by the shaking of the target object in the image. Finally, the fusion unit 604 fuses several fused frames to obtain the final image, thereby fusing multiple frames of images in the video into one image, increasing the amount of information in the picture, and compensating for the problem of incomplete image information caused by factors such as changes in illumination or shaking of the portrait.

[0138] In a possible implementation, the allocation unit 601 is specifically used to calculate the average grayscale value of each frame after each frame is converted into a grayscale frame; divide the grayscale values ​​in the range of 0 to 255 into N grayscale value segments in turn, and allocate the frames included in the video to the corresponding grayscale value segments according to the average grayscale of each frame, and the corresponding frames in each grayscale value segment form a set, and each grayscale value segment corresponds to at least one frame.

[0139] In one possible implementation, the first calculation unit 602 is specifically used to calculate the image information entropy and image momentum of each frame in each set of the N sets; multiply the image information entropy of each frame by the value of the first weight coefficient, and add the result to the image momentum of the frame multiplied by the value of the second weight coefficient to obtain the preferred score of the frame; and take the frame with the largest preferred score in the set as the optimal frame.

[0140] In a possible implementation, the first calculation unit 602 is further configured to calculate a motion vector of each pixel in each frame; and calculate an image momentum of the entire image of the frame according to the motion vector of each pixel in each frame;

[0141] The motion vector calculation formula of each pixel is:

[0142] Where v is the motion vector of the pixel; I is the brightness of the pixel at time t, expressed as (x, y, t), x is the horizontal coordinate position of the pixel, and y is the vertical coordinate position of the pixel; I t represents the partial derivative of I with respect to t;

[0143] The image momentum calculation formula for the entire image is:

[0144] Where M(A) is the total momentum of frame A, v z Represents the motion vector of the z-th pixel.

[0145] In a possible implementation, the second calculation unit 603 is specifically used to obtain a two-dimensional motion field according to the motion directions x and y on the two-dimensional plane where the image of the optimal frame is located, and calculate the motion field between two optimal frames; predict the minimum motion field of the K+1th optimal frame according to the motion fields of the first K optimal frames, where K is a positive integer greater than one and less than the total number of the optimal frames; and select multiple optimal frames with the smallest values ​​from the predicted minimum motion fields of multiple optimal frames as fused frames.

[0146] In a possible implementation, the fusion weight is:

[0147]

[0148] Where j is the coefficient; D c b(w) is the block difference with frequency w after Fourier transformation for the block position c in frame b, and e is a constant; is the noise variance.

[0149] In a possible implementation, the device further includes: an acquisition unit, configured to quickly adjust the aperture of the camera when the target face is aligned with the camera, so as to acquire a video with changes in brightness.

[0150] In a possible implementation, the image is an image including a target face; the device further includes: a recognition unit, configured to match the image with a portrait in a certificate library through a face recognition model to perform face recognition.

[0151] The embodiments of the present application also provide corresponding devices and computer storage media for implementing the solutions provided by the embodiments of the present application.

[0152] The device includes a memory and a processor, the memory is used to store instructions or codes, and the processor is used to execute the instructions or codes so that the device executes a multi-frame fusion method described in any embodiment of the present application.

[0153] The computer storage medium stores codes, and when the codes are executed, a device executing the codes implements a multi-frame fusion method described in any embodiment of the present application.

[0154] The "first" and "second" in the names such as "first" and "second" (if any) mentioned in the embodiments of the present application are only used as name identifiers and do not represent the first or second in order.

[0155] Through the description of the above implementation methods, it can be known that those skilled in the art can clearly understand that all or part of the steps in the above-mentioned embodiment method can be implemented by means of software plus a general hardware platform. Based on such an understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a storage medium, such as a read-only memory (ROM) / RAM, a magnetic disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network communication device such as a router) to execute the methods described in each embodiment of the present application or some parts of the embodiments.

[0156] Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative work.

[0157] The above description is merely an exemplary embodiment of the present application and is not intended to limit the protection scope of the present application.

Claims

1. A multi-frame fusion method, characterized in that: include: Dividing the frames included in the video into N sets according to the amount of incoming light; Determining an optimal frame in the set according to image information entropy and image momentum of each frame in the set; According to the motion field of the optimal frame, multiple fusion frames are selected from the optimal frame; The fused frame is divided into a plurality of blocks, and a fusion weight is assigned to each of the fused frames for the blocks at each position; according to the fusion weight, the blocks at the same position of the plurality of fused frames are fused, and an image is formed based on the fused blocks.

2. The method according to claim 1, characterized in that The frames included in the video are divided into N sets according to the amount of incoming light, including: Calculate the average grayscale value of each frame after converting each frame into a grayscale frame; The grayscale values ​​in the range of 0 to 255 are divided into N grayscale value segments in turn, and the frames included in the video are allocated to corresponding grayscale value segments according to the average grayscale of each frame. The corresponding frames in each grayscale value segment form a set, and each grayscale value segment corresponds to at least one frame.

3. The method according to claim 1, characterized in that Determining the optimal frame in the set according to the image information entropy and image momentum of each frame in the set includes: Calculating the image information entropy and image momentum of each frame in each set of the N sets; The image information entropy of each frame is multiplied by the value of the first weight coefficient, and the result is added to the image momentum of the frame multiplied by the value of the second weight coefficient to obtain a preferred score of the frame; The frame with the largest preference score in the set is taken as the optimal frame.

4. The method according to claim 3, characterized in that The method for calculating the image momentum includes: Calculate the motion vector of each pixel in each frame; Calculating the image momentum of the entire image of the frame according to the motion vector of each pixel in each frame; The motion vector calculation formula of each pixel point is: T v=-I t Where v is the motion vector of the pixel; I is the brightness of the pixel at time t, expressed as (x, y, t), x is the horizontal coordinate position of the pixel, and y is the vertical coordinate position of the pixel; I t represents the partial derivative of I with respect to t; The image momentum calculation formula for the entire image is: Where M(A) is the total momentum of frame A, v z Represents the motion vector of the z-th pixel.

5. The method according to claim 1, characterized in that The step of selecting a plurality of fused frames from the optimal frame according to the motion field of the optimal frame comprises: On the two-dimensional plane where the image of the optimal frame is located, a two-dimensional motion field is obtained according to the motion directions x and y, and the motion field between the two optimal frames is calculated; Predicting the minimum motion field of the K+1th optimal frame according to the motion fields of the first K optimal frames, where K is a positive integer greater than one and less than the total number of the optimal frames; A plurality of optimal frames with the smallest values ​​are selected as fused frames from among the minimum motion fields of the predicted plurality of optimal frames.

6. The method according to claim 1, characterized in that The fusion weight is: Where j is the coefficient; D c b(w) is the block difference with frequency w after Fourier transformation for the block position c in frame b, and e is a constant; is the noise variance.

7. The method according to claim 1, characterized in that: The method further comprises: When the target face is facing the camera, the aperture of the camera is quickly adjusted to obtain a video with changes in brightness and darkness.

8. The method according to claim 1, characterized in that: The image is an image including a target person's face; the method further includes: The image is matched with the portraits in the certificate library through a face recognition model to perform face recognition.

9. A multi-frame fusion device, characterized in that: include: An allocating unit, configured to divide the frames included in the video into N sets according to the amount of incoming light; A first calculation unit, configured to determine an optimal frame in the set according to image information entropy and image momentum of each frame in the set; A second calculation unit, configured to select a plurality of fused frames from the optimal frame according to the motion field of the optimal frame; The fusion unit is used to divide the fused frame into a plurality of blocks, and assign a fusion weight to each of the fused frames for the blocks at each position; according to the fusion weight, fuse the blocks at the same position of the plurality of fused frames, and form an image based on the fused blocks.

10. An electronic device, characterized in that: include: A memory and a processor, wherein the memory is used to store instructions or codes, and the processor is used to execute the instructions or codes so that the device executes a multi-frame fusion method as described in any one of claims 1-8.

Images