A multi-frame fusion method, device and equipment

Abstract

The application provides a multi-frame fusion method, device and equipment, and relates to the technical field of image processing. The method comprises the following steps: dividing frames included in a video into N sets according to light quantity; determining optimal frames in the sets according to image information entropy and image momentum of each frame in the sets; selecting a plurality of fusion frames from the optimal frames according to a motion field of the optimal frames; dividing the fusion frames into a plurality of blocks, and assigning fusion weights to each of the fusion frames for each block at a position; fusing blocks at the same position of the plurality of fusion frames according to the fusion weights, and forming an image based on the fused blocks. In this way, a plurality of frames of images in the video can be fused into one image, the information quantity of the image is increased, and the problem of incomplete image information caused by factors such as light change or portrait shaking is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a multi-frame fusion method, device and equipment. BACKGROUND

[0002] When taking a photo, the picture information is often imperfect due to overexposure, insufficient light, and portrait movement.

[0003] Currently, imperfect picture information greatly affects the image perception, and in addition, in image specific application scenarios such as face recognition, imperfect picture information also affects the recognition accuracy. SUMMARY

[0004] Therefore, the present application provides a multi-frame fusion method, device and equipment, aiming to solve the problem of imperfect picture information caused by light or shaking.

[0005] In a first aspect, a multi-frame fusion method comprises:

[0006] According to the light amount, the frames included in the video are divided into N sets;

[0007] According to the image information entropy and image momentum of each frame in the set, the optimal frame in the set is determined;

[0008] According to the motion field of the optimal frame, a plurality of fusion frames are selected from the optimal frame;

[0009] The fusion frames are divided into a plurality of tiles, and each fusion frame is assigned a fusion weight for each tile at a position; according to the fusion weight, the tiles at the same position of the plurality of fusion frames are fused, and an image is formed based on the fused tiles.

[0010] Optionally, the dividing the frames included in the video into N sets according to the light amount comprises:

[0011] Calculating the average gray value of each frame after converting each frame to a gray frame;

[0012] Dividing the gray value in the range of 0 to 255 into N gray value segments in turn, assigning the frames included in the video to the corresponding gray value segment according to the average gray value of each frame, and forming a set of corresponding frames in each gray value segment, each gray value segment corresponding to at least one frame.

[0013] Optionally, the determining the optimal frame in the set according to the image information entropy and image momentum of each frame in the set comprises:

[0014] Calculating the image information entropy and image momentum of each frame in each set of the N sets;

[0015] multiply the image information entropy of each frame by a value of a first weight coefficient, and add a value of the image momentum of the frame multiplied by a second weight coefficient to obtain a preferred score of the frame;

[0016] take the frame with the largest preferred score in the set as an optimal frame.

[0017] Optionally, the method for calculating the image momentum comprises:

[0018] calculating the motion vector of each pixel point in each frame;

[0019] calculating the image momentum of the entire image of the frame according to the motion vector of each pixel point in the frame;

[0020] the motion vector of each pixel point is calculated according to the following formula:

[0021] wherein v is the motion vector of the pixel point; I is the brightness of the pixel point at time t, expressed as (x, y, t), x is the horizontal coordinate position of the pixel point, and y is the vertical coordinate position of the pixel point; I t represents the partial derivative of I with respect to t;

[0022] the image momentum of the entire image is calculated according to the following formula:

[0023] wherein M(A) is the total momentum of the A-frame image, and v z represents the motion vector of the zth pixel point.

[0024] Optionally, the method for selecting a plurality of fusion frames from the optimal frames according to the motion field of the optimal frames comprises:

[0025] acquiring a two-dimensional motion field according to the motion directions x and y in the two-dimensional plane where the image of the optimal frame is located, and calculating the motion field between the two optimal frames;

[0026] predicting the minimum motion field of the K+1th optimal frame according to the motion fields of the first K optimal frames, wherein K is a positive integer greater than 1 and smaller than the total number of the optimal frames;

[0027] selecting a plurality of optimal frames with the smallest values from the predicted minimum motion fields of the plurality of optimal frames as the fusion frames.

[0028] Optionally, the fusion weight is:

[0029]

[0030] wherein j is a coefficient; D c b(w) is the tile difference of the tile position c in the B-frame after Fourier transform, e is a constant, and w is the frequency of the tile; is the noise variance.

[0031] Optionally, the method further comprises:

[0032] In the case of a target face aligning the camera, quickly adjusting the aperture of the camera to obtain a video with light and dark changes.

[0033] Optionally, the image is an image including a target face; the method further comprises:

[0034] Matching the image with the portraits in the certificate library through a face recognition model to perform face recognition.

[0035] In a second aspect, the present application provides a multi-frame fusion device, comprising:

[0036] An allocation unit configured to divide frames included in the video into N sets according to the light quantity;

[0037] A first calculation unit configured to determine optimal frames in the sets according to image information entropy and image momentum of each frame in the sets;

[0038] A second calculation unit configured to select a plurality of fusion frames from the optimal frames according to the motion field of the optimal frames;

[0039] A fusion unit configured to divide the fusion frames into a plurality of tiles, and assign a fusion weight to each of the fusion frames for a tile at a same position; fuse the tiles at the same position of the plurality of fusion frames according to the fusion weights, and form an image based on the fused tiles.

[0040] In a third aspect, the present application provides an electronic device, comprising a memory and a processor, wherein the memory is configured to store instructions or codes, and the processor is configured to execute the instructions or codes to enable the device to perform any of the multi-frame fusion methods described above.

[0041] The present application provides a multi-frame fusion method, comprising: dividing frames included in the video into N sets according to the light quantity; determining optimal frames in the sets according to image information entropy and image momentum of each frame in the sets; selecting a plurality of fusion frames from the optimal frames according to the motion field of the optimal frames; dividing the fusion frames into a plurality of tiles, and assigning a fusion weight to each of the fusion frames for a tile at a same position; fusing the tiles at the same position of the plurality of fusion frames according to the fusion weights, and forming an image based on the fused tiles. In this way, a plurality of images in the video can be fused into one image, the information quantity of the image is increased, and the problem of incomplete image information caused by factors such as light change or portrait shaking is compensated. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0043] Figure 1 A flowchart of a multi-frame fusion method provided by the embodiment of the present application is shown in the figure.

[0044] Figure 2 A figure showing the change of image information entropy from dark to light is shown in the figure.

[0045] Figure 3 A figure showing the distribution of pixel points in the range of 0 to 255 gray scale values in each of the nine optimal frames provided by the embodiment of the present application is shown in the figure.

[0046] Figure 4 A figure showing the formation of an image by three fusion frames provided by the embodiment of the present application is shown in the figure.

[0047] Figure 5 A flowchart of a multi-frame fusion for face authentication provided by the embodiment of the present application is shown in the figure.

[0048] Figure 6 A structure diagram of a multi-frame fusion device provided by the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0049] When taking pictures, it often happens that the picture information is not perfect due to overexposure, insufficient light, and portrait movement. Especially in the increasingly widespread application of portrait recognition security fields, such as access control systems, border inspection systems, etc. Through face recognition technology, fast and efficient identity verification can be achieved to improve security and convenience. Because the face picture may be captured due to overexposure or insufficient light, portrait movement, etc. which may result in imperfect picture information, which may reduce the success rate of face matching. In order to improve the success rate of face recognition in special scenarios, it is very necessary to perform image enhancement processing on the face pictures in the scenario.

[0050] Therefore, the application provides a multi-frame fusion method, comprising: dividing frames included in the video into N sets according to light quantity; determining optimal frames in the sets according to image information entropy and image momentum of each frame in the sets; selecting a plurality of fusion frames from the optimal frames according to motion fields of the optimal frames; dividing the fusion frames into a plurality of tiles, and assigning fusion weights to each of the fusion frames for tiles at each position; fusing tiles at the same position of the plurality of fusion frames according to the fusion weights, and forming an image based on the fused tiles. In this way, a plurality of frames of images in the video can be fused into one image, the information quantity of the image is increased, image enhancement processing is realized, the problem of incomplete image information caused by factors such as light change or portrait shaking is compensated, and the success rate of image matching in the face recognition field is improved.

[0051] In order to enable a person skilled in the art to more fully understand the features and technical contents of the embodiments of the present disclosure, the implementation of the embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings, which are only used for reference and do not limit the embodiments of the present disclosure. In the following technical description, in order to facilitate explanation, a plurality of details are provided to provide a full understanding of the disclosed embodiments. However, one or more embodiments can still be implemented without these details. In other cases, in order to simplify the drawings, well-known structures and devices can be simplified.

[0052] The terms "first", "second", and the like in the specification and claims of the embodiments of the present disclosure and the above drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion.

[0053] Unless otherwise specified, the term "plurality" means two or more.

[0054] In the embodiments of the present disclosure, the character " / " represents an "or" relationship between the objects before and after it. For example, A / B represents: A or B.

[0055] The term "and / or" is a description of the relationship between the objects, which means that there can be three relationships. For example, A and / or B means: A or B, or, A and B, the three relationships.

[0056] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work belong to the scope of protection of the present application.

[0057] Referring to Figure 1 , Figure 1 A flowchart of a multi-frame fusion method provided by the embodiments of the present application is shown in FIG. 1. The multi-frame fusion method comprises the following steps.

[0058] S101, dividing frames included in the video into N sets according to light quantity.

[0059] For example, the frames in the video are divided into sets according to the gray values of the images.

[0060] The video can be a video including a face, which can be used for face recognition.

[0061] S102, determining optimal frames in the sets according to image information entropy and image momentum of the frames in the sets.

[0062] Image information entropy, also known as image entropy, represents the average number of bits of a gray level set of an image, with a unit of bit / pixel, and describes the average information quantity of the image.

[0063] The image momentum is determined according to the motion vectors of the pixel points in each frame image.

[0064] Optionally, the determination of the optimal frames in the sets can be that at least one frame is included in each set, one optimal frame is selected from each set, and N optimal frames can be determined from N sets.

[0065] S103, selecting a plurality of fusion frames from the optimal frames according to the motion field of the optimal frames.

[0066] The motion field represents instantaneous motion velocity vector information of the pixel points. Specifically, in an image plane, the motion of an object is often embodied by different gray distributions in different images in an image sequence, so that the motion field in space is transferred to the image plane as an optical flow field. The optical flow field is a two-dimensional vector field, which reflects the change trend of the gray of each point in the image, and can be regarded as an instantaneous velocity field generated by the motion of the pixel points with gray in the image plane.

[0067] S104, dividing the fusion frames into a plurality of tiles, and assigning a fusion weight to each of the fusion frames for each tile at a position; fusing the tiles at the same position of the plurality of the fusion frames according to the fusion weights, and forming an image based on the fused tiles.

[0068] The conventional image fusion algorithm can be used to form an image from the plurality of the fusion frames according to the fusion weights of the fusion frames for each tile at each position.

[0069] According to the steps S101-S104, the plurality of images in the video are fused into one image, the information amount of the image is increased, and the problem of incomplete image information caused by factors such as illumination change or portrait shaking is solved.

[0070] In the embodiment of the present application, the above Figure 1 The step S101 has a plurality of possible implementation manners, which will be introduced below. It should be noted that the implementation manners given in the following introduction are only exemplary and do not represent all implementation manners of the embodiment of the present application.

[0071] In one possible implementation manner, the frames included in the video are divided into N sets according to the light amount, including:

[0072] The average gray value of each frame after conversion to a gray frame is calculated.

[0073] The gray value in the range of 0 to 255 is divided into N gray value segments in sequence, the frames included in the video are assigned to the corresponding gray value segments according to the average gray value of each frame, the corresponding frames in each gray value segment form a set, and each gray value segment corresponds to at least one frame.

[0074] The gray value ranges from 0 to 255.

[0075] Optionally, the video is a video from dark to light.

[0076] In one specific implementation manner, each frame can be assigned to the corresponding set by the following formula.

[0077]

[0078] Wherein, FrameSet g is the gth set; Average Frame is the average gray value of the Frame frame; and N is the total number of sets to be assigned.

[0079] In the embodiment of the present application, the above Figure 1The step S102 has various possible implementation manners, which are described below. It should be noted that the implementation manners given in the following description are only exemplary and do not represent all implementation manners of the embodiments of the present application.

[0080] According to the image information entropy and the image momentum of each frame in the set, the optimal frame in the set is determined, including:

[0081] Step A1, calculating the image information entropy and the image momentum of each frame in each set of the N sets.

[0082] The specific formula for calculating the image information entropy of the A frame can be H(A) = -∑(p(i) * log2(p(i)))

[0083] Wherein, H(A) is the image information entropy of the A frame; p(i) is the probability of the frame picture being the pixel value i. Referring to the image information entropy change diagram from dark to light shown in Figure 2 If the picture is too dark or too bright, p(i) will be concentrated in a smaller area, reducing the picture information entropy.

[0084] The specific method for calculating the image momentum of the A frame can be:

[0085] First, the motion vector of each pixel point in each frame is calculated.

[0086] Specifically, the motion vector calculation formula of each pixel point is ∇I T v = -I t

[0087] Wherein, v is the motion vector of the pixel point; I is the brightness of the pixel point at t moment, represented as (x, y, t), x is the horizontal coordinate position of the pixel point, and y is the vertical coordinate position of the pixel point; I t represents the partial derivative of I to t;

[0088] Then, according to the motion vector of each pixel point in each frame, the image momentum of the entire image of the frame is calculated.

[0089] Specifically, the image momentum calculation formula of the entire image is:

[0090] Wherein, M(A) is the total momentum of the A frame image, v z represents the motion vector of the zth pixel point.

[0091] Step A2, multiplying the image information entropy of each frame by the value of the first weight coefficient, and adding the value of the image momentum of the frame multiplied by the second weight coefficient to obtain the preferred score of the frame. The specific formula can be:

[0092] T(A) = r1H(A) + r2M(A)

[0093] T(A) is the preferred score of the A frame, H(A) is the image information entropy of the A frame, M(A) is the total momentum value of the A frame, r1 is the first weight coefficient, and r2 is the second weight coefficient.

[0094] Step A3, taking the frame with the maximum preferred score in the set as the optimal frame.

[0095] According to the preferred score ranking of all frames in the set, the frame corresponding to the maximum value is selected as the optimal frame of the set. For example, refer to FIG. 2. Figure 3 FIG. 3 shows the distribution of pixel points in the gray value range of 0 to 255 in each of the nine optimal frames shown in FIG. 2, where the horizontal axis is the gray value and the vertical axis is the number of pixel points.

[0096] In the embodiments of the present application, the preferred score T(A) of the A frame is calculated according to the following formula: Figure 1 The step S103 has various possible implementation manners, which will be introduced below. It should be noted that the implementation manners given in the following introduction are only exemplary and do not represent all implementation manners of the embodiments of the present application.

[0097] The step of selecting a plurality of fusion frames from the optimal frames according to the motion field of the optimal frames comprises:

[0098] Step B1, obtaining a two-dimensional motion field according to the motion directions x and y in the two-dimensional plane where the images of the optimal frames are located, and calculating the motion field between the two optimal frames.

[0099] The specific calculation method comprises: first calculating the motion field of each pixel point between the two optimal frames, and then adding the motion fields of the pixel points to obtain the motion field between the two optimal frames. The calculation formula for calculating the motion field of each pixel point between the two optimal frames is:

[0100]

[0101] In the above formula, ΔI t (x, y) is the luminance change of the pixel point at coordinates (x, y), is the gradient value of the pixel point at coordinates (x, y), is the motion field of the pixel point at coordinates (x, y).

[0102] Step B2, predicting the minimum motion field of the K+1th optimal frame according to the motion fields of the first K optimal frames, where K is a positive integer greater than 1 and less than the total number of optimal frames.

[0103] It can be understood that the first K optimal frames are in the time order of the video, and the Kth optimal frame and the K+1th optimal frame in the plurality of optimal frames are the same.

[0104] The specific prediction formula can be:

[0105] 1-Pr[V min <=V K ] K >=δ

[0106] In the above formula, V min is the minimum motion field of the predicted K+1 optimal frame, Pr[V min <=V K ] represents the probability that V min is less than the average of the modulus values of the previous K optimal frames, and δ is a set threshold value of prediction success.

[0107] For example, K is set to 3, and the modulus value of the first three optimal frames is calculated as follows: the square of the motion field between the first and second optimal frames is calculated, and the square of the motion field between the second and third optimal frames is added to obtain a sum value; then the sum value is square rooted to obtain the modulus value of the first three optimal frames.

[0108] Step B3, selecting the optimal frames with the minimum motion field from the plurality of optimal frames as the fusion frames.

[0109] In this way, by the above steps B1-B3, the optimal frames with the minimum motion field are selected from the plurality of optimal frames to minimize the blurring problem caused by shaking.

[0110] In the embodiments of the present application, the above-mentioned Figure 1 The fusion weight calculation formula in step S104 can be:

[0111]

[0112] wherein, j is a coefficient; D c b(w) is the tile difference of the tile position c in the b frame after Fourier transform, and e is a constant; is the noise variance.

[0113] In the above step S104, in a specific implementation mode, first, the fusion frames are divided into a plurality of tiles, and each position tile is assigned a fusion weight for each of the fusion frames. For example, the plurality of fusion frames are divided in the same division manner, each fusion frame includes a plurality of tiles, and then the tiles at the same position of the plurality of fusion frames are assigned fusion weights. Subsequently, the plurality of tiles at the position are fused by using the existing image fusion technology in combination with the corresponding fusion weights to obtain a fused image at the position. Further, the fused tiles are spliced to form an image, which can be specifically referred to as Figure 4 The three fusion frames are fused to form an image, as shown in the schematic diagram of the image formed by fusing three fusion frames. Figure 3The nine optimal frames shown in the extraction of the fusion frame, such as Figure 4 The pre-fusion frame one, the pre-fusion frame two and the pre-fusion frame three in the above formula, and the post-fusion frame (image) obtained after fusion.

[0114] In summary, the present application is based on the segmentation of gray values, obtains a set corresponding to different gray value breaks, and calculates the image information entropy and the image momentum from each set to avoid over-bright or over-dark images to extract the optimal frame. Then, several fusion frames are extracted from the multiple optimal frames based on the minimum motion field as a reference, reducing the blurring caused by the shaking of the target object in the image. Thus, several fusion frames are fused to obtain the final image, thereby fusing multiple frames of images in the video into one image, increasing the information amount of the image, and compensating for the imperfect image information caused by factors such as changes in lighting or shaking of the image.

[0115] Based on the above Figure 1 The steps S101-S104 shown in the above formula, the present application can be further applied in the field of face recognition. Referring to Figure 5 A flowchart of a multi-frame fusion for face authentication.

[0116] Optionally, before the above step S101, the above method further includes obtaining a video including a target face. Specifically, the specific method of obtaining a video including a target face includes: in the case that the target face is aligned with the camera, quickly adjusting the aperture of the camera, and obtaining a video with light and dark changes.

[0117] The above target face aligned with the camera can keep the target face in the position corresponding to the camera.

[0118] The above video with light and dark changes can be a video from dark to light or from light to dark. In this way, multiple frames included in the video can be divided into multiple sets in turn.

[0119] Optionally, after the above step S104, the method further includes: matching the image with the portraits in the certificate library through a face recognition model to perform face recognition authentication.

[0120] Specifically, the similarity is used for face feature matching, and the specific formula can be:

[0121]

[0122] In the above formula, dist is the similarity calculated by the feature information, Feature face is the feature of the image obtained by fusion, Feature lib is the feature of the certificate library.

[0123] Further, referring to Figure 5An application flow diagram of a multi-frame fusion for face authentication is shown. The application provides a multi-frame fusion method for face authentication, which comprises the following steps.

[0124] S501, in the case of a target face aligning a camera, quickly adjusting the aperture of the camera to obtain a video with light and dark changes.

[0125] S502, according to the light quantity, the frames included in the video are divided into N sets;

[0126] S503, according to the image information entropy and image momentum of each frame in the set, the optimal frame in the set is determined;

[0127] S504, according to the motion field of the optimal frame, a plurality of fusion frames are selected from the optimal frame;

[0128] S505, the fusion frames are divided into a plurality of tiles, and each fusion frame is assigned a fusion weight for each tile; according to the fusion weight, the tiles of the same position of the plurality of fusion frames are fused, and an image is formed based on the fused tiles.

[0129] S506, the image is matched with the portrait in the certificate library through a face recognition model to perform face recognition.

[0130] Based on the above steps S501-S506, a plurality of frames of images in the video are fused into an image, the information quantity of the image is increased, the image enhancement processing is realized, the problem of incomplete image information caused by factors such as light change or portrait shaking is compensated, and the success rate of image matching in the field of face recognition is improved.

[0131] The above is some specific implementation manners of the multi-frame fusion method provided by the embodiments of the application. Based on this, the application also provides a corresponding device. The device provided by the embodiments of the application will be introduced from the perspective of functional modularization.

[0132] Referring to Figure 6 A structure diagram of a multi-frame fusion device is shown. A multi-frame fusion device comprises:

[0133] The distribution unit 601 is configured to divide the frames included in the video into N sets according to the light quantity;

[0134] The first calculation unit 602 is configured to determine the optimal frame in the set according to the image information entropy and image momentum of each frame in the set;

[0135] The second calculation unit 603 is configured to select a plurality of fusion frames from the optimal frame according to the motion field of the optimal frame;

[0136] The fusion unit 604 is configured to divide the fusion frames into a plurality of tiles, and assign a fusion weight to each tile of each fusion frame; and fuse the tiles of the same position of the plurality of fusion frames according to the fusion weights, and form an image based on the fused tiles.

[0137] Based on the above device, the allocation unit 601 segments based on the light amount to obtain a set corresponding to different gray value segments, the first calculation unit 602 calculates the image information entropy and the image momentum of each set to avoid over-bright or over-dark images to extract the optimal frame, and then the second calculation unit 603 extracts several fusion frames from the plurality of optimal frames based on the minimum motion field as a reference to reduce the blur caused by the shaking of the target in the image. Finally, the fusion unit 604 fuses the several fusion frames to obtain the final image, thereby fusing multiple frames of images in the video into one image, increasing the information amount of the image, and making up for the problem of incomplete image information caused by factors such as changes in illumination or shaking of the image.

[0138] In a possible implementation, the allocation unit 601 is specifically configured to calculate the average gray value of each frame after the frame is converted into a gray frame; divide the gray value in the range of 0 to 255 into N gray value segments in sequence, and distribute the frames included in the video to the corresponding gray value segments according to the average gray value of each frame, so that the frames corresponding to each gray value segment form a set, and each gray value segment corresponds to at least one frame.

[0139] In a possible implementation, the first calculation unit 602 is specifically configured to calculate the image information entropy and the image momentum of each frame in each set of the N sets; multiply the image information entropy of each frame by the value of the first weight coefficient, add the value of the image momentum of the frame multiplied by the second weight coefficient, to obtain the preferred score of the frame; and take the frame with the maximum preferred score in the set as the optimal frame.

[0140] In a possible implementation, the first calculation unit 602 is further configured to calculate the motion vector of each pixel point in each frame; and calculate the image momentum of the entire image of the frame according to the motion vector of each pixel point in the frame.

[0141] The motion vector of each pixel point is calculated according to the following formula:

[0142] Wherein, v is the motion vector of the pixel point; I is the brightness of the pixel point at time t, represented as (x, y, t), x is the horizontal coordinate position of the pixel point, and y is the vertical coordinate position of the pixel point; I t represents the partial derivative of I with respect to t;

[0143] The image momentum of the entire image is calculated according to the following formula:

[0144] wherein M(A) is the total momentum of the A-frame image, v z represents the motion vector of the zth pixel point.

[0145] In a possible implementation, the second calculation unit 603 is specifically configured to: acquire a two-dimensional motion field according to the motion directions x and y on a two-dimensional plane where the images of the optimal frames are located, and calculate the motion field between two optimal frames; predict the minimum motion field of the K+1th optimal frame according to the motion fields of the first K optimal frames, K being a positive integer greater than 1 and smaller than the total number of the optimal frames; and select the optimal frames with the minimum values from the minimum motion fields of the multiple optimal frames as the fusion frames.

[0146] In a possible implementation, the fusion weight is:

[0147]

[0148] wherein j is a coefficient; D c b(w) is the block difference of the block with the frequency w after Fourier transform for the block position c in the b-frame, and e is a constant. is the noise variance.

[0149] In a possible implementation, the device further includes an acquisition unit configured to quickly adjust the aperture of the camera in the case that the target face is aligned with the camera, and acquire a video with light and dark changes.

[0150] In a possible implementation, the image is an image including a target face; and the device further includes an identification unit configured to match the image with a portrait in a certificate library through a face recognition model, and perform face recognition.

[0151] Embodiments of the present application also provide a corresponding device and a computer storage medium, for implementing the scheme provided by the embodiments of the present application.

[0152] The device includes a memory and a processor, the memory is configured to store instructions or codes, and the processor is configured to execute the instructions or codes, so that the device executes the multi-frame fusion method provided by any of the embodiments of the present application.

[0153] The computer storage medium stores codes, and when the codes are executed, a device executing the codes implements the multi-frame fusion method provided by any of the embodiments of the present application.

[0154] The names of “first” and “second” mentioned in the embodiments of the present application are only used for name identification, and do not represent the first and second in order.

[0155] Those skilled in the art can clearly understand the above-mentioned method of the embodiment, all or part of the steps in the above-mentioned method can be realized by means of software and a general hardware platform. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product. The computer software product can be stored in a storage medium, such as a read-only memory (ROM) / RAM, a magnetic disk, an optical disk, etc., and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network communication device such as a router) to execute the method described in each embodiment or some parts of the embodiments of the present application.

[0156] Each of the embodiments in the specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, it is described relatively simply, and the relevant part can be referred to the part of the method embodiment. According to the actual needs, part or all of the modules can be selected to achieve the purpose of the embodiment. Those skilled in the art can understand and implement without creative labor.

[0157] The above-mentioned is only an exemplary embodiment of the present application, and is not used to limit the protection scope of the present application.

Claims

1. A multi-frame fusion method, characterized in that, The method comprises the following steps: According to the light quantity, the frames included in the video are divided into N sets; According to the image information entropy and image momentum of each frame in the set, the optimal frame in the set is determined; According to the motion field of the optimal frame, a plurality of fusion frames are selected from the optimal frame; The fusion frames are divided into a plurality of tiles, and each fusion frame is assigned a fusion weight for each tile at a position; according to the fusion weight, the tiles at the same position of a plurality of fusion frames are fused, and an image is formed based on the fused tiles.

2. The method of claim 1, wherein, According to the light quantity, the frames included in the video are divided into N sets, which comprises the following steps: Calculate the average gray value of each frame after converting each frame to a gray frame; The gray value in the range of 0 to 255 is divided into N gray value segments in turn, and the frames included in the video are assigned to the corresponding gray value segment according to the average gray value of each frame. The corresponding frames in each gray value segment form a set, and each gray value segment corresponds to at least one frame.

3. The method of claim 1, wherein, According to the image information entropy and image momentum of each frame in the set, the optimal frame in the set is determined, which comprises the following steps: Calculate the image information entropy and image momentum of each frame in each set of the N sets; The image information entropy of each frame is multiplied by the value of the first weight coefficient, and the image momentum of the frame is multiplied by the value of the second weight coefficient to obtain the preferred score of the frame; The frame with the maximum preferred score in the set is selected as the optimal frame.

4. The method of claim 3, wherein, The method for calculating the image momentum comprises the following steps: Calculate the motion vector of each pixel point in each frame; According to the motion vector of each pixel point in each frame, the image momentum of the entire image of the frame is calculated; The motion vector calculation formula of each pixel point is: ∇I T v = -I t where v is the motion vector of the pixel point; I is the brightness of the pixel point at the t moment, represented as (x, y, t), x is the horizontal coordinate position of the pixel point, and y is the vertical coordinate position of the pixel point; I t represents the partial derivative of I with respect to t; The image momentum calculation formula of the whole image is: where M(A) is the total momentum of the A-frame image, v z denotes the motion vector of the zth pixel.

5. The method of claim 1, wherein, According to the motion field of the optimal frame, a plurality of fusion frames are selected from the optimal frame, which comprises the following steps: On the two-dimensional plane where the image of the optimal frame is located, a two-dimensional motion field is obtained according to the motion directions x and y, and the motion field between the two optimal frames is calculated; According to the motion field of the first K optimal frames, the minimum motion field of the K+1 optimal frame is predicted, K is a positive integer greater than 1 and less than the total number of optimal frames; Select the optimal frames with the smallest values from the predicted minimum motion fields of the optimal frames as the fusion frames.

6. The method of claim 1, wherein: The fusion weight is: where j is a coefficient; D c b(w) is the tile difference for tile position c in the b-frame after Fourier transform at frequency w, e is a constant; is the noise variance.

7. The method of claim 1, wherein, The method further comprises: In the case of aligning the target face to the camera, quickly adjusting the aperture of the camera to obtain a video segment with light and dark changes.

8. The method of claim 1, wherein: The image is an image including a target face; and the method further comprises: Matching the image with the portraits in the certificate library through a face recognition model to perform face recognition.

9. A multi-frame fusion device, characterized by, The method comprises the following steps: A distribution unit is configured to divide the frames included in the video into N sets according to the light quantity; A first calculation unit is configured to determine the optimal frame in the set according to the image information entropy and image momentum of each frame in the set; A second calculation unit is configured to select a plurality of fusion frames from the optimal frame according to the motion field of the optimal frame; A fusion unit is configured to divide the fusion frames into a plurality of tiles, and assign a fusion weight for each fusion frame for each tile at a position; according to the fusion weight, the tiles at the same position of a plurality of fusion frames are fused, and an image is formed based on the fused tiles.

10. An electronic device, comprising: The method comprises the following steps: A memory for storing instructions or codes, and a processor for executing the instructions or codes to make the device perform the multi-frame fusion method of any one of claims 1-8.

Images