Video frame alignment method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a video frame alignment method and device, electronic equipment and a storage medium, and relates to the technical field of image processing. The method comprises the following steps: determining a current to-be-aligned video frame set from a to-be-processed video according to a time sequence of video frames; obtaining pre-recorded current to-be-utilized optical flow information; determining a current newly-added frame from the current to-be-aligned video frame set; calculating optical flow information between each current newly-added frame and a previous video frame of the newly-added frame in the current to-be-aligned video frame set, and taking the optical flow information as current newly-added optical flow information; on the basis of the current to-be-utilized optical flow information and the newly-added optical flow information, aligning each current associated frame to the current reference frame to obtain an alignment result of each current associated frame; and returning to execute the step of determining the current to-be-aligned video frame set from the to-be-processed video according to the time sequence of the video frames so as to reduce the consumption of computing power resources of the electronic equipment.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly to a video frame alignment method, apparatus, electronic device, and storage medium. Background Art

[0002] In the technical field of image processing, video frames in a video can be processed. Video frame alignment can be applied to various processing tasks. For example, when performing video enhancement, for each video frame in the video, an associated video frame (hereinafter referred to as an associated frame) associated with the video frame can be aligned with the video frame, and then the video frame can be enhanced based on the alignment result. Exemplarily, for each associated frame, an electronic device can calculate the optical flow information between the associated frame and the video frame based on the optical flow method, and then align the associated frame with the video frame based on the optical flow information.

[0003] Obviously, the more the number of associated frames aligned with the video frame, the better the enhancement effect on the video frame. However, the more the number of associated frames, the more computing power resources are consumed during alignment. When aligning multiple associated frames with the video frame, how to reduce the consumption of computing power resources of the electronic device has become an urgent technical problem to be solved. Summary of the Invention

[0004] The purpose of the embodiments of the present invention is to provide a video frame alignment method, apparatus, electronic device, and storage medium to reduce the consumption of computing power resources of an electronic device when aligning multiple associated frames with the video frame. The specific technical solutions are as follows:

[0005] In the first aspect of the embodiments of the present invention, a video frame alignment method is provided. The method includes: determining a current set of video frames to be aligned from a video to be processed according to the time sequence of the video frames; where a set of video frames includes a reference frame and associated frames that need to be aligned with the reference frame; obtaining the optical flow information between every two adjacent video frames in the current set of associated video frames recorded in advance when aligning the current set of associated video frames, as the current optical flow information to be utilized; where the current set of associated video frames is the set of video frames when the multiple video frames in the current set of video frames to be aligned participated in alignment last time; determining the video frames that do not belong to the current set of associated video frames from the current set of video frames to be aligned, as the current new frames; calculating the optical flow information between each current new frame and the previous video frame of the current new frame in the current set of video frames to be aligned, as the current new optical flow information; based on the current optical flow information to be utilized and the new optical flow information, aligning each current associated frame with the current reference frame respectively to obtain the alignment result of each current associated frame; and returning to execute the step of determining the current set of video frames to be aligned from the video to be processed according to the time sequence of the video frames.

[0006] Optionally, aligning each current associated frame to the current reference frame based on the current optical flow information to be utilized and the new optical flow information to obtain the alignment result of each current associated frame includes: determining, from the current associated frames, the video frames belonging to the current associated video frame set as the current reused frames; aligning the current reused frames to the current reference frame based on the current optical flow information to be utilized to obtain the alignment result of the current reused frames; and aligning, based on the current optical flow information to be utilized and the new optical flow information, the video frames other than the current reference frame in the current new frames to the current reference frame to obtain the alignment result of the video frames other than the current reference frame in the current new frames.

[0007] Optionally, aligning the current multiplexed frame to the current reference frame based on the current optical flow information to be utilized to obtain the alignment result of the current multiplexed frame includes: determining the current first intermediate frame from the current set of video frames to be aligned according to the time sequence of the video frames; wherein the current first intermediate frame includes: the reference frame in the current associated video frame set, the current reference frame, and each video frame between the reference frame in the current associated video frame set and the current reference frame; obtaining the optical flow information between every two adjacent first intermediate frames in time sequence from the current optical flow information to be utilized as the current first optical flow information; for each multiplexed frame, when the time sequence of the multiplexed frame is earlier than the reference frame in the current associated video frame set, obtaining the alignment result of the multiplexed frame to the reference frame in the current associated video frame set obtained when aligning the current associated video frame set previously recorded as the current alignment result to be utilized; aligning the current alignment result to be utilized to the current reference frame in sequence using the current first optical flow information according to the time sequence of the video frames until the alignment result of the multiplexed frame to the current reference frame is obtained; when the multiplexed frame is the reference frame in the current associated video frame set, aligning the multiplexed frame to the current reference frame in sequence using the current first optical flow information according to the time sequence of the video frames to obtain the alignment result of the multiplexed frame; when the time sequence of the multiplexed frame is later than the reference frame in the current associated video frame set and earlier than the current reference frame, determining the current second intermediate frame corresponding to the multiplexed frame from the current set of video frames to be aligned according to the time sequence of the video frames; wherein the current second intermediate frame corresponding to the multiplexed frame includes: the multiplexed frame, the current reference frame, and each video frame between the multiplexed frame and the current reference frame; obtaining the optical flow information between every two adjacent second intermediate frames corresponding to the multiplexed frame in time sequence from the current optical flow information to be utilized as the current second optical flow information corresponding to the multiplexed frame; aligning the multiplexed frame to the current reference frame in sequence using the current second optical flow information corresponding to the multiplexed frame according to the time sequence of the video frames to obtain the alignment result of the multiplexed frame; when the time sequence of the multiplexed frame is later than the current reference frame, obtaining the alignment result of the multiplexed frame to the current reference frame obtained when aligning the current associated video frame set previously recorded.

[0008] Optionally, obtaining the optical flow information between every two adjacent first intermediate frames in the current time sequence from the current optical flow information to be utilized as the current first optical flow information includes: when there is no current newly added frame in the current first intermediate frame, obtaining the optical flow information between every two adjacent first intermediate frames in the current time sequence from the current optical flow information to be utilized as the current first optical flow information; when there is a current newly added frame in the current first intermediate frame and there are multiple current reused frames, for every two adjacent first intermediate frames in the current time sequence, if there is optical flow information between the two adjacent first intermediate frames in the current optical flow information to be utilized, obtaining the optical flow information between the two adjacent first intermediate frames from the current optical flow information to be utilized; otherwise, obtaining the optical flow information between the two adjacent first intermediate frames from the current newly added optical flow information to obtain the current first optical flow information; when there is a current newly added frame in the current first intermediate frame and there is one current reused frame, obtaining the optical flow information between every two adjacent first intermediate frames in the current time sequence from the current newly added optical flow information as the current first optical flow information; and / or, obtaining the optical flow information between every two adjacent second intermediate frames currently corresponding to the reused frame from the current optical flow information to be utilized includes: when there is a current newly added frame in the second intermediate frame currently corresponding to the reused frame, obtaining the optical flow information between every two adjacent second intermediate frames currently corresponding to the reused frame from the current optical flow information to be utilized as the second optical flow information currently corresponding to the reused frame; when there is a current newly added frame in the second intermediate frame currently corresponding to the reused frame and there are multiple current reused frames, for every two adjacent second intermediate frames currently corresponding to the reused frame, if there is optical flow information between the two adjacent second intermediate frames in the current optical flow information to be utilized, obtaining the optical flow information between the two adjacent second intermediate frames from the current optical flow information to be utilized; otherwise, obtaining the optical flow information between the two adjacent second intermediate frames from the current newly added optical flow information to obtain the second optical flow information currently corresponding to the reused frame; when there is a current newly added frame in the second intermediate frame currently corresponding to the reused frame and there is one current reused frame, obtaining the optical flow information between every two adjacent second intermediate frames currently corresponding to the reused frame from the current newly added optical flow information as the second optical flow information currently corresponding to the reused frame.

[0009] Optionally, after determining the current set of video frames to be aligned from the video to be processed according to the time sequence of video frames, the method further includes: updating the video frames recorded at each position in the current video frame queue in sequence according to the time sequence of each video frame in the current set of video frames to be aligned; wherein each video frame in the current associated video frame set is recorded in the current video frame queue; for every two adjacent positions in the video frame queue, they correspond to the positions in the optical flow information queue for recording the optical flow information between the video frames recorded at these two positions, and the positions in the alignment result queue for recording the respective alignment results of the video frames recorded at these two positions; after calculating the optical flow information between the current newly added frame and the previous video frame of the newly added frame in the current set of video frames to be aligned as the current newly added optical flow information, the method further includes: for every two adjacent video frames in time sequence in the current set of video frames to be aligned, if there is optical flow information between these two adjacent video frames in time sequence in the current optical flow information to be utilized, obtaining the optical flow information between these two adjacent video frames in time sequence from the current optical flow information to be utilized; otherwise, obtaining the optical flow information between these two adjacent video frames in time sequence from the current newly added optical flow information to obtain the current updated optical flow information; updating the optical flow information recorded at each position in the current optical flow information queue in sequence according to the corresponding relationship between every two adjacent positions in the video frame queue and the positions in the optical flow information queue; wherein the current optical flow information queue records the current optical flow information to be utilized; after aligning the video frames in the current newly added frame except the current reference frame to the current reference frame based on the current optical flow information to be utilized and the newly added optical flow information to obtain the alignment results of the video frames in the current newly added frame except the current reference frame, the method further includes: updating the alignment results of aligning the current associated frames to the current reference frame in sequence according to the corresponding relationship between every two adjacent positions in the video frame queue and the positions in the alignment result queue; wherein the current alignment result queue records the alignment results of aligning each associated frame in the current associated video frame set to the reference frame.

[0010] Optionally, aligning the current multiplexed frame to the current reference frame based on the current optical flow information to be utilized to obtain the alignment result of the current multiplexed frame includes: for each current multiplexed frame, determining the third intermediate frame currently corresponding to the multiplexed frame from the current set of video frames to be aligned according to the timing sequence of video frames; wherein the third intermediate frame currently corresponding to the multiplexed frame includes: the multiplexed frame, the current reference frame, and each video frame located between the multiplexed frame and the current reference frame; obtaining the optical flow information between every two adjacent third intermediate frames in time sequence corresponding to the multiplexed frame from the current optical flow information to be utilized as the third optical flow information corresponding to the multiplexed frame; calculating the optical flow information between the multiplexed frame and the current reference frame according to the third optical flow information corresponding to the multiplexed frame as the target optical flow information; and aligning the multiplexed frame to the current reference frame using the target optical flow information to obtain the alignment result of the multiplexed frame.

[0011] Optionally, obtaining the optical flow information between every two adjacent third intermediate frames in time sequence corresponding to the multiplexed frame from the current optical flow information to be utilized as the third optical flow information corresponding to the multiplexed frame includes: if there is no current newly added frame in the third intermediate frame currently corresponding to the multiplexed frame, obtaining the optical flow information between every two adjacent third intermediate frames in time sequence corresponding to the multiplexed frame from the current optical flow information to be utilized as the third optical flow information corresponding to the multiplexed frame; if there is a current newly added frame in the third intermediate frame currently corresponding to the multiplexed frame and there are multiple current multiplexed frames, for every two adjacent third intermediate frames in time sequence corresponding to the multiplexed frame, if there is optical flow information between the two adjacent third intermediate frames in the current optical flow information to be utilized, obtaining the optical flow information between the two adjacent third intermediate frames from the current optical flow information to be utilized; otherwise, obtaining the optical flow information between the two adjacent third intermediate frames from the current newly added optical flow information to obtain the third optical flow information corresponding to the multiplexed frame; if there is a current newly added frame in the third intermediate frame currently corresponding to the multiplexed frame and there is one current multiplexed frame, obtaining the optical flow information between every two adjacent third intermediate frames in time sequence corresponding to the multiplexed frame from the current newly added optical flow information as the third optical flow information corresponding to the multiplexed frame.

[0012] Optionally, aligning the video frames in the current new frame except the current reference frame to the current reference frame based on the current optical flow information to be utilized and the new optical flow information, to obtain the alignment result of the video frames in the current new frame except the current reference frame, includes: for each video frame in the current new frame except the current reference frame, determining the fourth intermediate frame currently corresponding to this video frame from the current set of video frames to be aligned according to the time sequence of the video frames; wherein, the fourth intermediate frame currently corresponding to this video frame includes: this video frame, the current reference frame, and each video frame located between this video frame and the current reference frame; obtaining the optical flow information between every two adjacent fourth intermediate frames currently corresponding to this video frame from the current optical flow information to be utilized and the new optical flow information, as the fourth optical flow information currently corresponding to this video frame; if the number of current reused frames in the fourth intermediate frame currently corresponding to this video frame is not greater than 1, aligning this video frame sequentially using the fourth optical flow information currently corresponding to this video frame until the alignment result of this video frame to the current reference frame is obtained; if there are multiple current reused frames in the fourth intermediate frame currently corresponding to this video frame, determining the fifth intermediate frame currently corresponding to this video frame from the fourth intermediate frame currently corresponding to this video frame according to the time sequence of the video frames; wherein, the fifth intermediate frame currently corresponding to this video frame includes: the current reference frame, the current reused frame with the latest time sequence; and each video frame located between the current reference frame and the current reused frame with the latest time sequence; obtaining the optical flow information between every two adjacent fifth intermediate frames currently corresponding to this video frame from the fourth optical flow information currently corresponding to this video frame, as the fifth optical flow information currently corresponding to this video frame; determining the sixth intermediate frame currently corresponding to this video frame from the fourth intermediate frame currently corresponding to this video frame according to the time sequence of the video frames; wherein, the sixth intermediate frame currently corresponding to this video frame includes: the current reused frame with the latest time sequence, this video frame, and each video frame located between the current reused frame with the latest time sequence and this video frame; obtaining the optical flow information between every two adjacent sixth intermediate frames currently corresponding to this video frame from the fourth optical flow information currently corresponding to this video frame, as the sixth optical flow information currently corresponding to this video frame; sequentially aligning this video frame to the current reused frame with the latest time sequence using the sixth optical flow information currently corresponding to this video frame according to the time sequence of the video frames until the alignment result of this video frame to the current reused frame with the latest time sequence is obtained, as the intermediate alignment result currently corresponding to this video frame; sequentially aligning the intermediate alignment result currently corresponding to this video frame to the current reference frame using the fifth optical flow information currently corresponding to this video frame according to the time sequence of the video frames until the alignment result of this video frame to the current reference frame is obtained.

[0013] Optionally, obtaining the optical flow information between every two temporally adjacent fourth intermediate frames corresponding to the current video frame from the current optical flow information to be utilized and the newly added optical flow information as the fourth optical flow information corresponding to the current video frame includes: if the number of current multiplexed frames existing in the fourth intermediate frames corresponding to the current video frame is no more than 1, obtaining the optical flow information between every two temporally adjacent fourth intermediate frames corresponding to the current video frame from the newly added optical flow information as the fourth optical flow information corresponding to the current video frame; if there are multiple current multiplexed frames in the fourth intermediate frames corresponding to the current video frame, for every two temporally adjacent fourth intermediate frames corresponding to the current video frame, if the optical flow information between the two temporally adjacent fourth intermediate frames exists in the current optical flow information to be utilized, obtaining the optical flow information between the two temporally adjacent fourth intermediate frames from the current optical flow information to be utilized; otherwise, obtaining the optical flow information between the two temporally adjacent fourth intermediate frames from the newly added optical flow information to obtain the fourth optical flow information corresponding to the current video frame.

[0014] Optionally, based on the current optical flow information to be utilized and the newly added optical flow information, aligning the video frames other than the current reference frame in the current newly added frames to the current reference frame to obtain the alignment result of the video frames other than the current reference frame in the current newly added frames, includes: for each video frame other than the current reference frame in the current newly added frames, determining the seventh intermediate frame corresponding to the current video frame from the current set of video frames to be aligned according to the time sequence of the video frames; wherein the seventh intermediate frame corresponding to the current video frame includes: the video frame, the current reference frame, and each video frame located between the video frame and the current reference frame; obtaining the optical flow information between every two temporally adjacent seventh intermediate frames corresponding to the current video frame from the current optical flow information to be utilized and the newly added optical flow information as the seventh optical flow information corresponding to the current video frame; calculating the optical flow information between the video frame and the current reference frame according to the seventh optical flow information corresponding to the current video frame as the optical flow information to be utilized; using the optical flow information to be utilized to align the video frame to the current reference frame to obtain the alignment result of the video frame.

[0015] Optionally, obtaining the optical flow information between every two temporally adjacent seventh intermediate frames corresponding to the current video frame from the current optical flow information to be utilized and the newly added optical flow information as the seventh optical flow information corresponding to the current video frame includes: when the number of current multiplexed frames existing in the seventh intermediate frames corresponding to the current video frame is no more than 1, obtaining the optical flow information between every two temporally adjacent seventh intermediate frames corresponding to the current video frame from the newly added optical flow information as the seventh optical flow information corresponding to the current video frame; when there are multiple current multiplexed frames in the seventh intermediate frames corresponding to the current video frame, if there is optical flow information between the two temporally adjacent seventh intermediate frames in the current optical flow information to be utilized, obtaining the optical flow information between the two temporally adjacent seventh intermediate frames from the current optical flow information to be utilized; otherwise, obtaining the optical flow information between the two temporally adjacent seventh intermediate frames from the newly added optical flow information to obtain the seventh optical flow information corresponding to the current video frame.

[0016] Optionally, calculating the optical flow information between each newly added frame and the previous video frame of the newly added frame in the current set of video frames to be aligned as the current newly added optical flow information includes: for each newly added frame, inputting the newly added frame and the previous video frame of the newly added frame in the current set of video frames to be aligned into a pre-trained optical flow prediction model to obtain the newly added optical flow information between the newly added frame and the previous video frame of the newly added frame in the current set of video frames to be aligned; wherein, the optical flow prediction model is trained based on sample video frame pairs and sample labels representing the optical flow information between the sample video frame pairs.

[0017] Optionally, the number of associated frames in a set of video frames is not less than 3; when the number of associated frames is an even number, among the associated frames included in the set of video frames, the number of associated frames earlier in time than the reference frame in the set of video frames is the same as the number of associated frames later in time than the reference frame in the set of video frames; when the number of associated frames is an odd number, among the associated frames included in the set of video frames, the difference between the number of associated frames earlier in time than the reference frame in the set of video frames and the number of associated frames later in time than the reference frame in the set of video frames is 1.

[0018] In a second aspect of the embodiments of the present invention, a video frame alignment device is provided. The device includes: a set determination module, configured to determine a current set of video frames to be aligned from a video to be processed according to the time sequence of the video frames; wherein, a set of video frames includes a reference frame and associated frames that need to be aligned with the reference frame; an information acquisition module, configured to acquire the optical flow information between every two adjacent video frames in the current set of associated video frames obtained when aligning the current set of associated video frames, as the current optical flow information to be utilized; wherein, the current set of associated video frames is the set of video frames when the multiple video frames in the current set of video frames to be aligned participated in alignment last time; a new frame determination module, configured to determine the video frames that do not belong to the current set of associated video frames from the current set of video frames to be aligned, as the current new frames; a calculation module, configured to calculate the optical flow information between each current new frame and the previous video frame of the current new frame in the current set of video frames to be aligned, as the current new optical flow information; an alignment module, configured to align each current associated frame with the current reference frame respectively based on the current optical flow information to be utilized and the new optical flow information, to obtain the alignment result of each current associated frame; and return to execute the step of determining the current set of video frames to be aligned from the video to be processed according to the time sequence of the video frames.

[0019] Optionally, the alignment module is specifically configured to: determine the video frames that belong to the current set of associated video frames from the current associated frames, as the current reused frames; align the current reused frames with the current reference frame based on the current optical flow information to be utilized, to obtain the alignment result of the current reused frames; and align the video frames other than the current reference frame in the current new frames with the current reference frame based on the current optical flow information to be utilized and the new optical flow information, to obtain the alignment result of the video frames other than the current reference frame in the current new frames.

[0020] Optionally, the alignment module is specifically configured to: determine a current first intermediate frame from the current set of video frames to be aligned according to the time sequence of the video frames; wherein the current first intermediate frame includes: a reference frame in the current associated video frame set, the current reference frame, and each video frame located between the reference frame in the current associated video frame set and the current reference frame; obtain the optical flow information between every two adjacent first intermediate frames in time sequence from the current optical flow information to be utilized as the current first optical flow information; for each multiplexing frame, when the time sequence of the multiplexing frame is earlier than the reference frame in the current associated video frame set, obtain the alignment result of the multiplexing frame to the reference frame in the current associated video frame set obtained when aligning the current associated video frame set previously recorded as the current alignment result to be utilized; align the current alignment result to be utilized to the current reference frame in sequence according to the time sequence of the video frames using the current first optical flow information until the alignment result of the multiplexing frame to the current reference frame is obtained; when the multiplexing frame is the reference frame in the current associated video frame set, align the multiplexing frame to the current reference frame in sequence according to the time sequence of the video frames using the current first optical flow information to obtain the alignment result of the multiplexing frame; when the time sequence of the multiplexing frame is later than the reference frame in the current associated video frame set and earlier than the current reference frame, determine the current second intermediate frame corresponding to the multiplexing frame from the current set of video frames to be aligned according to the time sequence of the video frames; wherein the current second intermediate frame corresponding to the multiplexing frame includes: the multiplexing frame, the current reference frame, and each video frame located between the multiplexing frame and the current reference frame; obtain the optical flow information between every two adjacent second intermediate frames corresponding to the multiplexing frame in time sequence from the current optical flow information to be utilized as the current second optical flow information corresponding to the multiplexing frame; align the multiplexing frame to the current reference frame in sequence according to the time sequence of the video frames using the current second optical flow information corresponding to the multiplexing frame to obtain the alignment result of the multiplexing frame; when the time sequence of the multiplexing frame is later than the current reference frame, obtain the alignment result of the multiplexing frame to the current reference frame obtained when aligning the current associated video frame set previously recorded.

[0021] Optionally, the alignment module is specifically configured to: in the case that the current new frame does not exist in the current first intermediate frame, obtain the optical flow information between every two adjacent first intermediate frames in time sequence from the current optical flow information to be utilized as the current first optical flow information; in the case that the current new frame exists in the current first intermediate frame and there are multiple current reused frames, for every two adjacent first intermediate frames in time sequence, if the optical flow information between the two adjacent first intermediate frames in time sequence exists in the current optical flow information to be utilized, obtain the optical flow information between the two adjacent first intermediate frames in time sequence from the current optical flow information to be utilized; otherwise, obtain the optical flow information between the two adjacent first intermediate frames in time sequence from the current new optical flow information to obtain the current first optical flow information; in the case that the current new frame exists in the current first intermediate frame and there is one current reused frame, obtain the optical flow information between every two adjacent first intermediate frames in time sequence from the current new optical flow information as the current first optical flow information; and / or, obtaining the optical flow information between every two adjacent second intermediate frames corresponding to the reused frame currently from the current optical flow information to be utilized includes: in the case that the current new frame exists in the second intermediate frame corresponding to the reused frame currently, obtain the optical flow information between every two adjacent second intermediate frames corresponding to the reused frame currently from the current optical flow information to be utilized as the second optical flow information corresponding to the reused frame currently; in the case that the current new frame exists in the second intermediate frame corresponding to the reused frame currently and there are multiple current reused frames, for every two adjacent second intermediate frames corresponding to the reused frame currently, if the optical flow information between the two adjacent second intermediate frames exists in the current optical flow information to be utilized, obtain the optical flow information between the two adjacent second intermediate frames from the current optical flow information to be utilized; otherwise, obtain the optical flow information between the two adjacent second intermediate frames from the current new optical flow information to obtain the second optical flow information corresponding to the reused frame currently; in the case that the current new frame exists in the second intermediate frame corresponding to the reused frame currently and there is one current reused frame, obtain the optical flow information between every two adjacent second intermediate frames corresponding to the reused frame currently from the current new optical flow information as the second optical flow information corresponding to the reused frame currently.

[0022] Optionally, the device further includes: a first update module, configured to, after the set determination module executes determining the current set of video frames to be aligned from the video to be processed according to the time sequence of video frames, execute updating the video frames recorded at each position in the current video frame queue in sequence according to the time sequence of each video frame in the current set of video frames to be aligned; wherein, each video frame in the current associated video frame set is recorded in the current video frame queue; for every two adjacent positions in the video frame queue, it corresponds to the position in the optical flow information queue for recording the optical flow information between the video frames recorded at these two positions, and the position in the alignment result queue for recording the alignment results of the video frames recorded at these two positions respectively;

[0023] a second update module, configured to, after the calculation module executes calculating the optical flow information between the current newly added frame and the previous video frame of the newly added frame in the current set of video frames to be aligned as the current newly added optical flow information, execute for every two adjacent video frames in time sequence in the current set of video frames to be aligned, if there is optical flow information between these two adjacent video frames in time sequence in the current optical flow information to be utilized, obtaining the optical flow information between these two adjacent video frames in time sequence from the current optical flow information to be utilized; otherwise, obtaining the optical flow information between these two adjacent video frames in time sequence from the current newly added optical flow information, to obtain the current updated optical flow information; updating the optical flow information recorded at each position in the current optical flow information queue in sequence according to the corresponding relationship between every two adjacent positions in the video frame queue and the positions in the optical flow information queue; wherein, the current optical flow information queue records the current optical flow information to be utilized;

[0024] a third update module, configured to, after the alignment module executes aligning the video frames in the current newly added frame except the current reference frame to the current reference frame based on the current optical flow information to be utilized and the newly added optical flow information, to obtain the alignment results of the video frames in the current newly added frame except the current reference frame, execute updating the alignment results of aligning the current associated frames to the current reference frame in sequence according to the corresponding relationship between every two adjacent positions in the video frame queue and the positions in the alignment result queue; wherein, the current alignment result queue records the alignment results of aligning each associated frame in the current associated video frame set to the reference frame.

[0025] Optionally, the alignment module is specifically configured to: for each current multiplexing frame, determine the third intermediate frame currently corresponding to the multiplexing frame from the current set of video frames to be aligned according to the time sequence of the video frames; wherein, the third intermediate frame currently corresponding to the multiplexing frame includes: the multiplexing frame, the current reference frame, and each video frame located between the multiplexing frame and the current reference frame; obtain the optical flow information between each two adjacent third intermediate frames currently corresponding to the multiplexing frame from the current optical flow information to be utilized as the third optical flow information currently corresponding to the multiplexing frame; calculate the optical flow information between the multiplexing frame and the current reference frame according to the third optical flow information currently corresponding to the multiplexing frame as the target optical flow information; and use the target optical flow information to align the multiplexing frame with the current reference frame to obtain the alignment result of the multiplexing frame.

[0026] Optionally, the alignment module is specifically configured to: if there is no current newly added frame in the third intermediate frame currently corresponding to the multiplexing frame, obtain the optical flow information between each two adjacent third intermediate frames currently corresponding to the multiplexing frame from the current optical flow information to be utilized as the third optical flow information currently corresponding to the multiplexing frame; if there is a current newly added frame in the third intermediate frame currently corresponding to the multiplexing frame and there are multiple current multiplexing frames, for each two adjacent third intermediate frames currently corresponding to the multiplexing frame, if there is optical flow information between the two adjacent third intermediate frames in the current optical flow information to be utilized, obtain the optical flow information between the two adjacent third intermediate frames from the current optical flow information to be utilized; otherwise, obtain the optical flow information between the two adjacent third intermediate frames from the current newly added optical flow information to obtain the third optical flow information currently corresponding to the multiplexing frame; if there is a current newly added frame in the third intermediate frame currently corresponding to the multiplexing frame and there is one current multiplexing frame, obtain the optical flow information between each two adjacent third intermediate frames currently corresponding to the multiplexing frame from the current newly added optical flow information as the third optical flow information currently corresponding to the multiplexing frame.

[0027] Optionally, the alignment module is specifically configured to: for each video frame in the current newly added frames except the current reference frame, determine the current corresponding fourth intermediate frame of the video frame from the current set of video frames to be aligned according to the time sequence of the video frames; wherein, the current corresponding fourth intermediate frame of the video frame includes: the video frame, the current reference frame, and each video frame located between the video frame and the current reference frame; obtain the optical flow information between every two adjacent fourth intermediate frames corresponding to the video frame in time sequence from the current optical flow information to be utilized and the newly added optical flow information, as the fourth optical flow information corresponding to the video frame; if the number of current reused frames in the current corresponding fourth intermediate frame of the video frame is not greater than 1, align the video frame with the fourth optical flow information corresponding to the video frame in sequence according to the time sequence of the video frames until the alignment result of the video frame to the current reference frame is obtained; if there are multiple current reused frames in the current corresponding fourth intermediate frame of the video frame, determine the current corresponding fifth intermediate frame of the video frame from the current corresponding fourth intermediate frame of the video frame according to the time sequence of the video frames; wherein, the current corresponding fifth intermediate frame of the video frame includes: the current reference frame, the current latest reused frame in time sequence; and each video frame located between the current reference frame and the current latest reused frame in time sequence; obtain the optical flow information between every two adjacent fifth intermediate frames corresponding to the video frame from the fourth optical flow information corresponding to the video frame, as the fifth optical flow information corresponding to the video frame; determine the current corresponding sixth intermediate frame of the video frame from the current corresponding fourth intermediate frame of the video frame according to the time sequence of the video frames; wherein, the current corresponding sixth intermediate frame of the video frame includes: the current latest reused frame in time sequence, the video frame, and each video frame located between the current latest reused frame in time sequence and the video frame; obtain the optical flow information between every two adjacent sixth intermediate frames corresponding to the video frame from the fourth optical flow information corresponding to the video frame, as the sixth optical flow information corresponding to the video frame; align the video frame with the current latest reused frame in time sequence with the sixth optical flow information corresponding to the video frame in sequence until the alignment result of the video frame to the current latest reused frame in time sequence is obtained, as the intermediate alignment result corresponding to the video frame; align the intermediate alignment result corresponding to the video frame with the fifth optical flow information corresponding to the video frame in sequence according to the time sequence of the video frames until the alignment result of the video frame to the current reference frame is obtained.

[0028] Optionally, the alignment module is specifically configured to: if the number of current multiplexed frames in the fourth intermediate frame currently corresponding to the video frame is no more than 1, obtain the optical flow information between every two adjacent fourth intermediate frames in time sequence currently corresponding to the video frame from the current newly added optical flow information as the fourth optical flow information currently corresponding to the video frame; if there are multiple current multiplexed frames in the fourth intermediate frame currently corresponding to the video frame, for every two adjacent fourth intermediate frames in time sequence currently corresponding to the video frame, if the optical flow information between the two adjacent fourth intermediate frames in time sequence exists in the current optical flow information to be utilized, obtain the optical flow information between the two adjacent fourth intermediate frames in time sequence from the current optical flow information to be utilized; otherwise, obtain the optical flow information between the two adjacent fourth intermediate frames in time sequence from the current newly added optical flow information to obtain the fourth optical flow information currently corresponding to the video frame.

[0029] Optionally, the alignment module is specifically configured to: for each video frame other than the current reference frame in the current newly added frames, determine the seventh intermediate frame currently corresponding to the video frame from the current set of video frames to be aligned according to the time sequence of the video frames; wherein, the seventh intermediate frame currently corresponding to the video frame includes: the video frame, the current reference frame, and each video frame located between the video frame and the current reference frame; obtain the optical flow information between every two adjacent seventh intermediate frames in time sequence currently corresponding to the video frame from the current optical flow information to be utilized and the newly added optical flow information as the seventh optical flow information currently corresponding to the video frame; calculate the optical flow information between the video frame and the current reference frame according to the seventh optical flow information currently corresponding to the video frame as the optical flow information to be utilized; use the optical flow information to be utilized to align the video frame to the current reference frame to obtain the alignment result of the video frame.

[0030] Optionally, the alignment module is specifically configured to: when the number of current multiplexed frames in the seventh intermediate frame currently corresponding to the video frame is no more than 1, obtain the optical flow information between every two adjacent seventh intermediate frames in time sequence currently corresponding to the video frame from the current newly added optical flow information as the seventh optical flow information currently corresponding to the video frame; when there are multiple current multiplexed frames in the seventh intermediate frame currently corresponding to the video frame, if the optical flow information between the two adjacent seventh intermediate frames in time sequence exists in the current optical flow information to be utilized, obtain the optical flow information between the two adjacent seventh intermediate frames in time sequence from the current optical flow information to be utilized; otherwise, obtain the optical flow information between the two adjacent seventh intermediate frames in time sequence from the current newly added optical flow information to obtain the seventh optical flow information currently corresponding to the video frame.

[0031] Optionally, the calculation module is specifically configured to: for each newly added frame currently, input the newly added frame and the previous video frame of the newly added frame in the current set of video frames to be aligned into a pre-trained optical flow prediction model, to obtain the newly added optical flow information between the newly added frame and the previous video frame of the newly added frame in the current set of video frames to be aligned; wherein, the optical flow prediction model is trained based on sample video frame pairs and sample labels representing the optical flow information between the sample video frame pairs.

[0032] Optionally, the number of associated frames of an associated frame in a set of video frames is not less than 3; in the case where the number of associated frames is an even number, among the associated frames included in the set of video frames, the number of frames of the associated frames earlier in time sequence than the reference frame in the set of video frames is the same as the number of frames of the associated frames later in time sequence than the reference frame in the set of video frames; in the case where the number of associated frames is an odd number, among the associated frames included in the set of video frames, the difference between the number of frames of the associated frames earlier in time sequence than the reference frame in the set of video frames and the number of frames of the associated frames later in time sequence than the reference frame in the set of video frames is 1.

[0033] In a third aspect of the embodiments of the present invention, there is provided an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory complete communication with each other through the communication bus; the memory is used for storing a computer program; the processor is used for implementing the video frame alignment method according to any one of the above first aspects when executing the program stored on the memory.

[0034] In a fourth aspect of the embodiments of the present invention, there is provided a computer-readable storage medium, in which a computer program is stored, and the computer program realizes the video frame alignment method according to any one of the above first aspects when being executed by a processor.

[0035] The embodiments of the present invention also provide a computer program product containing instructions, which when running on a computer, enables the computer to execute the video frame alignment method according to any one of the above.

[0036] The video frame alignment method provided by an embodiment of the present invention is such that an electronic device determines a current set of video frames to be aligned from a video to be processed according to the time sequence of video frames; a set of video frames includes a reference frame and associated frames that need to be aligned with the reference frame; obtaining the optical flow information between every two temporally adjacent video frames in the current set of associated video frames recorded in advance when aligning the current set of associated video frames, as the current optical flow information to be utilized; the current set of associated video frames is the set of video frames when multiple video frames in the current set of video frames to be aligned participated in alignment last time; determining, from the current set of video frames to be aligned, the video frames that do not belong to the current set of associated video frames as the current newly added frames; calculating the optical flow information between each current newly added frame and the previous video frame of the newly added frame in the current set of video frames to be aligned as the current newly added optical flow information; based on the current optical flow information to be utilized and the newly added optical flow information, aligning each current associated frame to the current reference frame respectively to obtain the alignment result of each current associated frame; returning to execute the step of determining the current set of video frames to be aligned from the video to be processed according to the time sequence of video frames.

[0037] Based on the above processing, after determining the current set of video frames to be aligned, the electronic device aligns the current associated frame with the current reference frame based on the optical flow information between every two temporally adjacent video frames in the current set of video frames to be aligned. Moreover, since there are multiple identical video frames between the current set of video frames to be aligned and the current set of associated video frames, when aligning the current set of associated video frames, the optical flow information between every two temporally adjacent video frames in the current set of associated video frames (i.e., the current optical flow information to be utilized) has been calculated and recorded. The current optical flow information to be utilized includes some of the optical flow information required for aligning the current set of video frames to be aligned. Therefore, the electronic device can directly obtain the current optical flow information to be utilized, that is, directly obtain some of the optical flow information required for aligning the current set of video frames to be aligned. Correspondingly, the electronic device only needs to newly calculate the optical flow information between each newly added frame and the previous video frame of the newly added frame in the current set of video frames to be aligned to obtain the current newly added optical flow information, that is, obtain the optical flow information required for aligning the current set of video frames to be aligned and not included in the current optical flow information to be utilized. Furthermore, the electronic device can obtain the alignment result of each associated frame based on the current newly added optical flow information and the directly obtained current optical flow information to be utilized. Compared with the prior art method of calculating the optical flow information between each associated frame and the reference frame when aligning multiple associated frames in a set of video frames with the reference frame, based on the video frame alignment method provided by the present invention, when aligning each associated frame in the current set of video frames to be aligned with the current reference frame, the current optical flow information to be utilized can be reused, that is, the previously calculated optical flow information can be reused, reducing the computing power resources consumed when calculating new optical flow information, that is, reducing the consumption of the computing power resources of the electronic device.

[0038] Of course, it is not necessary for any product or method implementing the present invention to achieve all of the above advantages simultaneously. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention, and those of ordinary skill in the art can also obtain other embodiments based on these drawings.

[0040] Figure 1 It is a flowchart of a video frame alignment method provided by an embodiment of the present invention;

[0041] Figure 2 It is a schematic diagram of a video frame alignment method provided by an embodiment of the present invention;

[0042] Figure 3 A structural diagram of the video frame alignment device provided by an embodiment of the present invention;

[0043] Figure 4 A structural diagram of the electronic device provided by an embodiment of the present invention. Specific embodiments

[0044] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art based on the present invention belong to the scope of protection of the present invention.

[0045] In the field of image processing technology, video frames in a video can be processed (which can also be called video processing). Video frame alignment can be applied to a variety of processing tasks. For example, video image enhancement is an important application field of video processing; generally, video image enhancement can include temporal enhancement and spatial enhancement. Spatial enhancement refers to using the image information included in a single video frame to perform processing such as detail restoration and repair on the video frame, such as filtering the video frame. Temporal enhancement refers to aligning pixels in multiple video frames and then fusing the aligned pixels. For example, using the image information of multiple associated frames of a reference frame to perform detail repair, denoising, etc. on the reference frame. In the prior art, when performing video processing, it is necessary to first determine a video frame to be temporally enhanced (i.e., the reference frame in the subsequent embodiments) from the video according to the time sequence of the video frames, and then determine multiple associated frames to be used when performing temporal enhancement on the reference frame. Align the multiple associated frames corresponding to the reference frame to the reference frame respectively. For example, for each associated frame corresponding to the reference frame, calculate the optical flow information between the associated frame and the reference frame using the optical flow method, and then align the associated frame to the reference frame based on the optical flow information, that is, obtain the alignment result of aligning the associated frame to the reference frame. Furthermore, use the multiple alignment results corresponding to the reference frame to perform temporal enhancement on the reference frame. After obtaining the enhancement result of performing temporal enhancement on the reference frame, the electronic device can continue to determine the next video frame to be temporally enhanced from the video according to the time sequence of the video frames, and so on. It can be seen that aligning and fusing pixels in multiple frames of images in the time dimension is one of the core steps of the image enhancement algorithm.

[0046] Obviously, on the premise that the alignment result is accurate, the more the number of associated frames aligned to the reference frame, the better the enhancement effect on the reference frame. That is, theoretically, the enhancement effect is directly proportional to the number of frames of the associated frames (which can be called the number of frames to be fused). However, the higher the number of frames to be fused, the more computing power resources are consumed when aligning each associated frame to the reference frame. That is, the higher the requirement for the computing power resources of the electronic device. In the case where the computing power resources of the electronic device are limited, only a relatively small number of associated frames can be aligned to the reference frame, resulting in a poor enhancement effect on the reference frame finally. Therefore, when aligning multiple associated frames with the reference frame, how to reduce the consumption of the computing power resources of the electronic device has become a technical problem to be solved urgently.

[0047] To solve the above technical problem, the present invention provides a video frame alignment method applied to an electronic device by using the characteristic that the alignment relationship has transitivity. The electronic device can be a server providing video frame alignment services, or can also be a client used by a user. The present invention does not limit this. The video frame alignment method includes: determining a current set of video frames to be aligned from the video to be processed according to the time sequence of the video frames; a set of video frames includes a reference frame and associated frames that need to be aligned to the reference frame; obtaining the optical flow information between every two adjacent video frames in the current set of associated video frames recorded in advance when aligning the current set of associated video frames, as the current optical flow information to be utilized; the current set of associated video frames is the set of video frames when multiple video frames in the current set of video frames to be aligned participated in alignment last time; determining the video frames that do not belong to the current set of associated video frames from the current set of video frames to be aligned as the current new frames; calculating the optical flow information between each current new frame and the previous video frame of the new frame in the current set of video frames to be aligned as the current new optical flow information; based on the current optical flow information to be utilized and the new optical flow information, aligning each current associated frame to the current reference frame respectively to obtain the alignment result of each current associated frame; returning to execute the step of determining a current set of video frames to be aligned from the video to be processed according to the time sequence of the video frames. That is, when aligning each associated frame in the current set of video frames to be aligned to the current reference frame, the current optical flow information to be utilized can be reused, that is, the optical flow information calculated historically can be reused, reducing the computing power resources consumed when calculating new optical flow information, that is, the consumption of the computing power resources of the electronic device can be reduced.

[0048] See Figure 1 , Figure 1 is a flowchart of a video frame alignment method provided by an embodiment of the present invention. The method may include the following steps:

[0049] S101: Determine a current set of video frames to be aligned from the video to be processed according to the time sequence of the video frames.

[0050] Among them, a set of video frames includes a reference frame and associated frames that need to be aligned with the reference frame.

[0051] S102: Obtain the optical flow information between every two temporally adjacent video frames in the set of associated video frames obtained when aligning the current set of associated video frames, which is pre-recorded, as the current optical flow information to be utilized. Among them, the current set of associated video frames is the set of video frames when the multiple video frames in the current set of video frames to be aligned participated in alignment last time.

[0052] S103: Determine the video frames that do not belong to the current set of associated video frames from the current set of video frames to be aligned as the current newly added frames.

[0053] S104: Calculate the optical flow information between each current newly added frame and the previous video frame of this newly added frame in the current set of video frames to be aligned as the current newly added optical flow information.

[0054] S105: Based on the current optical flow information to be utilized and the newly added optical flow information, align each current associated frame with the current reference frame respectively to obtain the alignment result of each current associated frame; return to the step of determining the current set of video frames to be aligned from the video to be processed according to the timing of the video frames.

[0055] Based on the video frame alignment method provided by the present invention, after determining the current set of video frames to be aligned, the electronic device aligns the current associated frame towards the current reference frame based on the optical flow information between every two temporally adjacent video frames in the current set of video frames to be aligned. Moreover, since there are multiple identical video frames between the current set of video frames to be aligned and the current set of associated video frames, when aligning the current set of associated video frames, the optical flow information between every two temporally adjacent video frames in the current set of associated video frames (i.e., the current optical flow information to be utilized) has been calculated and recorded. The current optical flow information to be utilized includes some of the optical flow information required for aligning the current set of video frames to be aligned. Therefore, the electronic device can directly obtain the current optical flow information to be utilized, that is, directly obtain some of the optical flow information required for aligning the current set of video frames to be aligned. Correspondingly, the electronic device only needs to newly calculate the optical flow information between each newly added frame and the previous video frame of the newly added frame in the current set of video frames to be aligned, to obtain the current newly added optical flow information, that is, to obtain the optical flow information required for aligning the current set of video frames to be aligned and not included in the current optical flow information to be utilized. Furthermore, the electronic device can obtain the alignment result of each associated frame based on the current newly added optical flow information and the directly obtained current optical flow information to be utilized. Compared with the prior art, when aligning multiple associated frames in a set of video frames towards a reference frame, the method of calculating the optical flow information between each associated frame and the reference frame is required. Based on the video frame alignment method provided by the present invention, when aligning each associated frame in the current set of video frames to be aligned towards the current reference frame, the current optical flow information to be utilized can be reused, that is, the previously calculated optical flow information can be reused, reducing the computing power resources required for calculating new optical flow information, that is, reducing the consumption of the computing power resources of the electronic device.

[0056] Due to the extremely complex situations in the actual scenario, for ease of understanding, the present invention first gives an embodiment of the process of video frame alignment in the actual scenario. It can be understood that this embodiment only includes some of the possible situations in the actual scenario and does not limit the specific situations that occur in the actual scenario of the present invention.

[0057] See Figure 2 , Figure 2A schematic diagram of the video frame alignment method provided by an embodiment of the present invention. In this embodiment, a video frame set includes 5 consecutive video frames in the video to be processed; according to the time sequence of the video frames, the reference frame in this video frame set is the 3rd video frame in this video frame set. After the electronic device obtains the alignment result of aligning the associated frame (hereinafter simply referred to as the current associated frame) in the current video frame set to be aligned to the current reference frame, it uses the next video frame of the current reference frame in the video to be processed as the current reference frame. t x represents the video frames in the video to be processed, x represents the time sequence of the video frames. For example, t0 represents the 1st video frame in the video to be processed, t1 represents the 2nd video frame in the video to be processed, t2 represents the 3rd video frame in the video to be processed, and so on. represents t A to t B of the optical flow information, including the pixel position change of each pixel point in t A at t B in. represents using to align t A to t B of the alignment result. For each video frame set, the process of aligning the associated frame in this video frame set to the reference frame in this video frame set can be called aligning the reference frame in this video frame set. The electronic device determines video frame set 1 from the video to be processed at the 1st moment and aligns the reference frame in video frame set 1; determines video frame set 2 from the video to be processed at the 2nd moment and aligns the reference frame in video frame set 2; determines video frame set 3 from the video to be processed at the 3rd moment and aligns the reference frame in video frame set 3, and so on.

[0058] At the 1st moment, the electronic device determines video frame set 1, obtains the current video frame set to be aligned, denoted as [t0, t1, t2, t3, t4].

[0059] Calculate

[0060] Obtain and Output That is, output the alignment result of aligning the reference frame in video frame set 1.

[0061] At the 2nd moment, the new video frame t5 is queued, video frame set 2 is determined, and the current video frame set to be aligned is obtained, denoted as [t1, t2, t3, t4, t5]. Based on what was obtained at the 1st moment and Calculate

[0062] obtain and output That is, output the alignment result of aligning the reference frame in the video frame set 2.

[0063] At the 3rd moment, a new video frame t6 is enqueued, and the video frame set 3 is determined to obtain the current video frame set to be aligned, denoted as [t2, t3, t4, t5, t6]. Based on what is obtained at the 2nd moment and calculate

[0064] obtain and output That is, output the alignment result of aligning the reference frame in the video frame set 3. And so on, until the alignment result of aligning the associated frame of each video frame in the video to be processed to this video frame is obtained, that is, the multi-frame video frame alignment of the video to be processed is completed. Subsequently, based on the alignment result corresponding to each video frame, image enhancement processing can be performed on this video frame. In this way, the multi-frame image enhancement of the video to be processed is realized.

[0065] Regarding step S101 and step S102, the arrangement order of each video frame in a video, that is, the time sequence of the video frames; the earlier the position of a video frame is in the video frames included in this video, the earlier the time sequence of this video frame; the later the position of this video frame is in the video frames included in this video, the later the time sequence of this video frame. As based on the foregoing Figure 2 illustrated embodiment, the time sequence of t2 is earlier than that of t3, and the time sequences of t2 and t3 are adjacent.

[0066] After obtaining the alignment result of aligning the reference frame in a video frame set, the electronic device can continue to determine the reference frame from the video to be processed, and then determine the associated frame of this reference frame to determine a video frame set. Or, the electronic device can also first determine the video frame set, and then determine the reference frame from each video frame included in this video frame set. The present invention does not limit this.

[0067] In an actual scenario, a set of video frames can be jointly determined from the video to be processed based on service requirements and the performance of the electronic device. For ease of description and understanding, in the following of the present invention, an example will be given in which the electronic device first determines a reference frame and then determines the associated frames of the reference frame. For each set of video frames, in the following of the present invention, an example will be given in which image enhancement is performed on the reference frames in the set of video frames based on the alignment result of aligning the reference frames in the set of video frames. In an actual scenario, video frame alignment can be applied to various processing tasks and is not limited to image enhancement. Exemplarily, the interval between the currently determined reference frame of the electronic device and the previously determined reference frame can be referred to as the reference frame interval. As in the foregoing Figure 2 In the embodiment shown, the reference frame interval is 0, that is, after the electronic device aligns the current reference frame, the next video frame of the current reference frame in the video to be processed is used as the current reference frame, and there are no other video frames between the current reference frame and the previously determined reference frame. The larger the reference frame interval, the fewer the number of reference frames determined when performing video enhancement on the video to be processed. Correspondingly, the fewer the number of sets of video frames determined from the video to be processed; then, in the process of aligning the associated frames of each reference frame in the video to be processed with the reference frame, the total computing power resources consumed are less, and the time taken to obtain the enhanced video is shorter; the smaller the reference frame interval, the more the number of reference frames determined when performing video enhancement on the video to be processed, and then in the finally obtained enhanced video, the more the number of video frames subjected to image enhancement, that is, the higher the video quality of the enhanced video. Therefore, when the service requirements indicate that a higher-quality enhanced video needs to be obtained and the performance of the electronic device is good, a smaller reference frame interval can be set, such as 0. When the service requirements indicate that the time taken to obtain the enhanced video needs to be short, a larger reference frame interval can be set, such as 3.

[0068] Furthermore, each time the electronic device determines the current reference frame, it can determine the current associated frame. Exemplarily, the video frames between the reference frame and the 6th video frame after the reference frame in the video to be processed can be used as the associated frames. Alternatively, the video frames between the reference frame and the 5th video frame before the reference frame in the video to be processed can be used as the associated frames. Alternatively, the previous video frame of the reference frame in the video to be processed and the video frames between the reference frame and the 4th video frame after the reference frame can both be used as the associated frames. Alternatively, the 1st, 3rd, and 5th video frames before the reference frame in the video to be processed and the 1st, 3rd, and 5th video frames after the reference frame can both be used as the associated frames of the reference frame. Obviously, there are far more specific ways to determine the associated frames in the actual scenario than this, and the present invention does not limit this. The more associated frames there are for each reference frame, the higher the quality of the enhanced reference frame obtained by enhancing the reference frame based on the alignment result, and thus the higher the video quality of the subsequent enhanced video. The fewer the associated frames of the reference frame, the less computing power resources are consumed when obtaining the alignment results of each associated frame of the reference frame aligning to the reference frame, the shorter the time taken to obtain the alignment results, and thus the shorter the time taken to obtain the enhanced video. Therefore, when the service requirement indicates that an enhanced video with higher quality needs to be obtained and the performance of the electronic device is good, the number of associated frames for each reference frame can be set to be relatively large, such as 8. When the service requirement indicates that the time taken to obtain the enhanced video needs to be short, the number of associated frames for each reference frame can be set to be relatively small, such as 4. Exemplarily, in order to obtain an enhanced video with higher quality and reduce the time taken to obtain the enhanced video, the electronic device can determine the video frame set from the video to be processed in the manner corresponding to the foregoing Figure 2 embodiment shown, that is, the reference frame interval is 0, and each reference frame has 4 associated frames, and determine the video frame set from the video to be processed in sequence.

[0069] It can be seen that the way to determine the video frame set in the actual scenario is set according to the requirements. Therefore, the present invention does not limit the specific way to obtain the video frame set, that is, it does not limit the specific number of video frames in each video frame set, the time sequence of the reference frames in the video frame set, nor the specific number of video frame sets determined from the video to be processed. For ease of description, the "current video frame set to be aligned" will be used as an example for introduction later. It can be understood that the current video frame set to be aligned can be any video frame set determined from the video to be processed.

[0070] According to the video frame alignment method provided by the present invention, after the electronic device determines the current reference frame, it determines the associated frames (i.e., the current associated frames) that need to be aligned with the current reference frame, so as to obtain the current set of video frames to be aligned. Subsequently, after obtaining the alignment results of each current associated frame with the current reference frame, the electronic device continues to determine the next reference frame from the video to be processed according to the timing of the video frames as the current reference frame, and then determines the associated frames that need to be aligned with the current reference frame to obtain the current set of video frames to be aligned. And so on in a cycle.

[0071] In the present invention, since there are multiple identical video frames between the set of video frames (i.e., the current associated video frame set) when multiple video frames in the current set of video frames to be aligned participated in alignment last time and the current set of video frames to be aligned, when performing video frame alignment according to the video frame alignment method provided by the present invention, it is possible to reuse the optical flow information (i.e., the current optical flow information to be utilized) between every two adjacent video frames in the associated video frame set obtained when aligning the current associated video frame set, thereby reducing the computing power resources consumed when calculating new optical flow information, that is, reducing the consumption of the computing power resources of the electronic device. For the specific manner in which the electronic device calculates the optical flow information between two video frames, refer to the detailed introduction in the subsequent embodiments.

[0072] Exemplarily, based on the foregoing Figure 2 illustrated embodiment, at the first moment, the optical flow information that the electronic device needs to calculate includes that is, newly calculating 4 optical flow information. At the second moment, the electronic device only needs to calculate the negative value of, and calculate that is, calculating 1 negative value and newly calculating 1 optical flow information. At the third moment, the electronic device only needs to calculate the negative value of, and calculate that is, calculating 1 negative value and newly calculating 1 optical flow information. And so on. Each time the electronic device determines the current set of video frames to be aligned and aligns the current associated frames with the current reference frame, it only needs to calculate 1 negative value and newly calculate 1 optical flow information to obtain the alignment results of the current 4 associated frames with the current reference frame, that is, newly calculating 1 optical flow information to complete the alignment of 5 video frames.

[0073] In the same scenario, based on the prior art, at the first moment, the optical flow information that the electronic device needs to calculate includes that is, newly calculating 4 optical flow information. At the second moment, the electronic device needs to calculate the negative value of, and calculate to obtain that is, it needs to calculate 1 negative value and newly calculate 3 optical flow information. At the third moment, the electronic device needs to calculate a negative value, and calculating to obtain That is, it is necessary to calculate the negative value once and newly calculate three optical flow information items.

[0074] And so on. Based on the prior art, every time the electronic device determines the current set of video frames to be aligned and aligns the current associated frame with the current reference frame, it is necessary to calculate the negative value once and newly calculate three optical flow information items.

[0075] It can be seen that compared with the prior art, based on the video frame alignment method provided by the present invention, only the negative value needs to be calculated once and only one optical flow information item needs to be newly calculated, and the calculation amount of two newly calculated optical flow information items can be reduced in each alignment process, that is, the overall calculation amount of the video frame alignment process can be reduced, and the consumption of the computing power resources of the electronic device can be reduced.

[0076] Since the current set of associated video frames is the set of video frames when multiple video frames in the current set of video frames to be aligned participated in the previous alignment, that is, there must be multiple identical video frames between the current set of associated video frames and the current set of video frames to be aligned. However, not all multiple video frames in the current set of video frames to be aligned necessarily participated in the previous video frame alignment, that is, the set of video frames for the previous video frame alignment is not necessarily the current set of associated video frames.

[0077] Exemplarily, in the foregoing Figure 2 shown embodiment, the set of video frames for the previous alignment is the current set of associated video frames. For example, at the second moment, [t1, t2, t3, t4] all participated in the video frame alignment at the first moment; at the third moment, [t2, t3, t4, t5] all participated in the video frame alignment at the second moment.

[0078] Exemplarily, if at the first moment, the electronic device determines the video frame set A, denoted as [t0, t2, t4, t6, t8], and t4 is the reference frame in the video frame set A. Then in a manner similar to the foregoing Figure 2 shown embodiment, the electronic device calculates the optical flow information for using to align [t0, t2, t6, t8] with t4 respectively. At the second moment, the electronic device determines the video frame set B, denoted as [t1, t3, t5, t7, t9], and t5 is the reference frame in the video frame set B. Then in a manner similar to the foregoing Figure 2 shown embodiment, the electronic device calculates the optical flow information for using Align [t1, t3, t7, t9] to t5 respectively. Obviously, the video frames in video frame set B do not participate in the video frame alignment at the first moment, so video frame set A is not the associated video frame set corresponding to video frame set B. At the third moment, the electronic device determines video frame set C, [t2, t4, t6, t8, t 10 , and t6 is the reference frame in video frame set C. Then, in a manner similar to the foregoing Figure 2 illustrated embodiment, the electronic device calculates the optical flow information for use to align [t2, t4, t8, t 10 to t6 respectively. Since multiple video frames (i.e., [t2, t4, t6, t8]) in video frame set C participate in the video frame alignment at the first moment, video frame set A is the associated video frame set corresponding to video frame set C. During the process of aligning the associated frames in video frame set C to t6, the optical flow information calculated during the process of aligning the associated frames in video frame set A to t4 can be used Then, at the third moment, there is no need to recalculate this optical flow information, which can reduce the consumption of computing power resources of the electronic device.

[0079] Regarding step S103, the video frames in the current video frame set to be aligned that do not belong to the current associated video frame set, that is, the video frames newly added to the current video frame set to be aligned relative to the current associated video frame set, can be called the current newly added frames. In an actual scenario, each current newly added frame may be a current associated frame or a current reference frame. The current newly added frames may be one or multiple.

[0080] Exemplarily, as in the foregoing Figure 2 illustrated embodiment, the current newly added frame is the current associated frame and there is only one. For example, at the second moment, t5 is the current newly added frame; at the third moment, t6 is the current newly added frame.

[0081] Exemplarily, if the video frame set at the first moment is [t0, t1, t2, t3, t4] and the reference frame is t4; the video frame set at the second moment is [t2, t3, t4, t5, t6] and the reference frame is t6, then there are multiple current newly added frames (i.e., t5 and t6), and t5 is the current associated frame and t6 is the current reference frame. This embodiment is hereinafter referred to as Example A for short.

[0082] In some embodiments, after the electronic device determines a set of video frames each time, it can record the set of video frames determined this time. After determining the current set of video frames to be aligned, the electronic device sequentially compares each historically recorded set of video frames with the set of video frames recorded this time (i.e., the current set of video frames to be aligned) in the order from the latest to the earliest recorded time until it determines a set of video frames that has multiple identical video frames with the current set of video frames to be aligned, which is used as the current associated set of video frames. Then, the electronic device compares the current set of video frames to be aligned with the current associated set of video frames to determine the video frames newly added to the current set of video frames to be aligned compared to the current associated set of video frames, that is, the current newly added frames.

[0083] In some embodiments, the electronic device can record the latest determined set of video frames in a video frame queue. For the specific recording method, please refer to the detailed description in the subsequent embodiments. The video frame queue can be a FIFO (First-In-First-Out) queue, and the newly enqueued video frame in the video frame queue is the current newly added frame.

[0084] Regarding step S104, since the current associated set of video frames does not include the current newly added frames, the current optical flow information to be utilized also does not include the optical flow information between each current newly added frame and the previous video frame of this newly added frame in the current set of video frames to be aligned. Therefore, after the electronic device determines the current newly added frames, it needs to calculate the optical flow information between each current newly added frame and the previous video frame of this newly added frame in the current set of video frames to be aligned to obtain the current newly added optical flow information. Exemplarily, as in the foregoing Figure 2 embodiment shown, at the second moment, the current newly added frame is t5, and the previous video frame of t5 in the current set of video frames to be aligned (i.e., in video frame set 2) is t4. The optical flow information obtained at the first moment does not include Therefore, the electronic device needs to calculate to obtain the current newly added optical flow information. The electronic device can calculate the optical flow information between two video frames according to a preset optical flow calculation method. For example, the preset optical flow calculation method can include: LK (Lucas-Kanade) pyramid algorithm, Dis Flow (turbulent flow) algorithm, HS (Horn–Schunck) algorithm, etc. The present invention does not limit this.

[0085] Regarding step S105, the current optical flow information to be utilized and the newly added optical flow information are all the optical flow information required to align the current associated frames to the current reference frame. Therefore, based on the current optical flow information to be utilized and the newly added optical flow information, the electronic device can align the current associated frames to the current reference frame to obtain the alignment results of the current associated frames. Then, continue to re-determine the current set of video frames to be aligned from the video to be processed according to the time sequence of the video frames until, according to the service requirements, the associated frames of each reference frame that needs to be aligned in the video to be processed are aligned to the reference frame. As based on the foregoing Figure 2 shown in the embodiment, when the electronic device obtains at the first moment, it can use the next video frame of t2 (i.e., t3) in the video to be processed as the current reference frame, and then determine [t1, t2, t4, t5] as the current associated frames, that is, re-determine the current set of video frames to be aligned (i.e., video frame set 2) from the video to be processed.

[0086] The foregoing optical flow information can also be denoted as where R represents an integer, w represents the width of each video frame in the video to be processed, h represents the height of the video frame, and 2 represents the dimension of pixel offset; an element in the matrix represents a pixel point in t A to a pixel point in t B optical flow information. When the length of the side parallel to the horizontal direction of the video frame is the width of the video frame, the x-axis is constructed along the horizontal direction and the y-axis is constructed along the vertical direction to construct the pixel coordinates, then the pixel offset includes the offset on the x-axis and the offset on the y-axis.

[0087] In the case where t A is a single-channel image, t A can be denoted as t A ∈R w × h × 2 ; an element in the matrix represents the pixel value of a pixel point in t A . The optical flow information of the pixel point with pixel coordinates (i, j) in t A is (x, y), indicating that the pixel point moves to the pixel coordinates (i + x, j + y) in t B . The electronic device obtains the pixel value of the pixel point at the pixel coordinates (i + x, j + y) in t B , that is, obtains the pixel resampling result of this pixel point in the video frame t using A . The pixel resampling results of each pixel point in t A , that is, the alignment result of t A to t B . Exemplarily, tA To t B Aligning can be denoted as Among them, Indicates the offset of the pixel point in t A On the x-axis; Indicates the offset of the pixel point in t A On the y-axis.

[0088] In some embodiments, the number of associated frames of the associated frames in a video frame set is not less than 3; when the number of associated frames is even, among the associated frames included in the video frame set, the number of frames of the associated frames whose time sequence is earlier than the reference frame in the video frame set is the same as the number of frames of the associated frames whose time sequence is later than the reference frame in the video frame set; when the number of associated frames is odd, among the associated frames included in the video frame set, the difference between the number of frames of the associated frames whose time sequence is earlier than the reference frame in the video frame set and the number of frames of the associated frames whose time sequence is later than the reference frame in the video frame set is 1.

[0089] Since the number of associated frames of the associated frames of a reference frame can be set as required, that is, the number of associated frames in the actual scenario may be even or odd. And it can be understood that when the time sequence of the reference frame in a video frame set is in the middle of multiple associated frames, that is, the multiple associated frames of the reference frame are all relatively close to the reference frame, the error between the multiple associated frames and the reference frame is small. Based on the alignment result of the associated frames with small error to the reference frame, image enhancement is performed on the reference frame, and the quality of the enhanced video frame obtained is high, and the quality of the enhanced video obtained is also high. Therefore, in order to improve the quality of the subsequent enhanced video, the number of associated frames whose time sequence is earlier than the current reference frame can be set to be closest to the number of associated frames whose time sequence is later than the current reference frame, that is, the current reference frame is located in the middle of the current video frame set to be aligned. That is, when the total number of current associated frames (i.e., the number of associated frames) is even, the number of frames of the associated frames included in the video frame set whose time sequence is earlier than the reference frame in the video frame set is the same as the number of frames of the associated frames whose time sequence is later than the reference frame in the video frame set; when the number of associated frames is odd, the difference between the number of frames of the associated frames whose time sequence is earlier than the reference frame in the video frame set and the number of frames of the associated frames whose time sequence is later than the reference frame in the video frame set is 1.

[0090] Exemplarily, as in the foregoing Figure 2 As shown in the embodiment, a video frame set includes 5 consecutive video frames in the video to be processed, that is, the number of associated frames in this embodiment is 4. According to the time sequence of the video frames, the reference frame in the video frame set is the 3rd video frame in the video frame set, that is, the reference frame in the video frame set is located in the middle of the video frame set; in the video frame set, the number of frames of the associated frames whose time sequence is earlier than the reference frame and the number of frames of the associated frames whose time sequence is later than the reference frame are both 2.

[0091] Exemplarily, if the current set of video frames to be aligned is denoted as [t N , t N+1 , t N+2 , t N+3 , that is, in this embodiment, the number of associated frames is 3. According to the time sequence of the video frames, the reference frame in this set of video frames is the second video frame in this set of video frames (i.e., t N+1 ), or it can also be the third video frame (i.e., t N+2 ), such that the difference between the number of associated frames whose time sequence is earlier than the reference frame in this set of video frames and the number of associated frames whose time sequence is later than the reference frame in this set of video frames is 1.

[0092] In some embodiments, step S104 described above may include the following steps: For each newly added frame currently, input the newly added frame and the previous video frame of the newly added frame in the current set of video frames to be aligned into a pre-trained optical flow prediction model, to obtain the new optical flow information between the newly added frame and the previous video frame of the newly added frame in the current set of video frames to be aligned. Among them, the optical flow prediction model is: trained based on sample video frame pairs and sample labels representing the optical flow information between the sample video frame pairs.

[0093] The optical flow prediction model can be constructed based on a CNN (Convolutional Neural Network). For example, a person skilled in the art can obtain multiple sample video frame pairs and sample labels representing the optical flow information between the sample video frame pairs. For each sample video frame pair, input the sample video frame pair into the optical flow prediction model with an initial structure, and obtain the optical flow information predicted by the optical flow prediction model between the sample video frame pairs as the prediction result. Then, calculate the loss function value representing the difference between the prediction result and the sample label based on a preset loss function, and then adjust the model parameters of the optical flow prediction model with the initial structure based on the loss function value until the model converges. For example, the preset loss function can be an MSE (Mean Squared Error Loss) function, a cross-entropy loss function, etc., and the present invention does not limit this.

[0094] Based on the above processing, compared with traditional optical flow calculation methods, such as the aforementioned LK pyramid algorithm, HS algorithm, etc., the electronic device can obtain the optical flow information between every two adjacent video frames in time sequence based on the pre-trained optical flow prediction model through convolutional calculation, improving the efficiency of obtaining optical flow information and the accuracy of the obtained optical flow information.

[0095] As described above, in an actual scenario, based on service requirements and the performance of the electronic device, a set of video frames is jointly determined from the video to be processed, which may lead to various different situations in the actual scenario. For example, the current reused frame may be one or multiple; the current newly added frames may be one or multiple; the current newly added frame may be the current reference frame, and the timing of the current newly added frame may be later or earlier than the current reference frame; among the current associated frames, the video frames belonging to the current associated video frame set (i.e., the current reused frames in subsequent embodiments) may be the reference frames in the current associated video frame set (hereinafter simply referred to as associated reference frames), and the timing of the current reused frame may be earlier or later than the associated reference frame; the current reused frame may also be the current reference frame.

[0096] Exemplarily, as described above Figure 2 In the embodiment shown, there is one current newly added frame, and its timing is later than the current reference frame; there are multiple current reused frames, and among the current reused frames, there are video frames whose timing is earlier than the associated reference frame, there is an associated reference frame, and there are also video frames whose timing is later than the associated reference frame.

[0097] Exemplarily, as in the foregoing Example A, there are multiple current newly added frames (i.e., t5 and t6), the timing of t5 is earlier than the current reference frame, and t6 is the current reference frame; there are multiple current reused frames (i.e., t2, t3, t4), t4 is the associated reference frame, and the timings of t2 and t3 are both earlier than the associated reference frame.

[0098] Obviously, not all possible situations will necessarily occur in a scenario. In the present invention, only the processing methods in all possible situations that may occur in the actual scenario will be introduced later. In the actual scenario, according to the determination methods of the reference frame and the associated frame, the situations that may occur in this scenario can be determined, and then only the processing methods in the possible situations are used to perform video frame alignment.

[0099] In some embodiments, the above step S105 may include the following steps:

[0100] Step A: Determine the video frames belonging to the current associated video frame set from the current associated frames as the current reused frames.

[0101] Step B: Align the current reused frames to the current reference frame based on the current available optical flow information to obtain the alignment result of the current reused frames.

[0102] Step C: Align the video frames other than the current reference frame in the current newly added frames to the current reference frame based on the current available optical flow information and the newly added optical flow information to obtain the alignment result of the video frames other than the current reference frame in the current newly added frames.

[0103] When the electronic device can record the currently determined set of video frames every time a set of video frames is determined, the current associated frame can be compared with the current associated set of video frames to determine the video frames that are repeated between the current associated frame and the current associated set of video frames as the current reused frames. When the electronic device records the most recently determined set of video frames in the video frame queue, the other video frames in the currently recorded video frame queue except for the newly enqueued video frames and except for the current reference frame can be used as the current reused frames. Then, the electronic device can determine the optical flow information required to align the current reused frames with the current reference frame based on the current optical flow information to be utilized (which can be abbreviated as the optical flow required for the reused frames), and then align the current reused frames with the current reference frame based on the optical flow required for the reused frames to obtain the alignment result of the current reused frames. Obviously, one optical flow information corresponds to two video frames, that is, the optical flow information corresponds to the time period between the respective timings of the two video frames. According to the timing of the video frames, the time period between every two adjacent video frames in time sequence can be determined, and there is a sequence among the determined multiple time periods. The sequence of each time period can indicate the usage sequence of the optical flow information corresponding to each time period. For example, when the timing of a current associated frame is earlier than the current reference frame, the optical flow information to be used can be sequentially used in the order from the earliest to the latest of the time periods corresponding to the optical flow information to be used; when the timing of a current associated frame is later than the current reference frame, the optical flow information to be used can be sequentially used in the order from the latest to the earliest of the time periods corresponding to the optical flow information to be used.

[0104] Exemplarily, based on the foregoing Example A, at the second moment, the time period between the respective moments of t2 and t3 can be denoted as time period 1; the time period between the respective moments of t3 and t4 can be denoted as time period 2; the time period between the respective moments of t4 and t5 can be denoted as time period 3; the time period between the respective moments of t5 and t6 can be denoted as time period 4. Obviously, the sequence among the determined time periods is: time period 1 is earlier than time period 2, time period 2 is earlier than time period 3, and time period 3 is earlier than time period 4. For the current reused frame t2, when aligning t2 with t6, it is necessary to sequentially use

[0105] In some embodiments, step B above may include the following steps: determining a current first intermediate frame from the current set of video frames to be aligned according to the timing sequence of the video frames; wherein the current first intermediate frame includes: a reference frame in the current associated set of video frames, the current reference frame, and each video frame located between the reference frame in the current associated set of video frames and the current reference frame; obtaining, from the current optical flow information to be utilized, the optical flow information between every two adjacent first intermediate frames in time sequence as the current first optical flow information; for each current multiplexing frame, when the timing of the multiplexing frame is earlier than the reference frame in the current associated set of video frames, obtaining the alignment result of the multiplexing frame to the reference frame in the current associated set of video frames obtained when aligning the current associated set of video frames previously recorded as the current alignment result to be utilized; aligning the current alignment result to be utilized to the current reference frame in sequence according to the timing sequence of the video frames using the current first optical flow information until the alignment result of the multiplexing frame to the current reference frame is obtained; when the multiplexing frame is the reference frame in the current associated set of video frames, aligning the multiplexing frame to the current reference frame in sequence according to the timing sequence of the video frames using the current first optical flow information to obtain the alignment result of the multiplexing frame; when the timing of the multiplexing frame is later than the reference frame in the current associated set of video frames and earlier than the current reference frame, determining the second intermediate frame currently corresponding to the multiplexing frame from the current set of video frames to be aligned according to the timing sequence of the video frames; wherein the second intermediate frame currently corresponding to the multiplexing frame includes: the multiplexing frame, the current reference frame, and each video frame located between the multiplexing frame and the current reference frame; obtaining, from the current optical flow information to be utilized, the optical flow information between every two adjacent second intermediate frames in time sequence corresponding to the multiplexing frame as the second optical flow information corresponding to the multiplexing frame; aligning the multiplexing frame to the current reference frame in sequence according to the timing sequence of the video frames using the second optical flow information corresponding to the multiplexing frame to obtain the alignment result of the multiplexing frame; when the timing of the multiplexing frame is later than the current reference frame, obtaining the alignment result of the multiplexing frame to the current reference frame obtained when aligning the current associated set of video frames previously recorded.

[0106] When there is a video frame in the current multiplexing frame whose timing is not later than the associated reference frame, the current set of video frames to be aligned must include the associated reference frame. After obtaining the current set of video frames to be aligned, the electronic device may determine the associated reference frame, the current reference frame, and each video frame located between the associated reference frame and the current reference frame from the current set of video frames to be aligned according to the timing sequence of the video frames to obtain the current first intermediate frame.

[0107] Exemplarily, as described above Figure 2In the illustrated embodiment, at the second moment, the current first intermediate frames include: t2, t3, and there is no current newly added frame.

[0108] Exemplarily, if the current set of video frames to be aligned is [t3, t4, t5, t6, t7, t8, t9], the current reference frame is t8; the current associated set of video frames is [t1, t2, t3, t4, t5, t6, t7]; the associated reference frame is t6. Then the current first intermediate frames include: t6, t7, t8, there is a current newly added frame, and there are also multiple current reused frames. This example will be hereinafter referred to as Example B for short.

[0109] Exemplarily, in the above Example B, if the current reference frame is t9 and the associated reference frame is t7, then the current first intermediate frames include: t7, t8, t9, that is, there is a current newly added frame and there is one current reused frame. This example will be hereinafter referred to as Example C for short.

[0110] Then, the electronic device can obtain the optical flow information between every two temporally adjacent first intermediate frames from the current optical flow information to be utilized as the current first optical flow information. Specifically, in the case where there is no current newly added frame in the current first intermediate frames, obtain the optical flow information between every two temporally adjacent first intermediate frames from the current optical flow information to be utilized as the current first optical flow information; in the case where there is a current newly added frame in the current first intermediate frames and there are multiple current reused frames, for every two temporally adjacent first intermediate frames, if the optical flow information between the two temporally adjacent first intermediate frames exists in the current optical flow information to be utilized, obtain the optical flow information between the two temporally adjacent first intermediate frames from the current optical flow information to be utilized; otherwise, obtain the optical flow information between the two temporally adjacent first intermediate frames from the current newly added optical flow information to obtain the current first optical flow information; in the case where there is a current newly added frame in the current first intermediate frames and there is one current reused frame, obtain the optical flow information between every two temporally adjacent first intermediate frames from the current newly added optical flow information as the current first optical flow information.

[0111] It can be understood that in the case where there is no current newly added frame in the current first intermediate frames, that is, the current first intermediate frames are all current reused frames, the optical flow information between every two temporally adjacent first intermediate frames must already exist in the current optical flow information to be utilized. Therefore, the electronic device can directly obtain the optical flow information between every two temporally adjacent first intermediate frames from the current optical flow information to be utilized as the current first optical flow information.

[0112] When there is a current newly added frame (hereinafter referred to as the first newly added frame) in the current first intermediate frame, the electronic device needs to obtain, from the current newly added optical flow information, the optical flow information between each first newly added frame and the previous video frame of the first newly added frame in the current set of video frames to be aligned. When there are multiple current multiplexed frames in the current first intermediate frame, the multiple current multiplexed frames existing in the first intermediate frame are simply referred to as the first multiplexed frames, and the electronic device needs to obtain, from the current optical flow information to be utilized, the optical flow information between every two temporally adjacent first multiplexed frames.

[0113] Therefore, when there is a current newly added frame and multiple current multiplexed frames in the current first intermediate frame, the electronic device obtains the current first optical flow information from the current newly added optical flow information and the optical flow information to be utilized. That is, for every two temporally adjacent first intermediate frames of the current, if there is optical flow information between the two temporally adjacent first intermediate frames in the current optical flow information to be utilized, obtain the optical flow information between the two temporally adjacent first intermediate frames from the current optical flow information to be utilized; otherwise, obtain the optical flow information between the two temporally adjacent first intermediate frames from the current newly added optical flow information to obtain the current first optical flow information.

[0114] When there is a current newly added frame and one current multiplexed frame in the current first intermediate frame, the electronic device only needs to obtain the current first optical flow information from the current optical flow information to be utilized. That is, obtain the optical flow information between every two temporally adjacent first intermediate frames from the current newly added optical flow information as the current first optical flow information. For example, based on the foregoing Example C, the current first intermediate frames include t7, t8, t9, t7 is the current multiplexed frame, and t8, t9 are the current newly added frames, and the current first optical flow information includes All belong to the current newly added optical flow information.

[0115] For each current multiplexed frame, when the timing of the multiplexed frame is earlier than the associated reference frame, when aligning the current set of associated video frames, the alignment result of aligning the multiplexed frame to the associated reference frame (i.e., the current alignment result to be utilized) has been obtained. Therefore, the electronic device can directly obtain the previously recorded current alignment result to be utilized, and then sequentially use the current first optical flow information to align the current alignment result to be utilized to the current reference frame according to the timing of the video frames until the alignment result of the multiplexed frame to the current reference frame is obtained.

[0116] Exemplarily, as in the foregoing Figure 2 illustrated embodiment, at the second moment, t1 is the current multiplexed frame whose timing is earlier than the associated reference frame (i.e., t2). At the first moment, the alignment result of aligning t1 to t2 has been obtained At the second moment, use to perform pixel resampling, that is, obtain the alignment result of aligning t1 to the current reference frame (i.e., t3)

[0117] Exemplarily, as in the aforementioned example C, the current first intermediate frame includes: t7, t8, t9, then the current first optical flow information includes and t3 is the current multiplexed frame whose time sequence is earlier than the associated reference frame (i.e., t7). When aligning the current set of associated video frames, the alignment result of aligning t3 to t7 has been obtained Then use to perform pixel resampling to obtain the alignment result of aligning t3 to t8 Then use to perform pixel resampling to obtain the alignment result of aligning t3 to t9 (i.e., the current reference frame)

[0118] For each current multiplexed frame, when the multiplexed frame is the associated reference frame, the current first optical flow information is: among the current set of video frames to be aligned, the optical flow information between every two adjacent video frames in time sequence between the multiplexed frame and the current reference frame. Then, according to the time sequence of the video frames, use the current first optical flow information in sequence to align the multiplexed frame to the current reference frame to obtain the alignment result of the multiplexed frame. Exemplarily, as in the aforementioned example C, t7 is the associated reference frame, that is, the current multiplexed frame. The electronic device can use to perform pixel resampling on t7 to obtain the alignment result of aligning t7 to t8 Then use to perform pixel resampling to obtain the alignment result of aligning t7 to t9

[0119] For each current multiplexing frame, when the timing of this multiplexing frame is later than the associated reference frame and earlier than the current reference frame, that is, the timing of this multiplexing frame is between the associated reference frame and the current reference frame. If this multiplexing frame needs to be aligned with the current reference frame, the electronic device needs to determine this multiplexing frame, the current reference frame, and each video frame between this multiplexing frame and the current reference frame from the current set of video frames to be aligned according to the timing of the video frames, so as to obtain the second intermediate frame currently corresponding to this multiplexing frame. Furthermore, from the currently available optical flow information, obtain the optical flow information between every two second intermediate frames currently corresponding to this multiplexing frame as the second optical flow information currently corresponding to this multiplexing frame. Specifically, when there is a current new frame in the second intermediate frame currently corresponding to this multiplexing frame, obtain the optical flow information between every two second intermediate frames with adjacent timings currently corresponding to this multiplexing frame from the currently available optical flow information as the second optical flow information currently corresponding to this multiplexing frame; when there is a current new frame in the second intermediate frame currently corresponding to this multiplexing frame and there are multiple current multiplexing frames, for every two second intermediate frames with adjacent timings currently corresponding to this multiplexing frame, if there is optical flow information between these two second intermediate frames with adjacent timings in the currently available optical flow information, obtain the optical flow information between these two second intermediate frames with adjacent timings from the currently available optical flow information; otherwise, obtain the optical flow information between these two second intermediate frames with adjacent timings from the currently available new optical flow information to obtain the second optical flow information currently corresponding to this multiplexing frame; when there is a current new frame in the second intermediate frame currently corresponding to this multiplexing frame and there is one current multiplexing frame, obtain the optical flow information between every two second intermediate frames with adjacent timings currently corresponding to this multiplexing frame from the currently available new optical flow information as the second optical flow information currently corresponding to this multiplexing frame.

[0120] It can be understood that when there is no current new frame in the second intermediate frame currently corresponding to this multiplexing frame, that is, all the second intermediate frames currently corresponding to this multiplexing frame are current multiplexing frames, the optical flow information between every two second intermediate frames with adjacent timings currently corresponding to this multiplexing frame must already exist in the currently available optical flow information. Therefore, the electronic device can directly obtain the optical flow information between every two second intermediate frames with adjacent timings currently corresponding to this multiplexing frame from the currently available optical flow information as the second optical flow information currently corresponding to this multiplexing frame.

[0121] When there is a current newly added frame (hereinafter referred to as the second newly added frame) in the second intermediate frame currently corresponding to the multiplexing frame, the electronic device needs to obtain, from the current newly added optical flow information, the optical flow information between the second newly added frame and the previous video frame of the second newly added frame in the current set of video frames to be aligned. When there are multiple current multiplexing frames in the second intermediate frame currently corresponding to the multiplexing frame, the multiple current multiplexing frames existing in the second intermediate frame currently corresponding to the multiplexing frame are simply referred to as the second multiplexing frames, and the electronic device needs to obtain, from the current optical flow information to be utilized, the optical flow information between every two temporally adjacent second multiplexing frames.

[0122] Therefore, when there is a current newly added frame and multiple current multiplexing frames in the second intermediate frame currently corresponding to the multiplexing frame, the electronic device needs to obtain the second optical flow information corresponding to the current multiplexing frame from the current newly added optical flow information and the optical flow information to be utilized. That is, for every two temporally adjacent second intermediate frames corresponding to the current multiplexing frame, if there is optical flow information between the two temporally adjacent second intermediate frames in the current optical flow information to be utilized, obtain the optical flow information between the two temporally adjacent second intermediate frames from the current optical flow information to be utilized; otherwise, obtain the optical flow information between the two temporally adjacent second intermediate frames from the current newly added optical flow information to obtain the second optical flow information corresponding to the current multiplexing frame.

[0123] When there is a current newly added frame and one current multiplexing frame in the second intermediate frame currently corresponding to the multiplexing frame, the electronic device only needs to obtain the second optical flow information corresponding to the current multiplexing frame from the current newly added optical flow information. That is, obtain the optical flow information between every two temporally adjacent second intermediate frames corresponding to the current multiplexing frame from the current newly added optical flow information as the second optical flow information corresponding to the current multiplexing frame.

[0124] Then, the electronic device can sequentially use the second optical flow information corresponding to the current multiplexing frame according to the time sequence of the video frames to align the multiplexing frame with the current reference frame to obtain the alignment result of the multiplexing frame. For example, as in the aforementioned Example B, t7 is a multiplexing frame located between the associated reference frame and the current reference frame. The second intermediate frames currently corresponding to t7 include: t7, t8, there is one current newly added frame (i.e., t8), and one current multiplexing frame (i.e., t7). Therefore, the electronic device obtains from the current newly added optical flow information That is, obtain the second optical flow information corresponding to t7. Use to perform pixel resampling on t7 to obtain the alignment result of aligning t7 with t8

[0125] For each current multiplexing frame, when the timing of this multiplexing frame is later than the current reference frame, that is, when aligning the current associated video frame set, the alignment result of aligning this multiplexing frame to the current reference frame has been obtained. Therefore, the alignment result of this multiplexing frame to the current reference frame obtained when aligning the current associated video frame set, which was pre-recorded, can be directly obtained. Exemplarily, as in the foregoing Figure 2 shown embodiment, at the second moment, t4 is a multiplexing frame whose timing is later than the current reference frame (i.e., t3). At the first moment, the electronic device has used to align t4 to t3 and obtained the alignment result Therefore, at the second moment, the electronic device can directly obtain the calculated and pre-recorded at the first moment, and thus directly obtain the alignment result of aligning t4 to t3.

[0126] Based on the above processing, for each current multiplexing frame, according to the specific situation corresponding to this multiplexing frame in the actual scenario, the alignment result of this multiplexing frame to the current reference frame can be obtained according to the corresponding processing method for this situation. The currently available optical flow information can be reused, that is, the previously calculated optical flow information can be reused, reducing the computing power resources consumed when calculating new optical flow information, that is, the consumption of the computing power resources of the electronic device can be reduced. Moreover, by aligning video frames to the current reference frame by sequentially using the corresponding optical flow information (which can be called the alignment result recursion method), for multiplexing frames whose timing is later than the current reference frame, the alignment result of this multiplexing frame to the current reference frame can be obtained when aligning the current associated video frame set, further reducing the consumption of the computing power resources of the electronic device.

[0127] In some embodiments, when the electronic device aligns each video frame set, it can directly record all the data involved in the alignment process, such as each video frame included in this video frame set, the optical flow information calculated between every two adjacent video frames in time sequence in this video frame set, the alignment result of each associated frame in this video frame set to the reference frame, and all the alignment results calculated during the alignment process based on the foregoing alignment result recursion method. Exemplarily, as in the foregoing Figure 2 shown embodiment, at the first moment, the electronic device can record [t0, t1, t2, t3, t4], and at the second moment, the electronic device can record [t1, t2, t3, t4, t5], and and so on. At each moment, the electronic device can record all the data involved in the alignment process executed at this moment.

[0128] In some embodiments, data obtained when aligning the current set of video frames to be aligned can also be used to update the data recorded in the queue, so as to reduce the occupation of storage space and further reduce the hardware requirements for the electronic device.

[0129] Specifically, after the above step S101, the method may further include the following steps: According to the time sequence of each video frame in the current set of video frames to be aligned, each video frame in the current set of video frames to be aligned is sequentially used to update the video frames recorded at each position in the current video frame queue. Wherein, each video frame in the current associated video frame set is recorded in the current video frame queue; for every two adjacent positions in the video frame queue, they correspond to the positions in the optical flow information queue for recording the optical flow information between the video frames recorded at these two positions, and the positions in the alignment result queue for recording the respective alignment results of the video frames recorded at these two positions; after the above step S104, the method may further include the following steps: For every two adjacent video frames in time sequence in the current set of video frames to be aligned, if there is optical flow information between these two adjacent video frames in time sequence in the current optical flow information to be utilized, obtain the optical flow information between these two adjacent video frames in time sequence from the current optical flow information to be utilized; otherwise, obtain the optical flow information between these two adjacent video frames in time sequence from the current newly added optical flow information to obtain the current updated optical flow information; According to the corresponding relationship between every two adjacent positions in the video frame queue and the positions in the optical flow information queue, the current updated optical flow information is sequentially used to update the optical flow information recorded at each position in the current optical flow information queue; wherein, the current optical flow information to be utilized is recorded in the current optical flow information queue; after the above step C, the method may further include the following steps: According to the corresponding relationship between every two adjacent positions in the video frame queue and the positions in the alignment result queue, the alignment results of aligning each current associated frame to the current reference frame are sequentially used to update the alignment results recorded at each position in the current alignment result queue; wherein, the alignment results of aligning each associated frame in the current associated video frame set to the reference frame are recorded in the current alignment result queue.

[0130] The electronic device can use the video frame queue to record the video frame set. For example, the electronic device can first obtain the current video frame queue recording each video frame in the current associated video frame set. After determining the current set of video frames to be aligned, according to the time sequence of the video frames, each video frame in the current set of video frames to be aligned is sequentially used to update the video frames recorded at each position in the current video frame queue, that is, update the current video frame queue, so that the updated current video frame queue records each video frame in the current set of video frames to be aligned. The present invention does not limit the specific manner of updating the video frame queue.

[0131] Exemplarily, the video frame queue can be a FIFO queue. The electronic device can perform an enqueue process on the current newly added frame at the tail of the video frame queue, and perform a dequeue process on the video frame at the head of the video frame queue, and the number of enqueued video frames is the same as the number of dequeued video frames. In this way, each video frame in the current set of video frames to be aligned is sequentially updated to the video frame recorded at each position in the current video frame queue.

[0132] Exemplarily, according to the timing of the video frames, the electronic device can first determine the corresponding recording position of each video frame in the current set of video frames to be aligned in the current video frame queue, and then directly use the video frame to overwrite the data recorded at the corresponding recording position.

[0133] Moreover, the electronic device can use an optical flow information queue to record the optical flow information between every two adjacent video frames in the current video frame queue in terms of timing. That is, for every two adjacent positions in the video frame queue, they correspond to the positions in the optical flow information queue for recording the optical flow information between the video frames recorded at these two positions. For example, if the video frame queue includes 5 positions for recording video frames, and the video frame recorded at the 3rd position is the reference frame. Then the 1st position in the optical flow information queue can be used to record: the optical flow information between the video frames recorded at the 1st and 2nd positions in the video frame queue; the 2nd position in the optical flow information queue can be used to record: the optical flow information between the video frames recorded at the 2nd and 3rd positions in the video frame queue; and so on. If the current video frame queue records [t0, t1, t2, t3, t4], then the current optical flow information queue records

[0134] Similarly, the electronic device can use an alignment result queue to record the alignment results obtained during the alignment of the video frames recorded in the current video frame queue. That is, for every two adjacent positions in the video frame queue, they correspond to the positions in the alignment result queue for recording the respective alignment results of the video frames recorded at these two positions.

[0135] Based on the above example, the 1st position in the alignment result queue can be used to record: the alignment result of aligning the video frame recorded at the 1st position in the video frame queue to the video frame recorded at the 2nd position; the 2nd position in the alignment result queue can be used to record: the alignment result of aligning the video frame recorded at the 1st position in the video frame queue to the video frame recorded at the 3rd position; the 3rd position in the alignment result queue can be used to record: the alignment result of aligning the video frame recorded at the 2nd position in the video frame queue to the video frame recorded at the 3rd position; and so on. If the current video frame queue records [t0, t1, t2, t3, t4], then the current optical flow information queue records

[0136] Regarding the optical flow information queue, obviously, after the current video frame queue is updated, the optical flow information to be recorded in the optical flow information queue currently includes: the current partial optical flow information to be utilized, and the current entire newly added optical flow information. Therefore, for every two temporally adjacent video frames in the current set of video frames to be aligned, if there is optical flow information between these two temporally adjacent video frames in the current optical flow information to be utilized, obtain the optical flow information between these two temporally adjacent video frames from the current optical flow information to be utilized; otherwise, obtain the optical flow information between these two temporally adjacent video frames from the current newly added optical flow information to obtain the current updated optical flow information. Then, according to the corresponding relationship between every two adjacent positions in the video frame queue and the positions in the optical flow information queue, sequentially update the optical flow information recorded at each position in the current optical flow information queue with the current updated optical flow information. As based on the foregoing example, the updated current video frame queue records [t1, t2, t3, t4, t5], then the electronic device can use to update the used to update the optical flow information recorded at the second position in the current optical flow information queue, and so on, to obtain the updated current optical flow information queue recording .

[0137] For the alignment result queue, after the electronic device aligns the current set of video frames to be aligned, it can, according to the corresponding relationship between every two adjacent positions in the video frame queue and the positions in the alignment result queue, sequentially update the alignment results of aligning the current associated frames to the current reference frame at each position recorded in the current alignment result queue; and, according to the requirements of the actual scenario, in the case of performing alignment based on the foregoing alignment result recursion method, other alignment results calculated during the process of aligning the current associated frames to the current reference frame can also be recorded in the alignment result queue. As based on the foregoing example, the updated current video frame queue records [t1, t2, t3, t4, t5], then the electronic device can use to update the used to update the alignment result recorded at the second position in the current alignment result queue, and so on, to obtain the updated current alignment result queue recording .

[0138] In some embodiments, the frame numbers of the respective video frames recorded in the aforementioned video frame queue remain unchanged, that is, regardless of the timing of the video frames recorded in the video frame queue in the video to be processed, the frame number of each video frame is always determined by the position of the video frame recorded in the video frame queue. Correspondingly, the frame numbers of the video frames corresponding to the respective optical flow information recorded in the aforementioned optical flow information queue, and the frame numbers of the video frames corresponding to the respective alignment results recorded in the aforementioned alignment result queue also remain unchanged.

[0139] Exemplarily, as in the aforementioned Figure 2 illustrated embodiment, at the first moment, the electronic device first reads the video stream image frame data, that is, sequentially obtains each video frame in the video to be processed, and uses a first-in-first-out (FIFO) queue to cache 5 frame data, that is, records 5 consecutive video frames in the video frame queue, obtaining the current video frame queue L = [t0, t1, t2, t3, t4]. The frame numbers of the respective video frames in the video frame queue are sequentially 0, 1, 2, 3, 4. Then, the current video frame queue L is initialized, that is, the optical flow information between every two adjacent video frames in the current video frame queue L is calculated, and the optical flow information calculated at the first moment is recorded in the current optical flow information queue, obtaining the current optical flow information queue. Furthermore, the electronic device aligns each associated frame in the current set of video frames to be aligned with respect to the reference frame according to the video frame alignment method provided by the present invention, and records the alignment result calculated at the first moment in the current alignment result queue, obtaining the current alignment result queue. Correspondingly, the electronic device can cache and And, the electronic device can output the current That is, the alignment results of each associated frame in the current set of video frames to be aligned with respect to the reference frame are output.

[0140] Furthermore, at the second moment, a new video frame t5 is enqueued. The electronic device can sequentially set t0 = t1; t1 = t2; t2 = t3; t3 = t4; t4 = t5, such that in the updated current video frame queue L = [t0, t1, t2, t3, t4], each video frame in the current set of video frames to be aligned is recorded. At this time, the frame numbers of the respective video frames in the video frame queue are still sequentially 0, 1, 2, 3, 4. Sequentially such that in the updated current optical flow information queue the optical flow information between every two adjacent video frames in the current set of video frames to be aligned is recorded. Sequentially such that in the updated current alignment result queue It records the alignment results of each associated frame in the current set of video frames to be aligned to the current reference frame. Cache the current optical flow information queue and the current alignment result queue, that is, cache and , for use in subsequent alignment processes. Moreover, the electronic device can output the current , that is, it outputs the alignment results of each associated frame in the current set of video frames to be aligned to the reference frame. And so on, until the alignment results of the associated frames of each video frame in the video to be processed to that video frame are obtained.

[0141] In some embodiments, step B above may include the following steps: For each current multiplexing frame, determine the third intermediate frame currently corresponding to the multiplexing frame from the current set of video frames to be aligned according to the timing of the video frames; wherein, the third intermediate frame currently corresponding to the multiplexing frame includes: the multiplexing frame, the current reference frame, and each video frame located between the multiplexing frame and the current reference frame; Obtain the optical flow information between every two adjacent third intermediate frames currently corresponding to the multiplexing frame from the current optical flow information to be utilized, as the third optical flow information currently corresponding to the multiplexing frame; Calculate the optical flow information between the multiplexing frame and the current reference frame according to the third optical flow information currently corresponding to the multiplexing frame, as the target optical flow information; Use the target optical flow information to align the multiplexing frame to the current reference frame to obtain the alignment result of the multiplexing frame.

[0142] For each current multiplexing frame, the electronic device can calculate the target optical flow information required to directly align the multiplexing frame to the current reference frame. To reduce the consumption of computing power resources of the electronic device, the electronic device can determine the multiplexing frame, the current reference frame, and each video frame located between the multiplexing frame and the current reference frame from the current set of video frames to be aligned, to obtain the third intermediate frame currently corresponding to the multiplexing frame. Then, obtain the optical flow information between every two adjacent third intermediate frames currently corresponding to the multiplexing frame from the current optical flow information to be utilized, as the third optical flow information currently corresponding to the multiplexing frame.

[0143] Specifically, if the current newly added frame does not exist in the third intermediate frame currently corresponding to the multiplexing frame, obtain the optical flow information between every two temporally adjacent third intermediate frames currently corresponding to the multiplexing frame from the currently available optical flow information to be utilized as the third optical flow information corresponding to the multiplexing frame. If the current newly added frame exists in the third intermediate frame currently corresponding to the multiplexing frame and there are multiple current multiplexing frames, for every two temporally adjacent third intermediate frames currently corresponding to the multiplexing frame, if the optical flow information between the two temporally adjacent third intermediate frames exists in the currently available optical flow information to be utilized, obtain the optical flow information between the two temporally adjacent third intermediate frames from the currently available optical flow information to be utilized; otherwise, obtain the optical flow information between the two temporally adjacent third intermediate frames from the currently newly added optical flow information to obtain the third optical flow information corresponding to the multiplexing frame. If the current newly added frame exists in the third intermediate frame currently corresponding to the multiplexing frame and there is one current multiplexing frame, obtain the optical flow information between every two temporally adjacent third intermediate frames currently corresponding to the multiplexing frame from the currently newly added optical flow information as the third optical flow information corresponding to the multiplexing frame.

[0144] It can be understood that when the current newly added frame does not exist in the third intermediate frame currently corresponding to the multiplexing frame, that is, all the third intermediate frames currently corresponding to the multiplexing frame are current multiplexing frames, the optical flow information between every two temporally adjacent third intermediate frames currently corresponding to the multiplexing frame must already exist in the currently available optical flow information to be utilized. Therefore, the electronic device can directly obtain the optical flow information between every two temporally adjacent third intermediate frames currently corresponding to the multiplexing frame from the currently available optical flow information to be utilized as the third optical flow information corresponding to the multiplexing frame.

[0145] When the current newly added frame (hereinafter referred to as the third newly added frame) exists in the third intermediate frame currently corresponding to the multiplexing frame, the electronic device needs to obtain the optical flow information between the third newly added frame and the previous video frame of the third newly added frame in the currently to-be-aligned video frame set from the currently newly added optical flow information. When there are multiple current multiplexing frames in the third intermediate frame currently corresponding to the multiplexing frame, if the multiple current multiplexing frames existing in the third intermediate frame currently corresponding to the multiplexing frame are simply referred to as third multiplexing frames, the electronic device needs to obtain the optical flow information between every two temporally adjacent third multiplexing frames from the currently available optical flow information to be utilized.

[0146] Therefore, when there is a current newly added frame in the third intermediate frame currently corresponding to the multiplexed frame and there are multiple current multiplexed frames, the third optical flow information currently corresponding to the multiplexed frame is obtained from the current newly added optical flow information and the optical flow information to be utilized. That is, for every two temporally adjacent third intermediate frames currently corresponding to the multiplexed frame, if there is optical flow information between the two temporally adjacent third intermediate frames in the current optical flow information to be utilized, the optical flow information between the two temporally adjacent third intermediate frames is obtained from the current optical flow information to be utilized; otherwise, the optical flow information between the two temporally adjacent third intermediate frames is obtained from the current newly added optical flow information, and the third optical flow information currently corresponding to the multiplexed frame is obtained.

[0147] When there is a current newly added frame in the third intermediate frame currently corresponding to the multiplexed frame and there is one current multiplexed frame, the electronic device only needs to obtain the third optical flow information currently corresponding to the multiplexed frame from the current newly added optical flow information. That is, the optical flow information between every two temporally adjacent third intermediate frames currently corresponding to the multiplexed frame is obtained from the current newly added optical flow information as the third optical flow information currently corresponding to the multiplexed frame.

[0148] Then, according to the time sequence of the video frames, the third optical flow information currently corresponding to the multiplexed frame is synthesized to calculate the target optical flow information required for directly aligning the multiplexed frame with the current reference frame.

[0149] It can be understood that if two temporally adjacent video frames are respectively denoted as video frame 1 and video frame 2, the optical flow information included in the currently to-be-utilized optical flow information between the two temporally adjacent video frames may include: the optical flow information from video frame 1 to video frame 2, and / or, the optical flow information from video frame 2 to video frame 1. For each current multiplexed frame, if the time sequence of the multiplexed frame is earlier than the current reference frame, and the third optical flow information currently corresponding to the multiplexed frame is: the optical flow information from the video frame with an earlier time sequence to the video frame with a later time sequence, then the electronic device may calculate the sum value of the third optical flow information currently corresponding to the multiplexed frame to obtain the target optical flow information. If the time sequence of the multiplexed frame is earlier than the current reference frame, and the third optical flow information currently corresponding to the multiplexed frame is: the optical flow information from the video frame with a later time sequence to the video frame with an earlier time sequence, then the electronic device may calculate the sum value of the third optical flow information currently corresponding to the multiplexed frame, and then calculate the negative value of the sum value to obtain the target optical flow information. Similarly, if the time sequence of the multiplexed frame is later than the current reference frame, and the third optical flow information currently corresponding to the multiplexed frame is: the optical flow information from the video frame with a later time sequence to the video frame with an earlier time sequence, then the electronic device may calculate the sum value of the third optical flow information currently corresponding to the multiplexed frame to obtain the target optical flow information. If the time sequence of the multiplexed frame is later than the current reference frame, and the third optical flow information currently corresponding to the multiplexed frame is: the optical flow information from the video frame with an earlier time sequence to the video frame with a later time sequence, then the electronic device may calculate the sum value of the third optical flow information currently corresponding to the multiplexed frame, and then calculate the negative value of the sum value to obtain the target optical flow information.

[0150] Exemplarily, as in the foregoing Figure 2 illustrated embodiment, at the second moment, the electronic device may calculate and sum value of to obtain (i.e., the target optical flow information). Then, use to perform pixel resampling on t1 to obtain the alignment result

[0151] Then, use the target optical flow information to align the multiplexed frame to the current reference frame, that is, use the target optical flow information to perform pixel resampling on the multiplexed frame to obtain the alignment result of aligning the multiplexed frame to the current reference frame.

[0152] Based on the above processing, for each current multiplexing frame, the electronic device can calculate the target optical flow information required to align the multiplexing frame with the current reference frame according to the third optical flow information currently corresponding to the multiplexing frame, and then directly align the multiplexing frame with the current reference frame using the target optical flow information. Moreover, by calculating the optical flow information required to align the video frame with the current reference frame and then directly aligning the video frame with the current reference frame using the calculated optical flow information (which can be called the optical flow information recursion method), the processing logic is simpler and the adaptability to complex situations in the actual scenario is better.

[0153] As described above, in the actual scenario, the current newly added frame may be the current reference frame, that is, it is not necessary to align each current newly added frame with the current reference frame. Therefore, the electronic device needs to determine the optical flow information (which can be simply referred to as the optical flow required for the newly added frame) required to align the video frames other than the current reference frame in the current newly added frame with the current reference frame based on the current optical flow information to be utilized and the newly added optical flow information, and then align the video frame with the current reference frame based on the optical flow required for the newly added frame to obtain the alignment result of the video frame. According to the similar processing method as described above when there are multiple optical flows required for the multiplexing frame, when there are multiple optical flows required for the newly added frame, the electronic device can determine the usage order of the optical flows required for each newly added frame according to the time sequence of the video frames.

[0154] In some embodiments, step C above may include the following steps: for each video frame in the current newly added frame except the current reference frame, determine the current corresponding fourth intermediate frame of the video frame from the current set of video frames to be aligned according to the time sequence of the video frames; wherein, the current corresponding fourth intermediate frame of the video frame includes: the video frame, the current reference frame, and each video frame located between the video frame and the current reference frame; obtain the optical flow information between each two adjacent fourth intermediate frames of the video frame in time sequence from the current optical flow information to be utilized and the newly added optical flow information, as the fourth optical flow information corresponding to the video frame; if the number of current reused frames in the current corresponding fourth intermediate frame of the video frame is not greater than 1, align the video frame with the fourth optical flow information corresponding to the video frame in sequence according to the time sequence of the video frames until the alignment result of the video frame to the current reference frame is obtained; if there are multiple current reused frames in the current corresponding fourth intermediate frame of the video frame, determine the current corresponding fifth intermediate frame of the video frame from the current corresponding fourth intermediate frame of the video frame according to the time sequence of the video frames; wherein, the current corresponding fifth intermediate frame of the video frame includes: the current reference frame, the current latest reused frame in time sequence; and each video frame located between the current reference frame and the current latest reused frame in time sequence; obtain the optical flow information between each two adjacent fifth intermediate frames of the video frame from the fourth optical flow information corresponding to the video frame, as the fifth optical flow information; determine the current corresponding sixth intermediate frame of the video frame from the current corresponding fourth intermediate frame of the video frame according to the time sequence of the video frames; wherein, the current sixth intermediate frame includes: the current latest reused frame in time sequence, the video frame, and each video frame located between the current latest reused frame in time sequence and the video frame; obtain the optical flow information between each two adjacent sixth intermediate frames of the video frame from the fourth optical flow information corresponding to the video frame, as the sixth optical flow information; align the video frame with the current latest reused frame in time sequence with the sixth optical flow information corresponding to the video frame in sequence until the alignment result of the video frame to the current latest reused frame in time sequence is obtained, as the intermediate alignment result corresponding to the video frame; align the intermediate alignment result corresponding to the video frame with the fifth optical flow information corresponding to the video frame in sequence according to the time sequence of the video frames until the alignment result of the video frame to the current reference frame is obtained.

[0155] For each video frame in the current newly added frames except the current reference frame, if the video frame needs to be aligned with the current reference frame, the electronic device needs to determine the video frame, the current reference frame, and each video frame located between the video frame and the current reference frame from the current set of video frames to be aligned according to the timing sequence of the video frames, so as to obtain the fourth intermediate frame currently corresponding to the video frame. Then, from the current optical flow information to be utilized and the newly added optical flow information, the optical flow information between every two adjacent fourth intermediate frames currently corresponding to the video frame is obtained as the fourth optical flow information currently corresponding to the video frame.

[0156] Specifically, if the number of current reused frames in the fourth intermediate frame currently corresponding to the video frame is no more than 1, the optical flow information between every two adjacent fourth intermediate frames currently corresponding to the video frame is obtained from the current newly added optical flow information as the fourth optical flow information currently corresponding to the video frame; if there are multiple current reused frames in the fourth intermediate frame currently corresponding to the video frame, for every two adjacent fourth intermediate frames currently corresponding to the video frame, if the optical flow information between the two adjacent fourth intermediate frames exists in the current optical flow information to be utilized, the optical flow information between the two adjacent fourth intermediate frames is obtained from the current optical flow information to be utilized; otherwise, the optical flow information between the two adjacent fourth intermediate frames is obtained from the current newly added optical flow information to obtain the fourth optical flow information currently corresponding to the video frame.

[0157] In the case where there are multiple current reused frames in the fourth intermediate frame currently corresponding to the video frame, if the multiple current reused frames existing in the fourth intermediate frame currently corresponding to the video frame are simply referred to as the fourth reused frames, the electronic device needs to obtain the optical flow information between every two adjacent fourth reused frames from the current optical flow information to be utilized.

[0158] Therefore, in the case where the number of current reused frames in the fourth intermediate frame currently corresponding to the video frame is no more than 1, the electronic device directly obtains the optical flow information between every two adjacent fourth intermediate frames currently corresponding to the video frame from the current newly added optical flow information. Furthermore, directly according to the timing sequence of the video frames, the fourth optical flow information currently corresponding to the video frame is used to align the video frame in sequence until the alignment result of the video frame with the current reference frame is obtained.

[0159] In the case that there are multiple current multiplexed frames in the fourth intermediate frame currently corresponding to the video frame, the electronic device needs to obtain the fourth optical flow information corresponding to the video frame currently from the current newly added optical flow information and the optical flow information to be utilized. That is, for every two temporally adjacent fourth intermediate frames currently corresponding to the video frame, if there is optical flow information between the two temporally adjacent fourth intermediate frames in the current optical flow information to be utilized, obtain the optical flow information between the two temporally adjacent fourth intermediate frames from the current optical flow information to be utilized; otherwise, obtain the optical flow information between the two temporally adjacent fourth intermediate frames from the current newly added optical flow information, to obtain the fourth optical flow information corresponding to the video frame currently.

[0160] Furthermore, the electronic device determines the fifth intermediate frame currently corresponding to the video frame from the fourth intermediate frame currently corresponding to the video frame according to the time sequence of the video frame; obtains the optical flow information between every two temporally adjacent fifth intermediate frames currently corresponding to the video frame from the fourth optical flow information corresponding to the video frame currently, as the fifth optical flow information corresponding to the video frame currently; determines the sixth intermediate frame currently corresponding to the video frame from the fourth intermediate frame currently corresponding to the video frame; obtains the optical flow information between every two temporally adjacent sixth intermediate frames currently corresponding to the video frame from the fourth optical flow information corresponding to the video frame currently, as the sixth optical flow information corresponding to the video frame currently. Then, according to the time sequence of the video frame, sequentially use the sixth optical flow information corresponding to the video frame currently to align the video frame to the multiplexed frame with the latest time sequence currently, until obtaining the alignment result of the video frame to the multiplexed frame with the latest time sequence currently (that is, the intermediate alignment result currently corresponding to the video frame). According to the time sequence of the video frame, sequentially use the fifth optical flow information corresponding to the video frame currently to align the intermediate alignment result currently corresponding to the video frame to the reference frame currently, until obtaining the alignment result of the video frame to the reference frame currently. The manner of sequentially using the sixth optical flow information to align the video frame to the multiplexed frame with the latest time sequence currently and the manner of sequentially using the fifth optical flow information to align the intermediate alignment result currently corresponding to the video frame to the reference frame currently are similar to the manner of sequentially using the current first optical flow information to align the current alignment result to be utilized to the reference frame currently, and the relevant introduction of the foregoing embodiments can be referred to. Exemplarily, as shown in the foregoing Figure 2 embodiment, at the second moment, the current set of video frames to be aligned is [t1, t2, t3, t4, t5], the current reference frame is t3, the multiplexed frame with the latest time sequence currently is t4, and the current newly added frame is t5. Then the fifth intermediate frames include t3 and t4, and the fifth optical flow information includes The sixth intermediate frames include t4 and t5, and the sixth optical flow information includes The electronic device can first use to align t5 to t4 to obtain the intermediate alignment result Then use to align with t3 to obtain the alignment result of t5 with respect to t3

[0161] Based on the above processing, for each video frame in the current newly added frames except the current reference frame, the electronic device can, based on the alignment result recursion method, align the video frame with the current reference frame. By repeatedly using the previously calculated optical flow information, the number of newly calculated optical flow information is less, which can reduce the consumption of computing power resources of the electronic device.

[0162] In some embodiments, the electronic device can also align each video frame in the current newly added frames except the current reference frame with the current reference frame based on the optical flow information pair push method. Specifically, step C above may include the following steps: for each video frame in the current newly added frames except the current reference frame, determine the current corresponding seventh intermediate frame of the video frame from the current set of video frames to be aligned according to the time sequence of the video frames; wherein, the current corresponding seventh intermediate frame of the video frame includes: the video frame, the current reference frame, and each video frame located between the video frame and the current reference frame; obtain the optical flow information between each two adjacent seventh intermediate frames corresponding to the video frame from the current optical flow information to be utilized and the newly added optical flow information, as the seventh optical flow information corresponding to the video frame; calculate the optical flow information between the video frame and the current reference frame according to the seventh optical flow information corresponding to the video frame, as the optical flow information to be utilized; use the optical flow information to be utilized to align the video frame with the current reference frame to obtain the alignment result of the video frame.

[0163] When calculating the optical flow information to be utilized, in order to reduce the consumption of computing power resources of the electronic device, the electronic device can determine the video frame, the current reference frame, and each video frame located between the video frame and the current reference frame from the current set of video frames to be aligned, so as to obtain the seventh intermediate frame currently corresponding to the video frame. Then, from the currently available optical flow information and the newly added optical flow information, the optical flow information between every two adjacent seventh intermediate frames currently corresponding to the video frame is obtained as the seventh optical flow information currently corresponding to the video frame. Specifically, when the number of currently reused frames existing in the seventh intermediate frame currently corresponding to the video frame is no more than 1, the optical flow information between every two adjacent seventh intermediate frames currently corresponding to the video frame is obtained from the currently newly added optical flow information as the seventh optical flow information currently corresponding to the video frame; when there are multiple currently reused frames existing in the seventh intermediate frame currently corresponding to the video frame, if there is optical flow information between the two adjacent seventh intermediate frames in the currently available optical flow information, the optical flow information between the two adjacent seventh intermediate frames is obtained from the currently available optical flow information; otherwise, the optical flow information between the two adjacent seventh intermediate frames is obtained from the currently newly added optical flow information to obtain the seventh optical flow information currently corresponding to the video frame.

[0164] When there are multiple currently reused frames existing in the seventh intermediate frame currently corresponding to the video frame, if the multiple currently reused frames existing in the seventh intermediate frame currently corresponding to the video frame are simply referred to as the seventh reused frames, the electronic device needs to obtain the optical flow information between every two adjacent seventh reused frames from the currently available optical flow information. Therefore, when the number of currently reused frames existing in the seventh intermediate frame currently corresponding to the video frame is no more than 1, the electronic device directly obtains the optical flow information between every two adjacent seventh intermediate frames currently corresponding to the video frame from the currently newly added optical flow information. When there are multiple currently reused frames existing in the seventh intermediate frame currently corresponding to the video frame, the electronic device needs to obtain the seventh optical flow information currently corresponding to the video frame from the currently newly added optical flow information and the available optical flow information. That is, for every two adjacent seventh intermediate frames currently corresponding to the video frame, if there is optical flow information between the two adjacent seventh intermediate frames in the currently available optical flow information, the optical flow information between the two adjacent seventh intermediate frames is obtained from the currently available optical flow information; otherwise, the optical flow information between the two adjacent seventh intermediate frames is obtained from the currently newly added optical flow information to obtain the seventh optical flow information currently corresponding to the video frame.

[0165] Then, the electronic device can, in a manner similar to the aforementioned determination of the target optical flow information, calculate the optical flow information to be utilized when directly aligning the video frame with the current reference frame by synthesizing the seventh optical flow information corresponding to the current video frame according to the timing sequence of the video frames. Then, the video frame is aligned with the current reference frame using the optical flow information to be utilized, that is, pixel resampling is performed on the video frame using the optical flow information to be utilized, and an alignment result of aligning the video frame with the current reference frame is obtained.

[0166] Exemplarily, as in the aforementioned Figure 2 embodiment shown, at the second moment, the seventh intermediate frames corresponding to the current newly added frame t5 include t3, t4, and t5; the seventh optical flow information corresponding to t5 currently includes and The electronic device can calculate and the sum value of to obtain Then, use to align t5 with t3 to obtain an alignment result

[0167] Based on the above processing, for each video frame in the current newly added frame except the current reference frame, the electronic device can align the video frame with the current reference frame based on the optical flow information recursion method. By repeatedly using the previously calculated optical flow information, the number of optical flow information that needs to be newly calculated is less, which can reduce the consumption of computing power resources of the electronic device. And the processing logic of the optical flow information recursion method is simpler, and it has better adaptability to complex situations in the actual scenario.

[0168] Based on the same inventive concept as the above video frame alignment method, the present invention also provides a video frame alignment device. Refer to Figure 3 , Figure 3 which is a structural diagram of a video frame alignment device provided by an embodiment of the present invention. The device includes:

[0169] A set determination module 301, configured to determine a current set of video frames to be aligned from the video to be processed according to the timing sequence of the video frames; wherein, a set of video frames includes a reference frame and associated frames that need to be aligned with the reference frame;

[0170] An information acquisition module 302, configured to acquire the optical flow information between every two adjacent video frames in the current set of associated video frames obtained when aligning the current set of associated video frames as the current optical flow information to be utilized; wherein, the current set of associated video frames is the set of video frames when the multiple video frames in the current set of video frames to be aligned participated in alignment last time;

[0171] A new frame determination module 303 is configured to determine, from the current set of video frames to be aligned, video frames that do not belong to the current set of associated video frames as the current new frames;

[0172] A calculation module 304 is configured to calculate the optical flow information between each current new frame and the previous video frame of the new frame in the current set of video frames to be aligned as the current new optical flow information;

[0173] An alignment module 305 is configured to align each current associated frame to the current reference frame respectively based on the current optical flow information to be utilized and the new optical flow information, to obtain the alignment result of each current associated frame; and return to execute the step of determining the current set of video frames to be aligned from the video to be processed according to the time sequence of the video frames.

[0174] Optionally, the alignment module 305 is specifically configured to: determine, from the current associated frames, video frames that belong to the current set of associated video frames as the current reused frames; align the current reused frames to the current reference frame based on the current optical flow information to be utilized, to obtain the alignment result of the current reused frames; and align, based on the current optical flow information to be utilized and the new optical flow information, the video frames other than the current reference frame in the current new frames to the current reference frame, to obtain the alignment result of the video frames other than the current reference frame in the current new frames.

[0175] Optionally, the alignment module 305 is specifically configured to: determine a current first intermediate frame from the current set of video frames to be aligned according to the time sequence of the video frames; wherein the current first intermediate frame includes: a reference frame in the current associated video frame set, the current reference frame, and each video frame located between the reference frame in the current associated video frame set and the current reference frame; obtain the optical flow information between every two adjacent first intermediate frames in time sequence from the current optical flow information to be utilized as the current first optical flow information; for each multiplexing frame, when the time sequence of the multiplexing frame is earlier than the reference frame in the current associated video frame set, obtain the alignment result of the multiplexing frame to the reference frame in the current associated video frame set obtained when aligning the current associated video frame set previously recorded as the current alignment result to be utilized; align the current alignment result to be utilized to the current reference frame in sequence according to the time sequence of the video frames using the current first optical flow information until the alignment result of the multiplexing frame to the current reference frame is obtained; when the multiplexing frame is the reference frame in the current associated video frame set, align the multiplexing frame to the current reference frame in sequence according to the time sequence of the video frames using the current first optical flow information to obtain the alignment result of the multiplexing frame; when the time sequence of the multiplexing frame is later than the reference frame in the current associated video frame set and earlier than the current reference frame, determine the current second intermediate frame corresponding to the multiplexing frame from the current set of video frames to be aligned according to the time sequence of the video frames; wherein the current second intermediate frame corresponding to the multiplexing frame includes: the multiplexing frame, the current reference frame, and each video frame located between the multiplexing frame and the current reference frame; obtain the optical flow information between every two adjacent second intermediate frames corresponding to the multiplexing frame in time sequence from the current optical flow information to be utilized as the current second optical flow information corresponding to the multiplexing frame; align the multiplexing frame to the current reference frame in sequence according to the time sequence of the video frames using the current second optical flow information corresponding to the multiplexing frame to obtain the alignment result of the multiplexing frame; when the time sequence of the multiplexing frame is later than the current reference frame, obtain the alignment result of the multiplexing frame to the current reference frame obtained when aligning the current associated video frame set previously recorded.

[0176] Optionally, the alignment module 305 is specifically configured to: when the current new frame does not exist in the current first intermediate frame, obtain the optical flow information between every two adjacent first intermediate frames in time sequence from the current optical flow information to be utilized as the current first optical flow information; when the current new frame exists in the current first intermediate frame and there are multiple current reused frames, for every two adjacent first intermediate frames in time sequence, if the optical flow information between the two adjacent first intermediate frames in time sequence exists in the current optical flow information to be utilized, obtain the optical flow information between the two adjacent first intermediate frames in time sequence from the current optical flow information to be utilized; otherwise, obtain the optical flow information between the two adjacent first intermediate frames in time sequence from the current new optical flow information to obtain the current first optical flow information; when the current new frame exists in the current first intermediate frame and there is one current reused frame, obtain the optical flow information between every two adjacent first intermediate frames in time sequence from the current new optical flow information as the current first optical flow information; and / or, obtaining the optical flow information between every two adjacent second intermediate frames corresponding to the current reused frame from the current optical flow information to be utilized includes: when the current new frame exists in the second intermediate frame corresponding to the current reused frame, obtain the optical flow information between every two adjacent second intermediate frames corresponding to the current reused frame from the current optical flow information to be utilized as the second optical flow information corresponding to the current reused frame; when the current new frame exists in the second intermediate frame corresponding to the current reused frame and there are multiple current reused frames, for every two adjacent second intermediate frames corresponding to the current reused frame, if the optical flow information between the two adjacent second intermediate frames in time sequence exists in the current optical flow information to be utilized, obtain the optical flow information between the two adjacent second intermediate frames in time sequence from the current optical flow information to be utilized; otherwise, obtain the optical flow information between the two adjacent second intermediate frames in time sequence from the current new optical flow information to obtain the second optical flow information corresponding to the current reused frame; when the current new frame exists in the second intermediate frame corresponding to the current reused frame and there is one current reused frame, obtain the optical flow information between every two adjacent second intermediate frames corresponding to the current reused frame from the current new optical flow information as the second optical flow information corresponding to the current reused frame.

[0177] Optionally, the apparatus further includes: a first update module, configured to, after the set determination module 301 determines the current set of video frames to be aligned from the video to be processed according to the timing of video frames, update the video frames recorded at each position in the current video frame queue in sequence according to the timing of each video frame in the current set of video frames to be aligned; wherein, each video frame in the current associated video frame set is recorded in the current video frame queue; for every two adjacent positions in the video frame queue, they respectively correspond to the position in the optical flow information queue for recording the optical flow information between the video frames recorded at these two positions, and the position in the alignment result queue for recording the alignment results of the video frames recorded at these two positions respectively.

[0178] A second update module, configured to, after the calculation module 304 calculates the optical flow information between the current newly added frame and the previous video frame of the newly added frame in the current set of video frames to be aligned as the current newly added optical flow information, for every two adjacent video frames in sequence in the current set of video frames to be aligned, if there is optical flow information between these two adjacent video frames in the current optical flow information to be utilized, obtain the optical flow information between these two adjacent video frames from the current optical flow information to be utilized; otherwise, obtain the optical flow information between these two adjacent video frames from the current newly added optical flow information, to obtain the current updated optical flow information; according to the corresponding relationship between every two adjacent positions in the video frame queue and the positions in the optical flow information queue, update the optical flow information recorded at each position in the current optical flow information queue in sequence with the current updated optical flow information; wherein, the current optical flow information queue records the current optical flow information to be utilized.

[0179] A third update module, configured to, after the alignment module 305 aligns the video frames other than the current reference frame in the current newly added frame to the current reference frame based on the current optical flow information to be utilized and the newly added optical flow information, to obtain the alignment results of the video frames other than the current reference frame in the current newly added frame, update the alignment results of aligning the current associated frames to the current reference frame in sequence according to the corresponding relationship between every two adjacent positions in the video frame queue and the positions in the alignment result queue, to the alignment results recorded at each position in the current alignment result queue; wherein, the current alignment result queue records the alignment results of aligning each associated frame in the current associated video frame set to the reference frame.

[0180] Optionally, the alignment module 305 is specifically configured to: for each current multiplexing frame, determine a third intermediate frame currently corresponding to the multiplexing frame from the current set of video frames to be aligned according to the time sequence of the video frames; wherein the third intermediate frame currently corresponding to the multiplexing frame includes: the multiplexing frame, the current reference frame, and each video frame located between the multiplexing frame and the current reference frame; obtain the optical flow information between each two adjacent third intermediate frames currently corresponding to the multiplexing frame from the current optical flow information to be utilized as the third optical flow information currently corresponding to the multiplexing frame; calculate the optical flow information between the multiplexing frame and the current reference frame according to the third optical flow information currently corresponding to the multiplexing frame as the target optical flow information; and use the target optical flow information to align the multiplexing frame to the current reference frame to obtain the alignment result of the multiplexing frame.

[0181] Optionally, the alignment module 305 is specifically configured to: if there is no current newly added frame in the third intermediate frame currently corresponding to the multiplexing frame, obtain the optical flow information between each two adjacent third intermediate frames currently corresponding to the multiplexing frame from the current optical flow information to be utilized as the third optical flow information currently corresponding to the multiplexing frame; if there is a current newly added frame in the third intermediate frame currently corresponding to the multiplexing frame and there are multiple current multiplexing frames, for each two adjacent third intermediate frames currently corresponding to the multiplexing frame, if there is optical flow information between the two adjacent third intermediate frames in the current optical flow information to be utilized, obtain the optical flow information between the two adjacent third intermediate frames from the current optical flow information to be utilized; otherwise, obtain the optical flow information between the two adjacent third intermediate frames from the current newly added optical flow information to obtain the third optical flow information currently corresponding to the multiplexing frame; if there is a current newly added frame in the third intermediate frame currently corresponding to the multiplexing frame and there is one current multiplexing frame, obtain the optical flow information between each two adjacent third intermediate frames currently corresponding to the multiplexing frame from the current newly added optical flow information as the third optical flow information currently corresponding to the multiplexing frame.

[0182] Optionally, the alignment module 305 is specifically configured to: for each video frame in the current newly added frames except the current reference frame, determine the fourth intermediate frame currently corresponding to the video frame from the current set of video frames to be aligned according to the time sequence of the video frames; wherein, the fourth intermediate frame currently corresponding to the video frame includes: the video frame, the current reference frame, and each video frame located between the video frame and the current reference frame; obtain the optical flow information between every two adjacent fourth intermediate frames currently corresponding to the video frame from the current optical flow information to be utilized and the newly added optical flow information, as the fourth optical flow information currently corresponding to the video frame; if the number of current reused frames in the fourth intermediate frame currently corresponding to the video frame is not greater than 1, align the video frame with the fourth optical flow information currently corresponding to the video frame in sequence according to the time sequence of the video frames until the alignment result of the video frame to the current reference frame is obtained; if there are multiple current reused frames in the fourth intermediate frame currently corresponding to the video frame, determine the fifth intermediate frame currently corresponding to the video frame from the fourth intermediate frame currently corresponding to the video frame according to the time sequence of the video frames; wherein, the fifth intermediate frame currently corresponding to the video frame includes: the current reference frame, the current latest reused frame in time sequence; and each video frame located between the current reference frame and the current latest reused frame in time sequence; obtain the optical flow information between every two adjacent fifth intermediate frames currently corresponding to the video frame from the fourth optical flow information currently corresponding to the video frame, as the fifth optical flow information currently corresponding to the video frame; determine the sixth intermediate frame currently corresponding to the video frame from the fourth intermediate frame currently corresponding to the video frame according to the time sequence of the video frames; wherein, the sixth intermediate frame currently corresponding to the video frame includes: the current latest reused frame in time sequence, the video frame, and each video frame located between the current latest reused frame in time sequence and the video frame; obtain the optical flow information between every two adjacent sixth intermediate frames currently corresponding to the video frame from the fourth optical flow information currently corresponding to the video frame, as the sixth optical flow information currently corresponding to the video frame; align the video frame to the current latest reused frame in time sequence with the sixth optical flow information currently corresponding to the video frame in sequence according to the time sequence of the video frames until the alignment result of the video frame to the current latest reused frame in time sequence is obtained, as the intermediate alignment result currently corresponding to the video frame; align the intermediate alignment result currently corresponding to the video frame to the current reference frame with the fifth optical flow information currently corresponding to the video frame in sequence according to the time sequence of the video frames until the alignment result of the video frame to the current reference frame is obtained.

[0183] Optionally, the alignment module 305 is specifically configured to: if the number of current multiplexed frames in the fourth intermediate frame currently corresponding to the video frame is not greater than 1, obtain the optical flow information between every two adjacent fourth intermediate frames in time sequence corresponding to the video frame from the current newly added optical flow information as the fourth optical flow information corresponding to the video frame; if there are multiple current multiplexed frames in the fourth intermediate frame currently corresponding to the video frame, for every two adjacent fourth intermediate frames in time sequence corresponding to the video frame, if the optical flow information between the two adjacent fourth intermediate frames in time sequence exists in the current optical flow information to be utilized, obtain the optical flow information between the two adjacent fourth intermediate frames in time sequence from the current optical flow information to be utilized; otherwise, obtain the optical flow information between the two adjacent fourth intermediate frames in time sequence from the current newly added optical flow information to obtain the fourth optical flow information corresponding to the video frame.

[0184] Optionally, the alignment module 305 is specifically configured to: for each video frame other than the current reference frame in the current newly added frames, determine the seventh intermediate frame currently corresponding to the video frame from the current set of video frames to be aligned according to the time sequence of the video frames; wherein, the seventh intermediate frame currently corresponding to the video frame includes: the video frame, the current reference frame, and each video frame located between the video frame and the current reference frame; obtain the optical flow information between every two adjacent seventh intermediate frames in time sequence corresponding to the video frame from the current optical flow information to be utilized and the newly added optical flow information as the seventh optical flow information corresponding to the video frame; calculate the optical flow information between the video frame and the current reference frame according to the seventh optical flow information corresponding to the video frame as the optical flow information to be utilized; use the optical flow information to be utilized to align the video frame to the current reference frame to obtain the alignment result of the video frame.

[0185] Optionally, the alignment module 305 is specifically configured to: when the number of current multiplexed frames in the seventh intermediate frame currently corresponding to the video frame is not greater than 1, obtain the optical flow information between every two adjacent seventh intermediate frames in time sequence corresponding to the video frame from the current newly added optical flow information as the seventh optical flow information corresponding to the video frame; when there are multiple current multiplexed frames in the seventh intermediate frame currently corresponding to the video frame, if the optical flow information between the two adjacent seventh intermediate frames in time sequence exists in the current optical flow information to be utilized, obtain the optical flow information between the two adjacent seventh intermediate frames in time sequence from the current optical flow information to be utilized; otherwise, obtain the optical flow information between the two adjacent seventh intermediate frames in time sequence from the current newly added optical flow information to obtain the seventh optical flow information corresponding to the video frame.

[0186] Optionally, the calculation module 304 is specifically configured to: for each newly added frame currently, input the newly added frame and the previous video frame of the newly added frame in the current set of video frames to be aligned into a pre-trained optical flow prediction model, to obtain the newly added optical flow information between the newly added frame and the previous video frame of the newly added frame in the current set of video frames to be aligned; wherein, the optical flow prediction model is trained based on sample video frame pairs and sample labels representing the optical flow information between the sample video frame pairs.

[0187] Optionally, the number of associated frames of an associated frame in a set of video frames is not less than 3; in the case where the number of associated frames is an even number, among the associated frames included in the set of video frames, the number of frames of the associated frames earlier in time than the reference frame in the set of video frames is the same as the number of frames of the associated frames later in time than the reference frame in the set of video frames; in the case where the number of associated frames is an odd number, among the associated frames included in the set of video frames, the difference between the number of frames of the associated frames earlier in time than the reference frame in the set of video frames and the number of frames of the associated frames later in time than the reference frame in the set of video frames is 1.

[0188] An embodiment of the present invention also provides an electronic device, as Figure 4 shown, including a processor 401, a communication interface 402, a memory 403, and a communication bus 404. Among them, the processor 401, the communication interface 402, and the memory 403 complete communication with each other through the communication bus 404. The memory 403 is used to store a computer program; the processor 401 is configured to, when executing the program stored on the memory 403, implement the steps of any one of the above video frame alignment methods.

[0189] The communication bus mentioned in the above electronic device may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience of representation, only a thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface is used for communication between the above electronic device and other devices. The memory may include a Random Access Memory (RAM), and may also include a Non-Volatile Memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located far from the aforementioned processor. The above processor may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0190] In another embodiment provided by the present invention, there is also provided a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, the steps of any of the above video frame alignment methods are implemented.

[0191] In another embodiment provided by the present invention, there is also provided a computer program product containing instructions, which when running on a computer, causes the computer to execute any of the video frame alignment methods in the above embodiments.

[0192] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present invention are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, server, or data center to another website, computer, server, or data center by wire (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer, or a data storage device such as a server or data center that includes one or more integrated available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)).

[0193] It should be noted that in this document, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise", or any other variation thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device that includes a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article, or device that includes the element.

[0194] Each embodiment in this specification is described in a related manner. For the same or similar parts among the embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the embodiments of the device, electronic device, computer-readable storage medium, and computer program product, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiments.

[0195] The above are only the preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention are all included in the protection scope of the present invention.

Claims

1. A video frame alignment method, characterized in that, The method includes: Determining a current set of video frames to be aligned from a video to be processed according to the time sequence of video frames; wherein, a set of video frames includes a reference frame and associated frames that need to be aligned to the reference frame; Obtaining the optical flow information between every two temporally adjacent video frames in the current set of associated video frames obtained when aligning the current set of associated video frames, as the current optical flow information to be utilized; wherein, the current set of associated video frames is the set of video frames when the multiple video frames in the current set of video frames to be aligned participated in alignment last time; Determining the video frames that do not belong to the current set of associated video frames from the current set of video frames to be aligned, as the current newly added frames; Calculating the optical flow information between each current newly added frame and the previous video frame of the newly added frame in the current set of video frames to be aligned, as the current newly added optical flow information; Based on the current optical flow information to be utilized and the newly added optical flow information, aligning each current associated frame to the current reference frame respectively to obtain the alignment result of each current associated frame; return to execute the step of determining the current set of video frames to be aligned from the video to be processed according to the time sequence of video frames.

2. The method according to claim 1, wherein The step of aligning each current associated frame to the current reference frame respectively based on the current optical flow information to be utilized and the newly added optical flow information to obtain the alignment result of each current associated frame includes: Determining the video frames that belong to the current set of associated video frames from the current associated frames, as the current reused frames; Aligning the current reused frames to the current reference frame based on the current optical flow information to be utilized to obtain the alignment result of the current reused frames; Aligning the video frames other than the current reference frame in the current newly added frames to the current reference frame based on the current optical flow information to be utilized and the newly added optical flow information to obtain the alignment result of the video frames other than the current reference frame in the current newly added frames.

3. The method according to claim 2, characterized in that, The step of aligning the current reused frames to the current reference frame based on the current optical flow information to be utilized to obtain the alignment result of the current reused frames includes: Determining a current first intermediate frame from the current set of video frames to be aligned according to the time sequence of video frames; wherein, the current first intermediate frame includes: the reference frame in the current set of associated video frames, the current reference frame, and each video frame between the reference frame in the current set of associated video frames and the current reference frame; obtaining the optical flow information between every two temporally adjacent first intermediate frames from the current optical flow information to be utilized, as the current first optical flow information; For each current reused frame, when the time sequence of the reused frame is earlier than the reference frame in the current set of associated video frames, obtaining the alignment result of the reused frame to the reference frame in the current set of associated video frames obtained when aligning the current set of associated video frames, as the current alignment result to be utilized; sequentially using the current first optical flow information to align the current alignment result to be utilized to the current reference frame according to the time sequence of video frames until the alignment result of the reused frame to the current reference frame is obtained; When the multiplexed frame is the reference frame in the current set of associated video frames, align the multiplexed frame to the current reference frame in sequence according to the time sequence of the video frames using the current first optical flow information, and obtain the alignment result of the multiplexed frame; When the time sequence of the multiplexed frame is later than the reference frame in the current set of associated video frames and earlier than the current reference frame, determine the second intermediate frame currently corresponding to the multiplexed frame from the current set of video frames to be aligned according to the time sequence of the video frames; wherein, the second intermediate frame currently corresponding to the multiplexed frame includes: the multiplexed frame, the current reference frame, and each video frame between the multiplexed frame and the current reference frame; obtain the optical flow information between every two adjacent second intermediate frames currently corresponding to the multiplexed frame from the current optical flow information to be utilized as the second optical flow information currently corresponding to the multiplexed frame; align the multiplexed frame to the current reference frame in sequence according to the time sequence of the video frames using the second optical flow information currently corresponding to the multiplexed frame, and obtain the alignment result of the multiplexed frame; When the time sequence of the multiplexed frame is later than the current reference frame, obtain the alignment result of the multiplexed frame to the current reference frame obtained when aligning the current set of associated video frames recorded in advance.

4. The method according to claim 3, wherein The obtaining the optical flow information between every two adjacent first intermediate frames currently as the current first optical flow information from the current optical flow information to be utilized includes: When there is no current newly added frame in the current first intermediate frame, obtain the optical flow information between every two adjacent first intermediate frames currently from the current optical flow information to be utilized as the current first optical flow information; When there is a current newly added frame in the current first intermediate frame and there are multiple current multiplexed frames, for every two adjacent first intermediate frames currently, if there is optical flow information between the two adjacent first intermediate frames in the current optical flow information to be utilized, obtain the optical flow information between the two adjacent first intermediate frames from the current optical flow information to be utilized; otherwise, obtain the optical flow information between the two adjacent first intermediate frames from the current newly added optical flow information to obtain the current first optical flow information; When there is a current newly added frame in the current first intermediate frame and there is one current multiplexed frame, obtain the optical flow information between every two adjacent first intermediate frames currently from the current newly added optical flow information as the current first optical flow information; and / or The obtaining the optical flow information between every two adjacent second intermediate frames currently corresponding to the multiplexed frame from the current optical flow information to be utilized includes: When there is a current newly added frame in the second intermediate frame currently corresponding to the multiplexed frame, obtain the optical flow information between every two adjacent second intermediate frames currently corresponding to the multiplexed frame from the current optical flow information to be utilized as the second optical flow information currently corresponding to the multiplexed frame; When there is a current newly added frame in the second intermediate frame currently corresponding to the multiplexing frame and there are multiple current multiplexing frames, for every two temporally adjacent second intermediate frames currently corresponding to the multiplexing frame, if there is optical flow information between the two temporally adjacent second intermediate frames in the currently available optical flow information, obtain the optical flow information between the two temporally adjacent second intermediate frames from the currently available optical flow information; otherwise, obtain the optical flow information between the two temporally adjacent second intermediate frames from the currently newly added optical flow information to obtain the second optical flow information currently corresponding to the multiplexing frame. When there is a current newly added frame in the second intermediate frame currently corresponding to the multiplexing frame and there is one current multiplexing frame, obtain the optical flow information between every two temporally adjacent second intermediate frames currently corresponding to the multiplexing frame from the currently newly added optical flow information as the second optical flow information currently corresponding to the multiplexing frame.

5. The method according to claim 3, characterized in that After determining the currently to-be-aligned video frame set from the video to be processed according to the time sequence of video frames, the method further includes: According to the time sequence of each video frame in the currently to-be-aligned video frame set, sequentially update the video frames recorded at each position in the currently video frame queue with the video frames in the currently to-be-aligned video frame set; wherein, each video frame in the currently associated video frame set is recorded in the currently video frame queue; for every two adjacent positions in the video frame queue, it corresponds to the position in the optical flow information queue for recording the optical flow information between the video frames recorded at the two positions, and the position in the alignment result queue for recording the respective alignment results of the video frames recorded at the two positions. After calculating the optical flow information between each current newly added frame and the previous video frame of the newly added frame in the currently to-be-aligned video frame set as the currently newly added optical flow information, the method further includes: For every two temporally adjacent video frames in the currently to-be-aligned video frame set, if there is optical flow information between the two temporally adjacent video frames in the currently available optical flow information, obtain the optical flow information between the two temporally adjacent video frames from the currently available optical flow information; otherwise, obtain the optical flow information between the two temporally adjacent video frames from the currently newly added optical flow information to obtain the currently updated optical flow information. According to the corresponding relationship between every two adjacent positions in the video frame queue and the positions in the optical flow information queue, sequentially update the optical flow information recorded at each position in the currently optical flow information queue with the currently updated optical flow information; wherein, the currently available optical flow information is recorded in the currently optical flow information queue. After aligning the video frames in the current newly added frame except the current reference frame to the current reference frame based on the currently available optical flow information and the newly added optical flow information to obtain the alignment results of the video frames in the current newly added frame except the current reference frame, the method further includes: According to the corresponding relationship between every two adjacent positions in the video frame queue and the positions in the alignment result queue, update the alignment results recorded at each position in the current alignment result queue in sequence with the alignment results of aligning the current associated frames to the current reference frame; wherein, the alignment results of aligning each associated frame in the current associated video frame set to the reference frame are recorded in the current alignment result queue.

6. The method according to claim 2, wherein The aligning the current multiplexing frame to the current reference frame based on the current optical flow information to be utilized to obtain the alignment result of the current multiplexing frame includes: For each current multiplexing frame, determine the third intermediate frame currently corresponding to the multiplexing frame from the current set of video frames to be aligned according to the time sequence of the video frames; wherein, the third intermediate frame currently corresponding to the multiplexing frame includes: the multiplexing frame, the current reference frame, and each video frame located between the multiplexing frame and the current reference frame; Obtain the optical flow information between every two adjacent third intermediate frames in time sequence corresponding to the multiplexing frame from the current optical flow information to be utilized as the third optical flow information corresponding to the multiplexing frame; Calculate the optical flow information between the multiplexing frame and the current reference frame according to the third optical flow information corresponding to the multiplexing frame as the target optical flow information; Use the target optical flow information to align the multiplexing frame to the current reference frame to obtain the alignment result of the multiplexing frame.

7. The method according to claim 6, wherein The obtaining the optical flow information between every two adjacent third intermediate frames in time sequence corresponding to the multiplexing frame from the current optical flow information to be utilized as the third optical flow information corresponding to the multiplexing frame includes: If there is no current newly added frame in the third intermediate frame currently corresponding to the multiplexing frame, obtain the optical flow information between every two adjacent third intermediate frames in time sequence corresponding to the multiplexing frame from the current optical flow information to be utilized as the third optical flow information corresponding to the multiplexing frame; If there is a current newly added frame in the third intermediate frame currently corresponding to the multiplexing frame and there are multiple current multiplexing frames, for every two adjacent third intermediate frames in time sequence corresponding to the multiplexing frame, if there is optical flow information between the two adjacent third intermediate frames in the current optical flow information to be utilized, obtain the optical flow information between the two adjacent third intermediate frames from the current optical flow information to be utilized; otherwise, obtain the optical flow information between the two adjacent third intermediate frames from the current newly added optical flow information to obtain the third optical flow information corresponding to the multiplexing frame; If there is a current newly added frame in the third intermediate frame currently corresponding to the multiplexing frame and there is one current multiplexing frame, obtain the optical flow information between every two adjacent third intermediate frames in time sequence corresponding to the multiplexing frame from the current newly added optical flow information as the third optical flow information corresponding to the multiplexing frame.

8. The method according to claim 2, wherein The aligning the video frames other than the current reference frame in the current newly added frames to the current reference frame based on the current optical flow information to be utilized and the newly added optical flow information to obtain the alignment results of the video frames other than the current reference frame in the current newly added frames includes: For each video frame in the current newly added frames except the current reference frame, determine the fourth intermediate frame currently corresponding to the video frame from the current set of video frames to be aligned according to the time sequence of the video frames; wherein, the fourth intermediate frame currently corresponding to the video frame includes: the video frame, the current reference frame, and each video frame between the video frame and the current reference frame; Obtain the optical flow information between every two adjacent fourth intermediate frames currently corresponding to the video frame from the current optical flow information to be utilized and the newly added optical flow information, as the fourth optical flow information currently corresponding to the video frame; If the number of current reused frames in the fourth intermediate frame currently corresponding to the video frame is not greater than 1, align the video frame with the fourth optical flow information currently corresponding to the video frame in sequence according to the time sequence of the video frames until the alignment result of the video frame to the current reference frame is obtained; If there are multiple current reused frames in the fourth intermediate frame currently corresponding to the video frame, determine the fifth intermediate frame currently corresponding to the video frame from the fourth intermediate frame currently corresponding to the video frame according to the time sequence of the video frames; wherein, the fifth intermediate frame currently corresponding to the video frame includes: the current reference frame, the current reused frame with the latest time sequence; and each video frame between the current reference frame and the current reused frame with the latest time sequence; Obtain the optical flow information between every two adjacent fifth intermediate frames currently corresponding to the video frame from the fourth optical flow information currently corresponding to the video frame, as the fifth optical flow information currently corresponding to the video frame; Determine the sixth intermediate frame currently corresponding to the video frame from the fourth intermediate frame currently corresponding to the video frame according to the time sequence of the video frames; wherein, the sixth intermediate frame currently corresponding to the video frame includes: the current reused frame with the latest time sequence, the video frame, and each video frame between the current reused frame with the latest time sequence and the video frame; Obtain the optical flow information between every two adjacent sixth intermediate frames currently corresponding to the video frame from the fourth optical flow information currently corresponding to the video frame, as the sixth optical flow information currently corresponding to the video frame; Align the video frame to the current reused frame with the latest time sequence in sequence with the sixth optical flow information currently corresponding to the video frame according to the time sequence of the video frames until the alignment result of the video frame to the current reused frame with the latest time sequence is obtained, as the intermediate alignment result currently corresponding to the video frame; Align the intermediate alignment result currently corresponding to the video frame to the current reference frame in sequence with the fifth optical flow information currently corresponding to the video frame according to the time sequence of the video frames until the alignment result of the video frame to the current reference frame is obtained.

9. The method according to claim 8, characterized in that The obtaining the optical flow information between every two adjacent fourth intermediate frames currently corresponding to the video frame from the current optical flow information to be utilized and the newly added optical flow information, as the fourth optical flow information currently corresponding to the video frame, includes: If the number of current multiplexed frames existing in the current fourth intermediate frame corresponding to the video frame is no more than 1, obtain the optical flow information between every two temporally adjacent fourth intermediate frames corresponding to the video frame from the current newly added optical flow information as the fourth optical flow information corresponding to the video frame; If there are multiple current multiplexed frames in the current fourth intermediate frame corresponding to the video frame, for every two temporally adjacent fourth intermediate frames corresponding to the video frame, if there is optical flow information between the two temporally adjacent fourth intermediate frames in the current optical flow information to be utilized, obtain the optical flow information between the two temporally adjacent fourth intermediate frames from the current optical flow information to be utilized; otherwise, obtain the optical flow information between the two temporally adjacent fourth intermediate frames from the current newly added optical flow information to obtain the fourth optical flow information corresponding to the video frame.

10. The method according to claim 2, wherein The aligning, based on the current optical flow information to be utilized and the newly added optical flow information, of the video frames other than the current reference frame in the current newly added frame to the current reference frame to obtain the alignment result of the video frames other than the current reference frame in the current newly added frame includes: For each video frame other than the current reference frame in the current newly added frame, determine the current seventh intermediate frame corresponding to the video frame from the current set of video frames to be aligned according to the time sequence of the video frames; wherein, the current seventh intermediate frame corresponding to the video frame includes: the video frame, the current reference frame, and each video frame located between the video frame and the current reference frame; Obtain the optical flow information between every two temporally adjacent seventh intermediate frames corresponding to the video frame from the current optical flow information to be utilized and the newly added optical flow information as the seventh optical flow information corresponding to the video frame; Calculate the optical flow information between the video frame and the current reference frame according to the seventh optical flow information corresponding to the video frame as the optical flow information to be utilized; Use the optical flow information to be utilized to align the video frame to the current reference frame to obtain the alignment result of the video frame.

11. The method according to claim 10, characterized in that, The obtaining, from the current optical flow information to be utilized and the newly added optical flow information, of the optical flow information between every two temporally adjacent seventh intermediate frames corresponding to the video frame as the seventh optical flow information corresponding to the video frame includes: In the case where the number of current multiplexed frames existing in the current seventh intermediate frame corresponding to the video frame is no more than 1, obtain the optical flow information between every two temporally adjacent seventh intermediate frames corresponding to the video frame from the current newly added optical flow information as the seventh optical flow information corresponding to the video frame; In the case where there are multiple current multiplexed frames in the current seventh intermediate frame corresponding to the video frame, if there is optical flow information between the two temporally adjacent seventh intermediate frames in the current optical flow information to be utilized, obtain the optical flow information between the two temporally adjacent seventh intermediate frames from the current optical flow information to be utilized; otherwise, obtain the optical flow information between the two temporally adjacent seventh intermediate frames from the current newly added optical flow information to obtain the seventh optical flow information corresponding to the video frame.

12. The method according to claim 1, characterized in that, Calculating the optical flow information between each current newly added frame and the previous video frame of the newly added frame in the current set of video frames to be aligned as the current newly added optical flow information, including: For each current newly added frame, inputting the newly added frame and the previous video frame of the newly added frame in the current set of video frames to be aligned into a pre-trained optical flow prediction model to obtain the newly added optical flow information between the newly added frame and the previous video frame of the newly added frame in the current set of video frames to be aligned; wherein, the optical flow prediction model is trained based on sample video frame pairs and sample labels representing the optical flow information between the sample video frame pairs.

13. The method according to claim 1, wherein The number of associated frames of an associated frame in a video frame set is not less than 3; In the case where the number of associated frames is an even number, among the associated frames included in the video frame set, the number of frames of the associated frames earlier in time sequence than the reference frame in the video frame set is the same as the number of frames of the associated frames later in time sequence than the reference frame in the video frame set; In the case where the number of associated frames is an odd number, among the associated frames included in the video frame set, the difference between the number of frames of the associated frames earlier in time sequence than the reference frame in the video frame set and the number of frames of the associated frames later in time sequence than the reference frame in the video frame set is 1.

14. A video frame alignment device, characterized in that, The device includes: A set determination module, configured to determine the current set of video frames to be aligned from the video to be processed according to the time sequence of the video frames; wherein, a video frame set includes a reference frame and associated frames that need to be aligned to the reference frame; An information acquisition module, configured to acquire the optical flow information between every two adjacent video frames in time sequence in the current set of associated video frames obtained when aligning the current set of associated video frames as the current optical flow information to be utilized; wherein, the current set of associated video frames is the set of video frames when multiple video frames in the current set of video frames to be aligned participated in alignment last time; A newly added frame determination module, configured to determine the video frames that do not belong to the current set of associated video frames from the current set of video frames to be aligned as the current newly added frames; A calculation module, configured to calculate the optical flow information between each current newly added frame and the previous video frame of the newly added frame in the current set of video frames to be aligned as the current newly added optical flow information; An alignment module, configured to align each current associated frame to the current reference frame based on the current optical flow information to be utilized and the newly added optical flow information to obtain the alignment result of each current associated frame; return to execute the step of determining the current set of video frames to be aligned from the video to be processed according to the time sequence of the video frames.

15. An electronic device, characterized in that, Including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory complete communication with each other through the communication bus; The memory is used for storing a computer program; The processor is configured to implement the method according to any one of claims 1-13 when executing the program stored on the memory.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and the computer program, when executed by the processor, implements the method according to any one of claims 1-13.

Images