Method performed by electronic device, electronic device, storage medium, and program product

Abstract

Embodiments of the present disclosure provide a method performed by an electronic device, an electronic device, a storage medium and a program product, relating to the fields of video processing, artificial intelligence and the like. The method comprises: obtaining image features of at least two images of a video, and first optical flow information between the at least two images; determining at least two first windows for dividing the image features; adjusting the at least two first windows based on the first optical flow information to obtain at least two second windows; and performing deblurring processing on the at least two images based on the image features of the at least two images in each second window of the at least two second windows. Embodiments of the present disclosure can more accurately interact the timing information of the moving object, ensure the continuity of the motion in the at least two images, and thus improve the deblurring effect of the video. Optionally, the above-mentioned method performed by the electronic device can be executed using an artificial intelligence model.

Description

Technical Field

[0001] This disclosure relates to the field of video processing technology, and more specifically, to a method executed by an electronic device, an electronic device, a storage medium, and a program product. Background Technology

[0002] With advancements in hardware such as sensors and lenses, the video recording quality of cameras in electronic devices is constantly improving, and users are also demanding higher quality recordings from these devices. However, factors such as hand shake and subject movement can cause motion blur in the recorded video, severely impacting the user experience.

[0003] Currently, most existing solutions to video motion blur rely on hardware improvements. For example, using image stabilization sensors compensates for significant vibrations and motion to capture smooth, stable video. However, this method relies on expensive hardware. Furthermore, when shooting high-speed scenes, whether the subject or the electronic device is moving, the blurring in the video is very severe, and existing image and video deblurring methods are not very effective for shooting high-speed scenes. Summary of the Invention

[0004] The purpose of this disclosure is to solve the problem of how to improve the deblurring effect of videos.

[0005] According to one aspect of the present disclosure, a method performed by an electronic device is provided, the method comprising: Acquire image features of at least two frames of the video, and first optical flow information between at least two frames; Determine at least two first windows for segmenting image features; Based on the first optical flow information, at least two first windows are adjusted to obtain at least two second windows; For each of the at least two second windows, at least two frames of images are deblurred based on the image features of the at least two frames of images within the second window respectively.

[0006] According to another aspect of the embodiments of this disclosure, another method performed by an electronic device is provided, the method comprising: Acquire at least two first frames and at least one second frame of a video, as well as second optical flow information between at least two first frames, wherein the sharpness of the first image is greater than that of the second image; Based on the fourth optical flow information and at least two frames of the first image, at least one frame of the third image is obtained, and the content of the third image corresponds to the content of the second image. At least one third image and at least one second image are fused together to obtain at least one fourth image; Deblurring is performed on at least two frames of the first image and at least one frame of the fourth image.

[0007] According to another aspect of the present disclosure, a method performed by an electronic device is also provided, the method comprising: Acquire at least two first frames and at least one second frame of a video, as well as fourth optical flow information between at least two first frames, wherein the sharpness of the first image is greater than that of the second image; The fourth optical flow information is decomposed to obtain the inter-frame optical flow information between each second image and its adjacent images; The video is processed based on the inter-frame optical flow information corresponding to at least one frame of the second image.

[0008] According to another aspect of the present disclosure, an electronic device is provided, comprising: a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method performed by the electronic device provided in the embodiments of the present disclosure.

[0009] According to another aspect of the present disclosure, a computer-readable storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the method performed by an electronic device provided in the embodiments of the present disclosure.

[0010] According to another aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method performed by an electronic device provided in the embodiments of the present disclosure.

[0011] The present disclosure provides a method, electronic device, storage medium, and program product executed by an electronic device, which acquires image features of at least two frames of a video and first optical flow information between the at least two frames; determines at least two first windows for dividing the image features; adjusts the at least two first windows based on the first optical flow information to obtain at least two second windows; and performs deblurring processing on the at least two frames of the video based on the image features within each of the at least two frames. In this disclosure, the window size is adaptively adjusted according to the optical flow information, dynamically coordinating the movement of each region, avoiding isolated movement of local pixels, and fully considering the coordination of temporal features. This allows for more accurate interaction of temporal information of moving objects within the adjusted second windows, ensuring the continuity of movement in the at least two frames, thereby improving the video deblurring effect. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments of this disclosure will be briefly introduced below.

[0013] Figure 1 A flowchart illustrating a method performed by an electronic device according to an embodiment of this disclosure; Figure 2 A schematic diagram illustrating a motion-guided adaptive window size according to an embodiment of this disclosure; Figure 3 A schematic diagram illustrating a video deblurring process provided in an embodiment of this disclosure; Figure 4 A schematic diagram illustrating a window update according to an embodiment of this disclosure; Figure 5 A schematic diagram of motion-guided adaptive window timing interaction provided in an embodiment of this disclosure; Figure 6 A schematic diagram illustrating motion-guided window coordinate updating provided in an embodiment of this disclosure; Figure 7 This is a schematic diagram illustrating the decomposition of optical flow according to an embodiment of the present disclosure; Figure 8 A schematic diagram of an optical flow decomposition process provided in an embodiment of this disclosure; Figure 9 A schematic diagram illustrating an optical flow decomposition process provided in an embodiment of this disclosure; Figure 10a A schematic diagram of another optical flow decomposition process provided in an embodiment of this disclosure; Figure 10b A schematic diagram illustrating another optical flow decomposition process provided in an embodiment of this disclosure; Figure 11 A schematic diagram illustrating a processing procedure of ControlNet provided in an embodiment of this disclosure; Figure 12 A schematic diagram illustrating a process for updating a blurred frame according to an embodiment of this disclosure; Figure 13 This is a schematic diagram illustrating a complete video demodulation process provided by an embodiment of the present disclosure; Figure 14 A flowchart illustrating another method performed by an electronic device according to an embodiment of this disclosure; Figure 15 A flowchart illustrating yet another method performed by an electronic device according to an embodiment of this disclosure; Figure 16 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0014] The following description, with reference to the accompanying drawings, is provided to aid in a thorough understanding of the various embodiments of this disclosure as defined by the claims and their equivalents. This description includes various specific details to aid understanding but should be considered exemplary only. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the various embodiments described herein without departing from the scope and spirit of this disclosure. Furthermore, for clarity and brevity, descriptions of well-known functions and structures may be omitted.

[0015] The terms and wording used in the following description and claims are not limited to their dictionary meanings, but are merely used by the inventors to enable a clear and consistent understanding of this disclosure. Therefore, it will be apparent to those skilled in the art that the following description of various embodiments of this disclosure is for illustrative purposes only and not for limiting the purpose of this disclosure as defined in the appended claims and their equivalents.

[0016] It should be understood that the singular forms of “a,” “an,” and “the” can also include plural references unless the context clearly indicates otherwise. Thus, for example, the reference to “component surface” includes referring to one or more such surfaces. When we say that an element is “connected” or “coupled” to another element, the element can be directly connected or coupled to the other element, or it can mean that the element and the other element are connected through an intermediate element. Furthermore, the use of “connected” or “coupled” herein can include wireless connections or wireless couplings.

[0017] The terms “comprising” or “may include” refer to the presence of a corresponding disclosed function, operation, or component that may be used in the various embodiments of this disclosure, rather than limiting the presence of one or more additional functions, operations, or features. Furthermore, the terms “comprising” or “having” may be interpreted as indicating certain characteristics, numbers, steps, operations, constituent elements, components, or combinations thereof, but should not be construed as excluding the possibility of the presence of one or more other characteristics, numbers, steps, operations, constituent elements, components, or combinations thereof.

[0018] The term "or" as used in the various embodiments of this disclosure includes any of the listed terms and all combinations thereof. For example, "A or B" may include A, may include B, or may include both A and B. When describing multiple (two or more) items, if the relationship between the multiple items is not explicitly defined, the multiple items may refer to one, more, or all of the multiple items. For example, the description "parameter A includes A1, A2, A3" can be implemented as parameter A includes A1 or A2 or A3, or it can be implemented as parameter A includes at least two of the three items A1, A2, and A3.

[0019] Unless otherwise defined, all terms used in this disclosure (including technical or scientific terms) have the same meaning as understood by those skilled in the art. Common terms as defined in dictionaries are to be interpreted as having a meaning consistent with their context in the relevant technical field and should not be interpreted ideally or overly formally, unless expressly defined in this disclosure.

[0020] At least some of the functions of the device or electronic device provided in this disclosure embodiment can be implemented by an AI model, such as implementing at least one module of a plurality of modules of the device or electronic device by an AI model. AI-related functions can be executed by non-volatile memory, volatile memory, and a processor.

[0021] The processor may include one or more processors. In this case, the one or more processors may be general-purpose processors, such as central processing unit (CPU), application processor (AP), etc., or pure graphics processing unit, such as graphics processing unit (GPU), visual processing unit (VPU), and / or AI-specific processors, such as neural processing unit (NPU).

[0022] The one or more processors control the processing of input data based on predefined operating rules or artificial intelligence (AI) models stored in non-volatile and volatile memory. These predefined operating rules or AI models are provided through training or learning.

[0023] Here, "providing through learning" refers to obtaining predefined operating rules or an AI model with desired characteristics by applying a learning algorithm to multiple learning datasets. This learning can be performed within the device or electronic device itself, in which the AI ​​is executed according to the embodiment, and / or can be implemented via a separate server / system.

[0024] AI models can contain multiple neural network layers. Each layer has multiple weight values, and each layer performs neural network computations by calculating the input data of that layer (such as the computation results of the previous layer and / or the input data of the AI ​​model) and the multiple weight values ​​of the current layer. Examples of neural networks include, but are not limited to, convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), generative adversarial networks (GANs), and deep Q-networks.

[0025] A learning algorithm is a method of training a predetermined target device (e.g., a robot) using multiple learning data sets to enable, allow, or control the target device to make determinations or predictions. Examples of such learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0026] The methods provided in this disclosure may relate to one or more fields in the technical fields of speech, language, image, video, or data intelligence.

[0027] Optionally, in the context of speech or language, according to this disclosure, in a method performed by an electronic device, a method for recognizing user speech and interpreting user intent may receive a speech signal as an analog signal via a speech acquisition device (e.g., a microphone) of the electronic device, and convert the speech portion into computer-readable text using an Automatic Speech Recognition (ASR) model. The user's utterance intent can be obtained by interpreting the converted text using a Natural Language Understanding (NLU) model. The ASR model or NLU model can be an artificial intelligence model. The artificial intelligence model can be processed by a dedicated artificial intelligence processor designed in a hardware architecture specified for processing the artificial intelligence model. The artificial intelligence model can be obtained through training. Here, "obtained through training" means obtaining a predefined operating rule or artificial intelligence model configured to perform desired features (or purposes) by training a basic artificial intelligence model with multiple training data through a training algorithm. Language understanding is a technique for recognizing and applying / processing human language / text, including, for example, natural language processing, machine translation, dialogue systems, question answering, or speech recognition / synthesis.

[0028] Optionally, in the context of images or videos, according to this disclosure, in methods performed in electronic devices, the method for video deblurring can obtain output data that identifies the image or optical flow information, image features, etc., in the image by using image or video data as input data for an artificial intelligence model. The artificial intelligence model can be obtained through training. Here, "obtained through training" means obtaining a predefined operating rule or artificial intelligence model configured to perform desired features (or purposes) by training a basic artificial intelligence model with multiple training data using a training algorithm. The methods of this disclosure can relate to the field of visual understanding in artificial intelligence technology, which is a technology for recognizing and processing things like human vision, and includes, for example, object recognition, object tracking, image retrieval, human recognition, scene recognition, 3D reconstruction / localization, or image enhancement.

[0029] Optionally, in the context of data intelligence processing, according to this disclosure, in methods executed in electronic devices, methods for reasoning or predicting motion information, optical flow information, windows, etc., can be recommended / executed using an artificial intelligence model by using motion trajectory data. The processor of the electronic device can perform preprocessing operations on the data to transform it into a form suitable for use as input to an artificial intelligence model. The artificial intelligence model can be obtained through training. Here, "obtained through training" means obtaining a predefined operating rule or artificial intelligence model configured to perform desired features (or purposes) by training a basic artificial intelligence model with multiple training data using a training algorithm. Reasoning and prediction are techniques for logical reasoning and prediction based on determined information, including, for example, knowledge-based reasoning, optimization prediction, preference-based planning, or recommendation.

[0030] To make the objectives, technical solutions, and advantages of this disclosure clearer, the technical solutions of the embodiments of this disclosure and the technical effects produced by the technical solutions of this disclosure are described below through the description of several exemplary embodiments. It should be noted that the following embodiments can be referred to, learned from, or combined with each other, and the same terms, similar features, and similar implementation steps in different embodiments will not be described again.

[0031] This disclosure provides a method executed by an electronic device, such as... Figure 1 As shown, the method includes: Step S110: Obtain image features of at least two frames of the video, and first optical flow information between at least two frames.

[0032] In this embodiment of the disclosure, the video may be recorded by the user in real time, selected by the user from the album, or received by the user from the network or other devices, etc. This embodiment of the disclosure does not specifically limit the method of obtaining the video.

[0033] In this embodiment of the disclosure, the video may be obtained by directly recording various objects such as people, animals, plants, landscapes, buildings, and objects and / or their movement process, or it may be obtained by synthesizing multiple video data or image data, or it may be a video segment obtained by splitting a complete video according to certain rules, such as according to the clarity or blur of the video frame. This embodiment of the disclosure does not specifically limit the content and source of the video.

[0034] In the case of splitting a complete video into multiple video segments, each video segment can be executed in accordance with the solution provided in the embodiments of this disclosure. That is, for a complete video, the solution in the embodiments of this disclosure can be executed multiple times to achieve video processing.

[0035] In this embodiment of the disclosure, the video may include a continuous sequence of image frames (also referred to simply as frames, and in some embodiments may be replaced by frame images or images). This embodiment of the disclosure does not specifically limit the number of image frames in the video. Among them, at least two image frames are the image frames to be processed. Those skilled in the art can set the method of determining at least two image frames according to the actual situation, such as all images or some images in the video, etc. This embodiment of the disclosure does not limit this.

[0036] It is understood that a moving scene refers to a subject in motion (e.g., filming a moving vehicle, but not limited to this), or an electronic device being used for filming (e.g., a person filming while skiing, but not limited to this), both of which can cause blurry video. High-speed moving scenes refer to subjects or electronic devices moving at high speeds, which may produce even greater video blur; this blur can also be called large motion blur, and high-speed moving scenes can also be called large motion scenes. In this embodiment of the disclosure, first optical flow information is used to reflect the motion information in the moving scene within the video image.

[0037] In this embodiment of the disclosure, the method for image feature extraction and the determination of the first optical flow information are not specifically limited. Those skilled in the art can adopt appropriate determination methods according to the actual situation. For example, the first optical flow information can be predicted using existing optical flow prediction methods or using the optical flow prediction methods provided in the embodiments of this disclosure below. As another example, image features can be obtained using existing extraction methods or using the clear frame prior reference method provided in the embodiments of this disclosure below; this embodiment of the disclosure does not limit these methods.

[0038] Step S120: Determine at least two first windows for segmenting image features.

[0039] In this embodiment of the disclosure, for the image to be processed, its image features can be divided into several first windows of a pre-defined size. The first window can refer to a window after initial division. Optionally, image features of each image covered within each first window can be sampled. Optionally, the window configuration (e.g., number of windows, window size, window position, etc.) corresponding to at least two frames of the video can be the same. Since the blur in the video usually has spatiotemporal locality (e.g., motion blur is concentrated near the trajectory of a moving object), at least two first windows can divide the video into local spatial locations, enabling the model to focus on the blur features of local areas, avoiding redundancy in global computation (e.g., information exchange between two completely unrelated spatial points), reducing computational complexity, and saving computational resources.

[0040] Step S130: Based on the first optical flow information, adjust at least two first windows to obtain at least two second windows.

[0041] In this embodiment, based on dividing the video into at least two first windows, and considering that in a moving scene, it cannot be guaranteed that the same object will always be in the same spatial position, the information interaction between fixed-size windows may not be accurate. To improve the reliability of temporal information interaction, the first windows are adjusted based on the first optical flow information, because the first optical flow information can capture the current motion trend, resulting in a second window that can cover the object. The second window can refer to a window adaptively adjusted according to the first optical flow information, or it can be understood as an adaptive window size based on motion-guided principles. Correspondingly, the sampling points within the second window are also adaptively adjusted. Alternatively, the image features covered within each second window can be directly sampled from each second window. Optionally, the adjusted window configurations corresponding to at least two frames of the video can be the same or different.

[0042] As an example, such as Figure 2 As shown, assuming four pre-defined first windows are determined, the size of the four windows can be adjusted by using motion-guided adaptive window size adjustment, so that the adjusted second window (taking the window in the lower right corner as an example) can cover the moving object.

[0043] Step S140: For each of the at least two second windows, deblur the at least two frames of images based on the image features of the at least two frames of images within the second window respectively.

[0044] Specifically, information can be exchanged between the image features of at least two frames within a second window to deblur the at least two frames.

[0045] In this embodiment, information interaction within the second window can be understood as the interaction of image features (image features of at least two images of the video) at different temporal intervals corresponding to each second window, or simply temporal information interaction. Optionally, the specific interaction method may include, but is not limited to, using optical flow, warping, and some temporal operations to interact with the image features at different temporal intervals corresponding to each second window. The temporal operations may include, but are not limited to, using temporal attention for information interaction to achieve adaptive window temporal information interaction, thereby ensuring the continuity of motion.

[0046] Optionally, the processing steps S120 to S140 described above can be used in the video deblurring process, such as... Figure 3 As shown, this processing can be used as a motion-guided adaptive window temporal interaction layer, cascaded with convolutional layers and spatial interaction layers (for interacting spatial location information in each frame of the image), for example, embedded after the spatial interaction layer, that is, after the spatial feature interaction, the temporal information interaction is performed. After encoding and decoding at multiple scales, video deblurring can be achieved.

[0047] The method performed by an electronic device provided in this disclosure can adaptively adjust the window size according to optical flow information, dynamically coordinate the movement of each region, avoid isolated movement of local pixels, and fully consider the coordination of temporal features. This allows for more accurate interaction of temporal information of moving objects within the adjusted second window, ensuring the continuity of motion in at least two frames of images, thereby improving the video deblurring effect.

[0048] In this embodiment of the disclosure, a feasible implementation of step S130 is provided, which specifically may include: Step S131: Based on the first optical flow information, determine the second optical flow information and the third optical flow information between at least two frames of images, wherein the second optical flow information is the optical flow information from the earlier image to the later image in the at least two frames of images, and the third optical flow information is the optical flow information from the later image to the earlier image in the at least two frames of images.

[0049] The second optical flow information, also known as forward optical flow information, can be used to capture the current motion trend. The third optical flow information, also known as reverse optical flow information, can be used to correct historical motion errors.

[0050] Step S132: Adjust at least two first windows based on the second optical flow information and the third optical flow information.

[0051] In this embodiment of the disclosure, by combining the second optical flow information and the third optical flow information to adjust at least two first windows, it can be ensured that the adjusted second window is still within the effective range of the feature and covers the corrected motion.

[0052] In this embodiment of the disclosure, a feasible implementation of step S132 is provided, which specifically may include: Step S1321: Based on the second optical flow information and the third optical flow information, use a cross-attention network to determine the correction information corresponding to the second optical flow information; Optionally, query features (Q) are extracted from the second optical flow information, and key features (K) and value features (V) are extracted from the third optical flow information. These features are input into a cross attention network, which can output correction information for the second optical flow information extracted from the third optical flow information.

[0053] Alternatively, the processing procedure of the cross-attention network can be represented as follows: Formula 1 Where K, Q, and V represent key features, query features, and value features, respectively; T represents matrix transpose; This represents the dimension of the K-feature.

[0054] Step S1322: Correct the second optical flow information based on the correction information; For example, the correction information can be added to the original second optical flow information to obtain the overall optical flow information, thereby achieving the purpose of using the third optical flow information to correct historical motion errors.

[0055] Step S1323: Adjust at least two first windows based on the corrected second optical flow information.

[0056] In this embodiment of the disclosure, adjusting at least two first windows based on the corrected second optical flow information can ensure that the adjusted second window is still within the effective range of the feature and covers the corrected motion.

[0057] In this embodiment of the disclosure, a feasible implementation method for step S130 is also provided, which specifically may include: Step SA1: Based on the first optical flow information, determine the scale change and offset corresponding to at least two first windows respectively; Specifically, the scale changes and offsets corresponding to at least two first windows can be determined based on the second and third optical flow information. For details on the implementation, please refer to the description of steps S1321-S1323, which will not be repeated here.

[0058] In this embodiment of the disclosure, predetermined optical flow information is used to calculate the scale change and offset corresponding to each window. The scale change represents the degree of scaling of the window adjustment, and the offset represents the positional difference of the window adjustment.

[0059] Optionally, each first window corresponds to a grid coordinate system, from which features contained within that window can be sampled. The grid size for the entire image features can be represented as:

[0060] in, This indicates the set window size. Indicates the height of the image. Indicates the width of the image, 2 These represent the number of features sampled on the horizontal and vertical axes, respectively.

[0061] Therefore, the scale change and offset corresponding to at least two first windows can refer to the scale and offset of the grid coordinates.

[0062] Step SA2: For each of the at least two first windows, adjust the first window based on the corresponding scale change and offset to obtain the corresponding second window.

[0063] Optionally, the grid coordinates can be updated by scaling and offsetting, and the update process can be represented as follows: Formula 2 based on You will then get a new second window.

[0064] As an example, such as Figure 4 As shown, assuming four pre-defined first windows of a specific size are determined, taking the bottom right window as an example, the coordinates of the first window can be represented as follows: Figure 4 The left side shows The grid size. After updating the bottom right window, the coordinates of the second window changed, such as... Figure 4 The right side shows The size of the grid (becomes a hollow circle).

[0065] In this embodiment of the disclosure, for Figure 3 The motion-guided adaptive window timing interaction layer in [the context of the text] provides an optional implementation method, specifically, as follows: Figure 5As shown, the process may include, but is not limited to, the following components: a window grid generation module, a window grid update module, and a window temporal attention module. The window grid generation module divides the features into several windows of a pre-defined size, generating grid coordinates for each first window. The window grid update module (e.g., executing steps SA1-SA2) uses the first optical flow information to update the grid coordinate points, obtaining a new second window distribution. The window temporal attention module (e.g., executing step S140) performs temporal information interaction, obtaining the temporal information interaction result of the image features (update information from at least two frames of images, i.e., new image features). Optionally, the image features input to this process can be... Figure 3 The features output by the spatial interaction layer in the process can be new image features output by this process. Figure 3 Features processed by the motion-guided adaptive window temporal interaction layer in the model.

[0066] In this embodiment of the disclosure, for Figure 5 The motion-guided window coordinate update module provides an optional implementation (e.g., steps S1321-S1323 related to step SA1 can be executed) to divide the feature into multiple non-overlapping default windows, such as... Figure 2 or Figure 4 As shown on the left. To determine the size and position of each target window, a default window is used as a reference, and bidirectional optical flow is used to predict scale changes and offsets. Specifically, as... Figure 6 As shown, the processing can include: the forward optical flow (second optical flow information) undergoes convolution processing (conv, which can be done once or multiple times). Figure 6 (Taking two iterations as an example) The data is then sent to the Query layer to generate... The features and the reverse optical flow (third optical flow information) are respectively processed by convolution (conv, which can be done once or multiple times). Figure 6 (Taking two iterations as an example) The data is then sent to the Key layer and Value layer for generation. Features and Features. Among them, forward optical flow enables the model to focus on potential future motion, while backward optical flow helps the model verify past motion information, thereby enhancing temporal consistency. The V feature can be represented as:

[0067] in, Indicates forward optical flow. This indicates reverse optical flow.

[0068] Furthermore, The information is fed into the cross-attention module for interaction, obtaining the correction information in the reverse optical flow. This correction is added to the original forward optical flow branch to obtain the overall optical flow information. The features after information interaction are then fed into the scale branch (e.g., through convolution) and the offset branch (e.g., through convolution) to obtain the predicted scale and offset. Optionally, the scale and offset are obtained through cross-attention. Further, in step SA2, the corresponding second window can be adjusted based on the above formula 2. Optionally, each second window has a different size, and the number of sampling points in each second window can be the same (e.g., by increasing the sampling interval or interpolation). After obtaining the new second window distribution, temporal attention processing based on the second window is performed to obtain the temporal interaction result.

[0069] In this embodiment of the disclosure, the aforementioned at least two frames may include at least two first frames and at least one second frame, wherein the clarity of the first image is greater than that of the second image.

[0070] The first image can also refer to an image in the video with relatively low blur (or degree of blur). The blur of a video or image can be determined based on motion-induced afterimages or trails. For example, blur can be associated with the number of pixels covered by the trail, but is not limited to this. For ease of understanding, the first image may be referred to as a sharp frame in the following text.

[0071] Similarly, the second image can also refer to an image with greater blur in the video. For ease of understanding, the second image may be referred to as a blur frame in the following text.

[0072] In one example, the first image is an image with a sharpness greater than or equal to a first threshold or a blurriness less than or equal to a second threshold, and the second image is an image with a sharpness less than the first threshold or a blurriness greater than the second threshold; wherein, the criteria for judging sharp frames and blurri frames (such as the first threshold or the second threshold) can be set by those skilled in the art according to the actual situation, and the embodiments of this disclosure are limited herein.

[0073] Optionally, the first image can be detected in the captured video, for example, by using a second network (e.g., a detection network) to detect sharp frames in the video. Alternatively, the first image can also be obtained by actively capturing sharp frames with short exposures, for example, by acquiring images based on shorter exposure parameters (e.g., exposure time) as the first image.

[0074] In this embodiment of the disclosure, the positions of the first image and the second image can be set by those skilled in the art according to actual needs. As an example, the captured video contains multiple video clips with related motion trajectories. Each video clip includes two clear frames (the first image) and at least one blurred frame (the second image), and the at least one blurred frame is located between the two clear frames, but is not limited thereto.

[0075] In this embodiment of the disclosure, a feasible implementation method is provided for determining the first optical flow information, such as... Figure 7 As shown, optical flow can be decomposed, specifically, it can include: Step S210: Decompose the fourth optical flow information between at least two frames of the first image to obtain the inter-frame optical flow information between each second image and its adjacent images; In this embodiment of the disclosure, the fourth optical flow information refers to the optical flow information between clear frames. For example, taking two clear frames as an example, the fourth optical flow information can be predicted in the following manner: Formula 3 in, This represents the optical flow estimation method. and Two clear frames representing the interval in a video sequence. Indicates the fourth optical flow information. .

[0076] It is understandable that the adjacent images of the second image may be the first image or other second images. Furthermore, the adjacent images of the second image may include two adjacent images on either side, or they may only include one adjacent image; that is, the inter-frame optical flow information determined for each second image may include one or two. If there are at least two second images, the inter-frame optical flow information between the second image and its adjacent images may be duplicated. Identical values ​​can be preprocessed, for example, only one value needs to be taken, or other algorithms can be used to calculate these identical values.

[0077] Step S220: Determine the inter-frame optical flow information corresponding to at least one second image frame as the first optical flow information between at least two images.

[0078] As an example, such as Figure 8As shown, assuming a video sequence (or video segment) includes two clear frames (clear frame 1 and clear frame 2, corresponding to the first image) and two blurred frames between the two clear frames (blurred frame 1 and blurred frame 2, corresponding to the second image), the optical flow between the two clear frames in the video sequence is predicted first. Under the premise of fully considering motion changes, the optical flow is decomposed according to the motion degree of the intermediate blurred frame to obtain optical flow 1 between blurred frame 1 and the adjacent clear frame 1, optical flow 2 between blurred frame 1 and the adjacent blurred frame 2, and optical flow 3 between blurred frame 2 and the adjacent clear frame 2. That is, the inter-frame optical flow information optical flow 1, optical flow 2 and optical flow 3 with accurate direction and small error are obtained.

[0079] In this embodiment of the disclosure, a feasible implementation of step S210 is provided, which specifically may include: Step S211: Determine the differences in image features between each second image and its neighboring images; Similarly, the neighboring images of the second image may be the first image or other second images. Furthermore, the neighboring images of the second image may include two images adjacent to each other on both sides, or they may include only one neighboring image; that is, the image feature differences determined for each second image may include one or two. If there are at least two second images, the image feature differences between the second image and its neighboring images may overlap. Identical values ​​can be preprocessed, for example, only one value needs to be taken, or other algorithms can be used to calculate these identical values.

[0080] Alternatively, the image feature differences (diff) between adjacent frames of a video sequence can be calculated in the following manner: Formula 4 in, , and This represents two adjacent images.

[0081] As an example, such as Figure 9 As shown, assuming a video sequence (or video segment) includes two clear frames (clear frame 1 and clear frame 2, corresponding to the first image) and two blurry frames between the two clear frames (blurry frame 1 and blurry frame 2, corresponding to the second image), the diff1 between blurry frame 1 and the adjacent clear frame 1, the diff2 between blurry frame 1 and the adjacent blurry frame 2, and the diff3 between blurry frame 2 and the adjacent clear frame 2 can be calculated.

[0082] Step S212: Based on the image feature differences and fourth optical flow information corresponding to at least one frame of the second image, the first network is used for decomposition processing to obtain the inter-frame optical flow information between each second image and its neighboring images.

[0083] Optionally, the differences in image features corresponding to at least one frame of the second image (e.g., multiple differences calculated using Formula 4) will be used. With the fourth optical flow information (e.g., obtained by formula three) The second image is concatenated, and the concatenated features are decomposed using the first network to obtain the inter-frame optical flow information between each second image and its neighboring images.

[0084] Alternatively, the stitching based on the image feature differences corresponding to at least one frame of the second image and the fourth optical flow information can also be performed using the following stitching method:

[0085] in, , This indicates the number of frames when the first and second images are combined, and 5 indicates the number of channels (including optical flow information channels and image channels). Indicates from the first Frame to the Inter-frame optical flow information of +1 frame.

[0086] In other words, the image feature differences corresponding to at least one frame of the second image are first concatenated with the fourth optical flow information to obtain the concatenated result corresponding to at least one frame of the second image. Then, the concatenated result is concatenated again, and the first network is used for decomposition processing to obtain the inter-frame optical flow information between each second image and its adjacent images. That is, the output of the first network can be expressed as:

[0087] in, , This indicates the number of frames when the first and second images are combined, and 2 indicates the number of channels (including optical flow information channels). Indicates from the first Frame to the Inter-frame optical flow information of +1 frame.

[0088] like Figure 10a As shown, the differences in image features between adjacent images ( Optical flow information between clear frames and () The optical flow is concatenated and input into the first network, which then decomposes the optical flow into multiple inter-frame optical flow information. ).

[0089] As an example, in Figure 9 On the basis of, such as Figure 10b As shown, splicing and The input to the model (first network) is obtained, and the output of the model is the three optical flows obtained from the decomposition. , , , representing the optical flow of clear frame 1 and blurry frame 1, blurry frame 1 and blurry frame 2, and blurry frame 2 and clear frame 2, respectively.

[0090] In this embodiment of the disclosure, after determining the inter-frame optical flow information corresponding to at least one frame of the second image as the first optical flow information between at least two frames in step S220, the determination of the second and third optical flow information between at least two frames based on the first optical flow information in step S131 may include: Step S1311: Based on the transferability of optical flow, the second optical flow information can be determined based on multiple inter-frame optical flow information, for example:

[0091] in, , Indicates the first Frame to the Optical flow of frames, This represents the superposition of optical flows. Optionally, multiple optical flows can be unified. The coordinates are obtained and sampled in the same coordinate system.

[0092] Step S1312: By swapping the positions of the images, the third optical flow information can be determined based on multiple inter-frame optical flow information, for example:

[0093] in, Indicates a clear frame To clear frame The optical flow information between them. Furthermore, it can be obtained through optical flow decomposition and... ,get ,in, .

[0094] In this embodiment of the disclosure, the training samples used in the training process of the first network may include first video samples and second video samples. The first video samples include clear frames and blurry frames, and the second video samples are videos corresponding to the first video samples, all of which are clear frames. Based on each pair of adjacent image frames in the second video samples, the desired decomposed inter-frame optical flow information can be obtained. By using the first network to decompose the first video samples, inter-frame optical flow information samples can be obtained.

[0095] Optionally, the loss function used for training the first network may include the L1 loss function. L1 represents the difference between the inter-frame optical flow information samples and the target inter-frame optical flow information.

[0096] Optionally, to ensure the continuity of optical flow decomposition, the loss function used in training the first network may also include a continuity loss. :

[0097] in, This represents the total optical flow information of the first video sample (taking the first video sample as an example, which includes two clear frames and a blurred frame between the clear frames, the total optical flow is the optical flow between the two clear frames). Represents the first video sample after decomposition. Frame to the Inter-frame optical flow information samples between frames This represents the transmission of optical flow, used to connect multiple inter-frame optical flow information samples into complete optical flow information for comparison with the total optical flow information.

[0098] Optionally, to ensure that the decomposition of optical flow corresponds to the speed of motion, the loss function used in training the first network may also include non-uniform decomposition loss. :

[0099] in, This represents the total optical flow information of the first video sample. Represents the first video sample after decomposition. Frame to the Inter-frame optical flow information samples between frames Represents the first video sample after decomposition. Frame to the The optical flow decomposition index (also known as the non-uniformity index) between frames is used to represent the relative degree of motion and to constrain the degree of optical flow decomposition, making the optical flow decomposition more consistent with the non-uniformity index so that the optical flow decomposition conforms to the proportional relationship between the speed of motion.

[0100] Alternatively, the optical flow decomposition index can be calculated based on the average frame rate. Optical flow decomposition is constrained by averaging the number of blurred frames. When creating the first training sample, multiple image frames can be averaged at random frame numbers (e.g., 7, 9, 11, 13, 15, etc.) to create the effect of blurred frames, simulating non-uniform motion. The number of frames used for averaging is called the average frame number. Therefore, for the first training sample, the average frame number of sharp frames is 1, and the average frame number of blurred frames could be 7, 9, 11, 13, 15, etc. When acquiring the first training sample, the average exponent corresponding to each frame of the first training sample can be simultaneously acquired to guide the optical flow decomposition during the training phase.

[0101] Optionally, optical flow decomposition index It can be represented as:

[0102] in, To obtain the averaging method Average number of frames used per frame, the first Frame optical flow decomposition index You can refer to the average frame rate of the current frame and the average frame rate of the previous frame. For normalization operations, such as normalization based on the total average number of frames of the first training sample, the optical flow decomposition index can be normalized to (0, 1).

[0103] Furthermore, the first network can be trained using any of the above loss functions or a combination of at least two loss functions. Taking the training of the first network using the above three loss functions as an example, the total loss used by the first network can be expressed as:

[0104] in, , , These represent the coefficients corresponding to each loss function.

[0105] The optical flow prediction method provided in this disclosure addresses the inaccuracy of existing optical flow prediction methods for videos with large motion blur. The inventors of this disclosure have discovered that the failure of existing optical flow prediction methods for large motion is due to the difficulty of feature matching. When an object in an image undergoes large displacement, the local features extracted by traditional CNNs (Convolutional Neural Networks) may not have a corresponding relationship in another frame, leading to matching failure. This feature matching error severely affects the prediction of optical flow, resulting in inaccurate predicted optical flow or incorrect optical flow direction. Furthermore, the Transformer method is also ineffective in predicting optical flow during large motion.

[0106] The optical flow prediction method provided in this embodiment no longer uses feature matching. Instead, it proposes an optical flow decomposition method, which decomposes the optical flow of two clear frames into the intermediate blurred frame. This decomposition, while fully considering the degree of motion, yields inter-frame optical flow with accurate direction and small error. This avoids the problem of inaccurate feature matching when image features change drastically with significant motion. The optical flow predicted by the optical flow decomposition method is more accurate and can play a positive prior role in the deblurring process.

[0107] In this embodiment of the disclosure, for cases where at least two images include at least two first images and at least one second image, and the sharpness of the first image is greater than that of the second image (details of which are described above and will not be repeated here), a feasible implementation process is also provided. This process can borrow prior information from the sharp image to lay a high-quality feature prior foundation for subsequent deblurring processing. Specifically, this process may include: Step S310: Based on the first optical flow information and at least two frames of the first image, obtain at least one frame of the third image, the content of the third image corresponding to the content of the second image; Optionally, the first optical flow information can be obtained by decomposing multiple inter-frame optical flow information using the optical flow decomposition method described above.

[0108] In this embodiment of the disclosure, the prior information of the clear frame (first image) can be referenced by the first optical flow information to bring the clear content into the blurred frame according to the pre-calculated optical flow. The generated third image is used for video deblurring, which is equivalent to implicitly using the optical flow to reference the features of the clear frame.

[0109] Step S320: Fuse at least one frame of the third image and at least one frame of the second image to obtain at least one frame of the fourth image; In this embodiment of the disclosure, the fusion of the referenced prior information and the blurred frame (second image) can generate a new blurred frame.

[0110] Step S330: Based on at least two first images and at least one fourth image, determine the image features of at least two images in step S110.

[0111] In this embodiment, the new blurred frame has a high-quality feature prior basis and replaces the original blurred frame for subsequent deblurring processing. This can add conditional constraints to the deblurring process, accurately control the image restoration process, and improve restoration quality and controllability.

[0112] In this embodiment of the disclosure, the execution process of steps S310 to S330 can also be based on image features of image frames. For example, the process can also be described as follows: Step S310: Based on the first optical flow information and the image features of at least two frames of the first image, obtain the image features of at least one frame of the third image, wherein the content of the third image corresponds to the content of the second image; Step S320: Fuse the image features of at least one frame of the third image and the image features of at least one frame of the second image to obtain the image features of at least one frame of the fourth image; Step S330: Based on the image features of at least two first images and at least one fourth image, determine the image features of at least two images in step S110.

[0113] In this embodiment of the disclosure, a feasible implementation of step S310 is provided, which specifically may include: Step S311: Based on the first optical flow information, determine the fifth optical flow information and the sixth optical flow information between at least two frames of the first image, wherein the fifth optical flow information is the optical flow information from the first image in time sequence to the second image in time sequence in at least two frames of the first image, and the third optical flow information is the optical flow information from the first image in time sequence to the first image in time sequence in at least two frames of the first image. The fifth optical flow information can also be called forward optical flow information, and the sixth optical flow information can also be called reverse optical flow. Optionally, in certain scenarios, such as when at least one second image is located between two first images, the fifth optical flow information can be the same as the second optical flow information mentioned above, and the sixth optical flow information can be the same as the third optical flow information mentioned above.

[0114] Step S312: Based on the fifth optical flow information and the first image with the earlier time sequence, obtain at least one fifth image; In this embodiment of the disclosure, prior information can be borrowed from the clear frame (first image) that is earlier in the time sequence using forward optical flow information (fifth optical flow information) so that the content of the clear frame that is earlier in the time sequence flows into the blurred frame according to the pre-calculated forward optical flow. The content of the generated fifth image is also corresponding to the content of the second image.

[0115] Step S313: Based on the sixth optical flow information and the first image that is in later time sequence, obtain at least one sixth image; In this embodiment of the disclosure, prior information can be borrowed from the later clear frame (first image) using reverse optical flow information (sixth optical flow information) so that the content of the later clear frame can be fed into the blurred frame according to the pre-calculated reverse optical flow. The content of the generated sixth image is also corresponding to the content of the second image.

[0116] Step S314: Based on at least one fifth image and at least one sixth image, obtain at least one third image.

[0117] Optionally, at least one fifth image and at least one sixth image are fused to obtain at least one third image.

[0118] In this embodiment of the disclosure, the method of borrowing prior information from clear frames can be applied to ControlNet, where ControlNet extracts features from both clear and blurry frames and feeds them into subsequent networks for video deblurring. For example... Figure 11 As shown, assuming a video sequence includes two sharp frames and two blurry frames between them, when extracting image features of blurry frames in ControNet, the prior reference module for sharp frames ( Figure 11 The shadowed module in ControlNet uses the forward and reverse optical flow to reference the feature priors of two clear frames before and after, and updates the blurry frame output by the previous module (e.g., the feature processing layer), i.e. the features of the clear frame and the blurry frame. After the clear frame prior reference module generates new blurry frame features, the new blurry frame features replace the original blurry frame features, and are sent to the subsequent network along with the clear frame for video deblurring.

[0119] In other words, the input to the clear frame prior reference module is the output of the previous module in ControlNet (e.g., Figure 11 The module without shadows in ControlNet outputs two sharp frame features and two blurry frame features, and the output is the updated image features (only the blurry frame features are updated, outputting two sharp frame features and two updated blurry frame features). In this embodiment of the disclosure, the process of updating the blurred frame in the clear frame prior reference module can be as follows: Figure 12 As shown, this may include, in order to direct the clear content into the blurred frame according to a pre-calculated optical flow, the clear frame of the first frame... The first image (with the first image in the sequence) is added to the forward optical flow (the fifth optical flow information), resulting in a clear frame at the end. The second image (time-following image) is added to the reverse optical flow (sixth optical flow information), and then fused with the clear content and motion information optical flow through their respective convolution operations (conv) to obtain clear content with the same motion state as the intermediate blurred frame. After obtaining the clear content, it is concatenated with the original blurred frame features and fed into a convolution operation for further feature fusion to obtain the updated blurred frame features.

[0120] The inventors of this disclosure have discovered that even when optical flow prediction is accurate, existing methods for deblurring using sharp frames are not very effective. For example, simply moving pixels in a sharp frame to their corresponding positions based solely on optical flow information can lead to distortion, unnaturalness, and holes. This is because existing methods simply move pixel positions without considering overall consistency, resulting in distortion and other issues. Directly moving pixel positions via optical flow relies solely on the displacement information of local pixels. When the motion is complex (such as rotation, occlusion, or non-rigid deformation), the independent movement of local pixels can disrupt overall consistency, leading to tearing, blurring, or misalignment.

[0121] The method for referencing prior information from clear frames provided in this disclosure improves the problem of poor optical flow prior reference in complex situations by implicitly referencing optical flow priors, thereby improving the overall coordination of the perception of the global structure of the image (such as object contours and texture continuity).

[0122] Furthermore, by fully considering video temporal information and combining it with the more accurate optical flow predicted by the optical flow prediction method provided in the embodiments of this disclosure above, motion blur in the video can be effectively removed, especially in scenes with large motion.

[0123] Based on at least one of the above embodiments, this disclosure provides a complete video demodulation processing flow, which may include, for example, optical flow decomposition, clear frame prior reference, motion-guided adaptive window size temporal attention, etc. Figure 13 As shown, specifically, it can include: a video sequence with the first and last two frames being clear frames and the middle frames being blurry. First, optical flow (fourth optical flow information) is estimated from the two clear frames, and then it passes through an optical flow decomposition module (the specific processing procedure can be found in the above description, for example, for...). Figures 7-10b (The introduction will not be repeated here.) Based on the motion degree of the blurred frame, the optical flow result (inter-frame optical flow information) of the intermediate blurred frame is obtained. The optical flow is encoded (if the optical flow generates forward and reverse optical flow information, it needs to be encoded separately) and then fed into ControlNet and UNet (e.g., including but not limited to the diffusion model). This helps ControlNet to better extract video features and helps UNet to better utilize temporal information. The input to ControlNet also includes image features of the video sequence encoded by the Encoder. The role of ControlNet is to add conditional constraints to the diffusion model without compromising its generative capabilities, precisely controlling the image restoration process and improving restoration quality and controllability (the specific processing steps can be found in the above description, for example, for...). Figures 11-12 The introduction of [the specific steps] will not be repeated here. Features from the sharp frames and updated blurred frames extracted from ControlNet, along with noise, are fed into UNet. The quality of the features fed into UNet directly affects the model's performance. UNet performs one or more (e.g., T-1 times) deblurring processes (the specific processing steps can be found above, for example, [the specific steps are described in the text]). Figures 2-6 (The introduction of the output image features will not be repeated here.) The output image features are decoded by the Decoder to obtain a clean video sequence.

[0124] The video deblurring method provided in this disclosure has good performance for large motion images. Not only is the optical flow prediction for motion-blurred images more accurate, but it also incorporates optical flow priors, allowing these priors to play a positive role. Furthermore, it employs an implicit method of utilizing optical flow, generating overall harmonious feature priors even when encountering rotation, occlusion, or non-rigid deformation, thus improving the problems of inaccurate optical flow prediction for large motion images and poor optical flow prior referencing in complex situations. This effectively removes motion blur from video sequences.

[0125] This disclosure provides another method performed by an electronic device, where clear frame prior referencing can be combined with any video deblurring method, such as... Figure 14 As shown, the method includes: Step S1410: Obtain at least two first frames and at least one second frame of the video, as well as fourth optical flow information between at least two first frames, wherein the sharpness of the first image is greater than that of the second image; In this embodiment of the disclosure, the description of the video source, content, acquisition method, large motion scene, etc., as well as the description of the first image and the second image, can be found above and will not be repeated here.

[0126] In this embodiment, the fourth optical flow information refers to the optical flow information between clear frames. The specific calculation method can be found in the above description and will not be repeated here.

[0127] Step S1420: Based on the fourth optical flow information and at least two frames of the first image, obtain at least one frame of the third image, the content of the third image corresponding to the content of the second image; In this embodiment of the disclosure, the prior information of the clear frame (first image) can be referenced by the fourth optical flow information so that the clear content is fed into the blurred frame according to the pre-calculated optical flow. The generated third image is used for video deblurring, which is equivalent to implicitly using the optical flow to reference the features of the clear frame.

[0128] Step S1430: Fuse at least one frame of the third image and at least one frame of the second image to obtain at least one frame of the fourth image; In this embodiment of the disclosure, the fusion of the referenced prior information and the blurred frame (second image) can generate a new blurred frame.

[0129] Step S1440: Deblur at least two frames of the first image and at least one frame of the fourth image.

[0130] In this embodiment, the new blurred frame has a high-quality feature prior basis and replaces the original blurred frame for subsequent deblurring processing. This can add conditional constraints to the deblurring process, accurately control the image restoration process, and improve restoration quality and controllability.

[0131] In this embodiment of the disclosure, the execution process of steps S1410 to S1440 can also be based on image features of the image frame. For example, the process can also be described as follows: Step S1410: Obtain image features of at least two first frames and at least one second frame of the video, as well as fourth optical flow information between at least two first frames, wherein the sharpness of the first image is greater than that of the second image; Step S1420: Based on the fourth optical flow information and the image features of at least two frames of the first image, obtain the image features of at least one frame of the third image, the content of the third image corresponding to the content of the second image; Step S1430: Fuse the image features of at least one frame of the third image and the image features of at least one frame of the second image to obtain the image features of at least one frame of the fourth image; Step S1440: Deblur the image features of at least two frames of the first image and at least one frame of the fourth image.

[0132] In this embodiment of the disclosure, a feasible implementation of step S1420 is provided, which specifically may include: Step S1421: Based on the fourth optical flow information, determine the fifth optical flow information and the sixth optical flow information between at least two frames of the first image, wherein the fifth optical flow information is the optical flow information from the first image with a earlier time sequence to the second image with a later time sequence in at least two frames of the first image, and the third optical flow information is the optical flow information from the first image with a later time sequence to the first image with a earlier time sequence in at least two frames of the first image. Step S1422: Based on the fifth optical flow information and the first image with the earlier time sequence, obtain at least one fifth image; Step S1423: Based on the sixth optical flow information and the first image that is in later time sequence, obtain at least one sixth image; Step S1424: Based on at least one fifth image and at least one sixth image, obtain at least one third image.

[0133] For details regarding steps S1421 to S1424 that are not fully explained, please refer to the explanations of steps S311 to S314 and... Figures 11-12 The details of its introduction will not be repeated here.

[0134] In this embodiment of the disclosure, the accurate optical flow information can also be determined by combining the optical flow decomposition method. Specifically, step S1421 may include: Step SB1: Decompose the fourth optical flow information to obtain the inter-frame optical flow information between each second image and its adjacent images; Step SB2: Determine the fifth optical flow information and the sixth optical flow information based on the inter-frame optical flow information corresponding to at least one frame of the second image.

[0135] Furthermore, step SB1 may specifically include: Step SB11: Determine the differences in image features between each second image and its neighboring images; Step SB12: Based on the image feature differences and fourth optical flow information corresponding to at least one frame of the second image, the first network is used for decomposition processing to obtain the inter-frame optical flow information between each second image and its neighboring images.

[0136] For details regarding steps SB1-SB2 and SB11-SB12 that are not fully explained, please refer to the sections on steps S210-S220, S221-S222, and... Figures 7-10b The details of its introduction will not be repeated here.

[0137] The video deblurring method provided in this disclosure has good performance for large motion images. Not only is the optical flow prediction for motion-blurred images more accurate, but it also incorporates optical flow priors, allowing these priors to play a positive role. Furthermore, it employs an implicit method of utilizing optical flow, generating overall harmonious feature priors even when encountering rotation, occlusion, or non-rigid deformation, thus improving the problems of inaccurate optical flow prediction for large motion images and poor optical flow prior referencing in complex situations. This effectively removes motion blur from video sequences.

[0138] This disclosure also provides a method executed by an electronic device that can accurately determine optical flow information. This method can be combined with any video processing procedure that uses optical flow information, including but not limited to video deblurring. Figure 15 As shown, the method includes: Step S1510: Obtain at least two first frames and at least one second frame of the video, as well as fourth optical flow information between at least two first frames, wherein the sharpness of the first image is greater than that of the second image; In this embodiment of the disclosure, the description of the video source, content, acquisition method, large motion scene, etc., as well as the description of the first image and the second image, can be found above and will not be repeated here.

[0139] In this embodiment, the fourth optical flow information refers to the optical flow information between clear frames. The specific calculation method can be found in the above description and will not be repeated here.

[0140] Step S1520: Decompose the fourth optical flow information to obtain the inter-frame optical flow information between each second image and its adjacent images; It is understandable that the adjacent images of the second image may be the first image or other second images. Furthermore, the adjacent images of the second image may include two adjacent images on either side, or they may only include one adjacent image; that is, the inter-frame optical flow information determined for each second image may include one or two. If there are at least two second images, the inter-frame optical flow information between the second image and its adjacent images may be duplicated. Identical values ​​can be preprocessed, for example, only one value needs to be taken, or other algorithms can be used to calculate these identical values.

[0141] Step S1530: Process the video based on the inter-frame optical flow information corresponding to at least one frame of the second image.

[0142] In this embodiment of the disclosure, step S1520 may specifically include: Step S1521: Determine the differences in image features between each second image and its neighboring images; Step S1522: Based on the image feature differences corresponding to at least one frame of the second image and the fourth optical flow information, the first network is used for decomposition processing to obtain the inter-frame optical flow information between each second image and its neighboring images.

[0143] For details regarding steps S1521 to S1522 that are not fully explained, please refer to the explanations of steps S221 to S222 and... Figures 7-10b The details of its introduction will not be repeated here.

[0144] In this embodiment, the optical flow of two clear frames is decomposed into blurred frames. This decomposition, taking motion into full account, yields inter-frame optical flow with accurate direction and minimal error. The optical flow prediction method based on this decomposition is more accurate and can play a positive role in video processing.

[0145] The technical solutions provided in this disclosure can be applied to various electronic devices capable of capturing images (including data acquisition devices), including but not limited to mobile terminals and smart terminals, such as smartphones, tablets, laptops, smart wearable devices (e.g., watches, glasses), smart speakers, in-vehicle terminals, personal digital assistants, portable multimedia players, navigation devices, etc., but are not limited thereto. Those skilled in the art will understand that, in addition to components specifically designed for mobile purposes, the constructions according to this disclosure can also be applied to fixed-type terminals, such as digital televisions and desktop computers.

[0146] The technical solutions provided in this disclosure can also be applied to video deblurring in servers, such as independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0147] Optionally, the video deblurring scheme provided in this disclosure can also be completed collaboratively by multiple computing-capable electronic devices. For example, different electronic devices may each complete a portion of the steps of the methods provided in this disclosure. As an example, optical flow decomposition may be performed by one electronic device, and clear frame prior reference may be performed by another electronic device, etc., but this is not the only possibility.

[0148] Through extensive experimentation, the inventors of this disclosure have discovered that the optical flow decomposition module provided in the embodiments of this disclosure achieves a result closer to the target annotation effect than existing technologies, and the clean video sequence texture output by the clear frame prior reference module provided in the embodiments of this disclosure is more harmonious and natural.

[0149] This disclosure also provides an electronic device including a processor, and optionally, a transceiver and / or memory coupled to the processor, the processor being configured to perform the steps of the method provided in any optional embodiment of this disclosure.

[0150] Figure 16 The diagram shows a structural schematic of an electronic device to which an embodiment of the present invention applies, such as... Figure 16 As shown, Figure 16 The illustrated electronic device 4000 includes a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of this disclosure. Optionally, the electronic device may be a first network node, a second network node, or a third network node.

[0151] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with this disclosure. Processor 4001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0152] Bus 4002 may include a pathway for transmitting information between the aforementioned components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 4002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 16 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0153] The memory 4003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium capable of carrying or storing computer programs and capable of being read by a computer, without limitation herein.

[0154] The memory 4003 is used to store computer programs that execute embodiments of the present disclosure, and is controlled by the processor 4001 to execute them. The processor 4001 is used to execute the computer programs stored in the memory 4003 to implement the steps shown in the foregoing method embodiments.

[0155] This disclosure provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the steps and corresponding content of the aforementioned method embodiments.

[0156] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, can implement the steps and corresponding content of the aforementioned method embodiments.

[0157] The terms “first,” “second,” “third,” “fourth,” “1,” “2,” etc. (if present) in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in a sequence other than that shown in the figures or text.

[0158] It should be understood that although arrows indicate various operation steps in the flowcharts of the embodiments of this disclosure, the order in which these steps are implemented is not limited to the order indicated by the arrows. Unless explicitly stated herein, in some implementation scenarios of the embodiments of this disclosure, the implementation steps in each flowchart can be executed in other orders as required. Furthermore, some or all of the steps in each flowchart may include multiple sub-steps or multiple stages based on the actual implementation scenario. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage can also be executed at different times. In scenarios where execution times differ, the execution order of these sub-steps or stages can be flexibly configured as required, and the embodiments of this disclosure do not limit this.

[0159] The above text and accompanying drawings are provided as examples only to help the reader understand this disclosure. They are not intended and should not be construed as limiting the scope of this disclosure in any way. Although certain embodiments and examples have been provided, it will be apparent to those skilled in the art, based on the content disclosed herein, that changes can be made to the illustrated embodiments and examples, and other similar implementations based on the technical concept of this disclosure can be adopted without departing from the scope of this disclosure, and these modifications and modifications are also within the protection scope of the embodiments of this disclosure.

Claims

1. A method performed by an electronic device, characterized in that, include: Acquire image features of at least two frames of the video, and first optical flow information between the at least two frames; Determine at least two first windows for segmenting the image features; Based on the first optical flow information, the at least two first windows are adjusted to obtain at least two second windows; For each of the at least two second windows, the at least two frames of images are deblurred based on the image features of the at least two frames of images within the second window.

2. The method according to claim 1, characterized in that, Based on the first optical flow information, the at least two first windows are adjusted, including: Based on the first optical flow information, second optical flow information and third optical flow information between the at least two frames are determined, wherein the second optical flow information is the optical flow information from the earlier image to the later image in the at least two frames, and the third optical flow information is the optical flow information from the later image to the earlier image in the at least two frames. Based on the second optical flow information and the third optical flow information, the at least two first windows are adjusted.

3. The method according to claim 2, characterized in that, The adjustment of the at least two first windows based on the second optical flow information and the third optical flow information includes: Based on the second optical flow information and the third optical flow information, a cross-attention network is used to determine the correction information corresponding to the second optical flow information; Based on the correction information, the second optical flow information is corrected; The at least two first windows are adjusted based on the corrected second optical flow information.

4. The method according to any one of claims 1-3, characterized in that, The step of adjusting the at least two first windows based on the first optical flow information to obtain at least two second windows includes: Based on the first optical flow information, determine the scale change and offset corresponding to the at least two first windows respectively; For each of the at least two first windows, the first window is adjusted based on the corresponding scale change and offset to obtain the corresponding second window.

5. The method according to claim 2, characterized in that, The at least two frames of images include at least two first frames and at least one second frame, wherein the sharpness of the first image is greater than that of the second image, and the method further includes: Decompose the fourth optical flow information between at least two frames of the first image to obtain the inter-frame optical flow information between each second image and its adjacent images; The inter-frame optical flow information corresponding to at least one second image frame is determined as the first optical flow information between the at least two images.

6. The method according to claim 5, characterized in that, The step of decomposing the fourth optical flow information between at least two frames of the first image to obtain the inter-frame optical flow information between each second image and its adjacent images includes: Determine the differences in image features between each second image and its neighboring images; Based on the image feature differences corresponding to at least one frame of the second image and the fourth optical flow information, the first network is used for decomposition processing to obtain the inter-frame optical flow information between each second image and its adjacent images.

7. The method according to any one of claims 1-6, characterized in that, The at least two frames of images include at least two first frames and at least one second frame, wherein the sharpness of the first image is greater than that of the second image, and the method further includes: Based on the first optical flow information and the at least two first images, at least one third image is obtained, and the content of the third image corresponds to the content of the second image. The at least one third image and the at least one second image are fused together to obtain at least one fourth image; Based on the at least two first images and the at least one fourth image, determine the image features of the at least two images.

8. The method according to claim 7, characterized in that, The step of obtaining at least one third image based on the first optical flow information and the at least two first images includes: Based on the first optical flow information, a fifth optical flow information and a sixth optical flow information are determined between the at least two frames of the first image. The fifth optical flow information is the optical flow information from the first image in time sequence to the second image in time sequence in the at least two frames of the first image. The third optical flow information is the optical flow information from the first image in time sequence to the first image in time sequence in the at least two frames of the first image. Based on the fifth optical flow information and the first image with the earlier time sequence, at least one fifth image is obtained; Based on the sixth optical flow information and the first image that follows in time, at least one sixth image is obtained; Based on the at least one fifth image and the at least one sixth image, at least one third image is obtained.

9. A method performed by an electronic device, characterized in that, include: Acquire at least two first frames and at least one second frame of a video, as well as fourth optical flow information between the at least two first frames, wherein the sharpness of the first image is greater than that of the second image; Based on the fourth optical flow information and the at least two first images, at least one third image is obtained, and the content of the third image corresponds to the content of the second image. The at least one third image and the at least one second image are fused together to obtain at least one fourth image; The at least two first images and the at least one fourth image are deblurred.

10. The method according to claim 9, characterized in that, The process of obtaining at least one third image based on the fourth optical flow information and the at least two first images includes: Based on the fourth optical flow information, the fifth optical flow information and the sixth optical flow information between the at least two frames of the first image are determined, wherein the fifth optical flow information is the optical flow information from the first image in time sequence to the second image in time sequence in the at least two frames of the first image, and the third optical flow information is the optical flow information from the first image in time sequence to the first image in time sequence in the at least two frames of the first image; Based on the fifth optical flow information and the first image with the earlier time sequence, at least one fifth image is obtained; Based on the sixth optical flow information and the first image that follows in time, at least one sixth image is obtained; Based on the at least one fifth image and the at least one sixth image, at least one third image is obtained.

11. The method according to claim 9 or 10, characterized in that, The step of determining the fifth and sixth optical flow information between the at least two frames of the first image based on the fourth optical flow information includes: The fourth optical flow information is decomposed to obtain the inter-frame optical flow information between each second image and its adjacent images; The fifth optical flow information and the sixth optical flow information are determined based on the inter-frame optical flow information corresponding to at least one frame of the second image.

12. The method according to claim 11, characterized in that, The step of decomposing the fourth optical flow information to obtain inter-frame optical flow information between each second image and its adjacent images includes: Determine the differences in image features between each second image and its neighboring images; Based on the image feature differences corresponding to at least one frame of the second image and the fourth optical flow information, the first network is used for decomposition processing to obtain the inter-frame optical flow information between each second image and its adjacent images.

13. A method performed by an electronic device, characterized in that, include: Acquire at least two first frames and at least one second frame of a video, as well as fourth optical flow information between the at least two first frames, wherein the sharpness of the first image is greater than that of the second image; The fourth optical flow information is decomposed to obtain the inter-frame optical flow information between each second image and its adjacent images; The video is processed based on the inter-frame optical flow information corresponding to at least one frame of the second image.

14. The method according to claim 13, characterized in that, The step of decomposing the fourth optical flow information to obtain inter-frame optical flow information between each second image and its adjacent images includes: Determine the differences in image features between each second image and its neighboring images; Based on the image feature differences corresponding to at least one frame of the second image and the fourth optical flow information, the first network is used for decomposition processing to obtain the inter-frame optical flow information between each second image and its adjacent images.

15. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the method according to any one of claims 1-14.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-14.

17. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-14.

Images