Methods performed by electronic devices and electronic devices

Abstract

Embodiments of the present disclosure provide a method executed by an electronic device and an electronic device, relating to the field of artificial intelligence technology. The method comprises: obtaining a first image feature related to a global of an image; obtaining, by a first artificial intelligence (AI) network, a second image feature related to a motion subject in the image, and performing optical flow prediction on different image scales based on the first image feature and the second image feature to obtain optical flow information of different image scales respectively; the optical flow information of different image scales has a correlation relationship; and generating a video based on the optical flow information. Optionally, the above method executed by the electronic device can be executed using an artificial intelligence model.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and more specifically, to a method and an electronic device performed by an electronic device. Background Technology

[0002] In image and video generation technology, a scheme is proposed that predicts an optical flow sequence from the input image, uses the predicted optical flow sequence as motion guidance information, and injects it into a diffusion model to generate video. However, in this process, the accuracy of the optical flow prediction result for the input image is very low, resulting in very poor visual quality of the generated video. Summary of the Invention

[0003] The purpose of this disclosure is to at least solve one of the aforementioned technical defects. The technical solution provided by the embodiments of this disclosure is as follows: In a first aspect, embodiments of this disclosure provide a method performed by an electronic device, comprising: Obtain the first image features that are globally relevant to the image; The first artificial intelligence (AI) network is used to acquire second image features related to the moving subject in the image. Based on the first and second image features, optical flow prediction is performed at different image scales to obtain optical flow information at each image scale. There is a correlation between the optical flow information at different image scales. Video is generated based on the optical flow information.

[0004] In one feasible embodiment, the first AI network includes a top-down connected multilayer encoder and a bottom-up connected multilayer decoder, and the encoder and decoder at each level are horizontally connected, with different levels corresponding to different image scales; The step of predicting optical flow at different image scales based on the first image features and the second image features to obtain optical flow information at each image scale includes: The first image feature and the second image feature are downsampled step by step at each image scale by the encoder to obtain the output corresponding to each image scale respectively. At each image scale, the decoder upsamples the output of the previous level and the output of the encoder at the same image scale level to obtain the optical flow information corresponding to each image scale.

[0005] In one feasible embodiment, the processing of each of the encoders and / or each of the decoders includes: Spatiotemporal attention processing is performed on the first input information and the second input information respectively to obtain the first output information and the second output information; The first input information of the encoder includes the first image feature or the modulated first image feature in the output of the previous level; the second input information of the encoder includes the second image feature or the modulated second image feature in the output of the previous level. The first input information of the decoder includes the modulated first image features in the output of the previous level and the modulated first image features in the output of the encoder at the same image scale level; the second input information of the decoder includes the modulated second image features in the output of the previous level and the modulated second image features in the output of the encoder at the same image scale level.

[0006] In one feasible embodiment, spatiotemporal attention processing is performed on the second input information to obtain second output information, including: Spatiotemporal attention processing is performed on the second input information to obtain a third image feature corresponding to the moving subject and a fourth image feature corresponding to the template of the moving subject; The fourth image features are transformed to obtain the first template of the moving subject; Based on the first template, features corresponding to the moving subject are obtained from the third image features and used as the second output information after modulation.

[0007] In one feasible embodiment, spatiotemporal attention processing is performed on the first input information to obtain first output information, including: Spatiotemporal attention processing is performed on the first input information to obtain a fifth image feature that is globally corresponding to the image. Based on the first template, a sixth image feature corresponding to the moving subject is obtained from the fifth image feature; Based on the first template, the seventh image feature of the non-moving subject part in the image is obtained from the fifth image feature; Based on the sixth image feature and the seventh image feature, the first output information after modulation is obtained.

[0008] In a feasible embodiment, the step of performing optical flow prediction at different image scales based on the first image features and the second image features to obtain optical flow information for each image scale includes: Based on the first image features and the second image features, feature information is extracted step by step through network layers of different image scales to obtain the first motion information of each image scale. Obtain second motion information for different image scales obtained by processing the first image features at different image scales; For each image scale, optical flow prediction is performed based on the first motion information and the second motion information to obtain the optical flow information for the corresponding image scale.

[0009] In one feasible embodiment, the first image feature includes the image and / or feature information corresponding to the motion trajectory of the moving subject determined based on the image.

[0010] In one feasible embodiment, obtaining the second image features related to the moving subject in the image includes: Obtain a second template corresponding to the moving body; Based on the second template, a second image feature of the region where the moving subject is located is obtained in the image.

[0011] In one feasible embodiment, generating video based on the optical flow information includes: Using a second AI network, based on the optical flow information, a first frequency domain feature related to the motion of pixels in the image is obtained; The video is generated based on the first frequency domain features using a third AI network.

[0012] In one feasible embodiment, obtaining the first frequency domain features related to the motion of pixels in the image based on the optical flow information includes: The optical flow information is Fourier encoded to obtain a first temporal feature related to the motion of pixels in the image; Perform a Fourier transform on the first time-domain feature to obtain the second frequency-domain feature; The second frequency domain feature is transformed at each image scale to obtain the first frequency domain feature for each image scale.

[0013] In one feasible embodiment, the step of performing Fourier encoding on the optical flow information to obtain a first temporal feature related to the motion of pixels in the image includes: The optical flow information is subjected to Fourier transform to obtain the third frequency domain features; Based on the third frequency domain features, a fourth frequency domain feature related to the motion of pixels in the image is obtained; The fourth frequency domain feature is subjected to an inverse Fourier transform to obtain the first time domain feature.

[0014] In one feasible embodiment, generating video based on the first frequency domain features includes: For each image scale, a ninth image feature for the corresponding image scale is obtained based on the first frequency domain feature and the eighth image feature encoded by the image at the corresponding image scale. Videos are generated based on the ninth image features at each image scale.

[0015] In one feasible embodiment, based on the first frequency domain features and the eighth image features encoded by the image at the corresponding image scale, a ninth image feature at the corresponding image scale is obtained, including: Obtain the eighth image feature obtained by encoding the image at the corresponding image scale; Perform a Fourier transform on the eighth image feature to obtain the fifth frequency domain feature; Cross-attention processing is performed based on the first frequency domain feature and the fifth frequency domain feature to obtain the tenth image feature; Perform an inverse Fourier transform on the tenth image feature to obtain the ninth image feature in the time domain.

[0016] In one feasible embodiment, training the first AI network includes: Acquire a sample image and first sample motion information corresponding to the original image scale of the sample image; Based on the sample image, the first AI network is trained to obtain the first predicted motion information output by the first AI network, which corresponds to the scale of the original sample image. Obtain second sample motion information based on the sample image processed at the original image scale; By fusing the first predicted motion information and the second sample motion information, a second predicted motion information is obtained; Based on the second predicted motion information and the first sample motion information, the network parameters of the first AI network are adjusted.

[0017] In a second aspect, embodiments of this disclosure provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in the first aspect and any of its embodiments.

[0018] Thirdly, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect and any of its embodiments.

[0019] Fourthly, embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in the first aspect and any of its embodiments.

[0020] The beneficial effects of the technical solutions provided in this disclosure are: This disclosure provides a method executed by an electronic device. Specifically, it involves acquiring first image features globally related to an image, acquiring second image features related to moving subjects in the image through a first AI network, and performing optical flow prediction at different image scales based on the first and second image features to obtain optical flow information for each image scale. The optical flow information at different image scales is correlated, and a video can then be generated based on this optical flow information. This disclosure enables the acquisition of correlated optical flow information at different image scales, providing more accurate optical flow information for subsequent video generation, thereby improving the visual effect of the generated video. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments of this disclosure will be briefly introduced below.

[0022] Figure 1 A flowchart illustrating a method executed by an electronic device, as provided in this disclosure embodiment; Figure 2 A schematic diagram of an overall framework provided for an embodiment of this disclosure; Figure 3 This is a schematic diagram of the overall structure of an optical flow prediction network provided in an embodiment of the present disclosure; Figure 4 A schematic diagram of a motion pyramid generator network structure provided in an embodiment of this disclosure; Figure 5 A schematic diagram of an MSAM module provided in an embodiment of this disclosure; Figure 6 This is a schematic diagram of the internal structure of an MSAM module provided in an embodiment of the present disclosure; Figure 7 This is a schematic diagram of the internal structure of a motion information extractor provided in an embodiment of the present disclosure; Figure 8 A schematic diagram illustrating a motion information adapter and its position in a video generation architecture, provided as an embodiment of this disclosure; Figure 9 This is a schematic diagram of the internal structure of a motion information adapter provided in an embodiment of the present disclosure; Figure 10 A schematic diagram illustrating a cross-attention processing method provided in an embodiment of this disclosure; Figure 11 A schematic diagram illustrating an application example provided by an embodiment of this disclosure; Figure 12 A schematic diagram illustrating another application example provided by an embodiment of this disclosure; Figure 13a This illustrates the optical flow in an example scenario; Figure 13b Showing with Figure 13a The corresponding motion frequency curve; Figure 14 The motion frequency curve of an example scene is shown; Figure 15 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0023] The following description, with reference to the accompanying drawings, is provided to aid in a thorough understanding of the various embodiments of this disclosure as defined by the claims and their equivalents. This description includes various specific details to aid understanding but should be considered exemplary only. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the various embodiments described herein without departing from the scope and spirit of this disclosure. Furthermore, for clarity and brevity, descriptions of well-known functions and structures may be omitted.

[0024] The terms and wording used in the following description and claims are not limited to their dictionary meanings, but are merely used by the inventors to enable a clear and consistent understanding of this disclosure. Therefore, it will be apparent to those skilled in the art that the following description of various embodiments of this disclosure is for illustrative purposes only and not for limiting the purpose of this disclosure as defined in the appended claims and their equivalents.

[0025] It should be understood that the singular forms of “a,” “an,” and “the” can also include plural references unless the context clearly indicates otherwise. Thus, for example, the reference to “component surface” includes referring to one or more such surfaces. When we say that an element is “connected” or “coupled” to another element, the element can be directly connected or coupled to the other element, or it can mean that the element and the other element are connected through an intermediate element. Furthermore, the use of “connected” or “coupled” herein can include wireless connections or wireless couplings.

[0026] The terms “comprising” or “may include” refer to the presence of a corresponding disclosed function, operation, or component that may be used in the various embodiments of this disclosure, rather than limiting the presence of one or more additional functions, operations, or features. Furthermore, the terms “comprising” or “having” may be interpreted as indicating certain characteristics, numbers, steps, operations, constituent elements, components, or combinations thereof, but should not be construed as excluding the possibility of the presence of one or more other characteristics, numbers, steps, operations, constituent elements, components, or combinations thereof.

[0027] The term "or" as used in the various embodiments of this disclosure includes any of the listed terms and all combinations thereof. For example, "A or B" may include A, may include B, or may include both A and B. When describing multiple (two or more) items, if the relationship between the multiple items is not explicitly defined, the multiple items may refer to one, more, or all of the multiple items. For example, the description "parameter A includes A1, A2, A3" can be implemented as parameter A includes A1 or A2 or A3, or it can be implemented as parameter A includes at least two of the three items A1, A2, and A3.

[0028] The term "based on" as used in the various embodiments of this disclosure can be interpreted as meaning that the premises, conditions, or information upon which it is based are not unique, but at least one or a part of them. That is, it indicates that at least one explicit basis exists, and does not exclude other possible basis.

[0029] Unless otherwise defined, all terms used in this disclosure (including technical or scientific terms) have the same meaning as understood by one of those skilled in the art as described herein. Common terms as defined in dictionaries are to be interpreted as having a meaning consistent with the context in the relevant technical field and should not be interpreted ideally or overly formally unless expressly defined in this disclosure.

[0030] At least some of the functions of the device or electronic device provided in this disclosure embodiment can be implemented by an AI model, such as implementing at least one module of a plurality of modules of the device or electronic device by an AI model. AI-related functions can be executed by non-volatile memory, volatile memory, and a processor.

[0031] The processor may include one or more processors. In this case, the one or more processors may be general-purpose processors, such as central processing unit (CPU), application processor (AP), etc., or pure graphics processing unit, such as graphics processing unit (GPU), visual processing unit (VPU), and / or AI-specific processors, such as neural processing unit (NPU).

[0032] The one or more processors control the processing of input data based on predefined operating rules or artificial intelligence (AI) models stored in non-volatile and volatile memory. These predefined operating rules or AI models are provided through training or learning.

[0033] Here, "providing through learning" refers to obtaining predefined operating rules or an AI model with desired characteristics by applying a learning algorithm to multiple learning datasets. This learning can be performed within the device or electronic device itself, in which the AI ​​is executed according to the embodiment, and / or can be implemented via a separate server / system.

[0034] AI models can contain multiple neural network layers. Each layer has multiple weight values, and each layer performs neural network computations by calculating the input data of that layer (such as the computation results of the previous layer and / or the input data of the AI ​​model) and the multiple weight values ​​of the current layer. Examples of neural networks include, but are not limited to, convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), generative adversarial networks (GANs), and deep Q-networks.

[0035] A learning algorithm is a method of training a predetermined target device (e.g., a robot) using multiple learning data sets to enable, allow, or control the target device to make determinations or predictions. Examples of such learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0036] The methods provided in this disclosure may relate to one or more fields in the technical fields of speech, language, image, video, or data intelligence.

[0037] Optionally, in the context of speech or language, according to this disclosure, in a video generation method executed by an electronic device, a speech signal as an analog signal can be received via a speech module (e.g., a microphone), and the speech portion can be converted into computer-readable text using an Automatic Speech Recognition (ASR) model. The user's utterance intent can be obtained by interpreting the converted text using a Natural Language Understanding (NLU) model. The ASR model or NLU model can be an artificial intelligence (AI) model. The AI ​​model can be processed by a dedicated AI processor designed in a hardware architecture specified for processing the AI ​​model. The AI ​​model can be acquired through training. Here, "acquired through training" means obtaining a predefined operating rule or AI model configured to perform desired features (or purposes) by training a basic AI model with multiple training data using a training algorithm. Language understanding is a technique for recognizing and applying / processing human language / text, including, for example, natural language processing, machine translation, dialogue systems, question answering, or speech recognition / synthesis.

[0038] Optionally, in the context of images or videos, according to this disclosure, in a video generation method executed in an electronic device, output data recognizing images or visual content within images can be obtained by using image data as input data for an artificial intelligence model. The artificial intelligence model can be acquired through training. Here, "acquired through training" means obtaining a predefined operating rule or artificial intelligence model configured to perform desired features (or purposes) by training a basic artificial intelligence model with multiple training data using a training algorithm. The methods of this disclosure can relate to the field of visual understanding in artificial intelligence technology, which is a technique for recognizing and processing things like human vision, and includes, for example, object recognition, object tracking, image retrieval, human recognition, scene recognition, 3D reconstruction / localization, or image enhancement.

[0039] Optionally, in the context of data intelligence processing, according to this disclosure, in a video generation method executed in an electronic device, the method for optical flow prediction can use an artificial intelligence model to predict optical flow information step-by-step at different semantic levels by using features of moving subjects in the image and features of the entire image. The processor of the electronic device can perform preprocessing operations on the data to transform it into a form suitable for use as input to an artificial intelligence model. The artificial intelligence model can be obtained through training. Here, "obtained through training" means obtaining a predefined operating rule or artificial intelligence model configured to perform desired features (or purposes) by training a basic artificial intelligence model with multiple training data using a training algorithm. Inference prediction is a technique for logical reasoning and prediction based on determined information, including, for example, knowledge-based reasoning, optimization prediction, preference-based planning, or recommendation.

[0040] In image and video generation technologies, when using a predicted optical flow sequence from an input image as motion guidance information to inject into a diffusion model to guide video generation, the following problems exist: (1) Inaccurate prediction of optical flow for pixels in the input image leads to disordered video structure and visual jumps in time sequence; (2) The prediction of this scheme is relative to the optical flow of the first frame. For new pixels that do not exist in the first frame, there is a problem of loss of optical flow information in the way optical flow is utilized.

[0041] To address at least one of the aforementioned technical problems, this disclosure proposes a video generation scheme. To improve the accuracy of optical flow prediction, it provides complete motion guidance consistent with the moving subject, avoiding structural loss in the generated video and improving the temporal stability of the optical prediction results. This reduces visual jump defects in video frames. Furthermore, it uses multi-scale optical flow results instead of scaled single-scale results to provide accurate motion guidance, thereby reducing texture distortion of moving objects and improving the accuracy of optical flow guidance information. Additionally, it extracts the temporal changes from the optical flow and maps them to the frequency domain to represent its motion information, providing additional motion supervision information for video generation and reducing the loss of optical flow information.

[0042] The following description of several optional embodiments illustrates the technical solutions of this disclosure and the technical effects produced by these solutions. It should be noted that the following embodiments can be referenced, learned from, or combined with each other. Identical terms, similar features, and similar implementation steps in different embodiments will not be repeated.

[0043] The method executed by an electronic device according to the embodiments of this disclosure will be described in detail below.

[0044] Specifically, such as Figure 1 As shown, the method provided in this embodiment includes S101 to S103.

[0045] S101. Obtain the first image feature that is globally relevant to the image.

[0046] S102. The second image features of the moving subject in the image are obtained through the first artificial intelligence (AI) network. Based on the first and second image features, optical flow prediction is performed at different image scales to obtain optical flow information at different image scales. There is a correlation between the optical flow information at different image scales.

[0047] S103. Generate video based on optical flow information.

[0048] Optionally, the image can be a static image, and the first image feature includes global image features extracted from the image as a whole. The first image feature can be extracted using image feature extraction techniques. For example, such as... Figure 3 As shown, the input image (such as the first frame image) can be used for feature extraction by an image encoder.

[0049] Optionally, the first image features may further include feature information obtained from feature extraction of the motion trajectory drawn on the image. For example, such as... Figure 2 and Figure 3As shown, in an interactive video generation scenario, the operating object can draw motion trajectories in an image for specific moving subjects according to personalized needs. For example, it can specify the final location reached by the moving subject (e.g., a location in the image), and then draw the motion trajectory curve using the subject's current position and the specified position. Alternatively, the operating object can directly draw the motion trajectory curve on the image. During feature extraction, a mask image including the motion trajectory curve can be input into the encoder (e.g., ...). Figure 3 Feature extraction is performed using a sparse encoder (as shown). Motion trajectory curves can be used to define the motion path of a moving subject and can provide the optical flow of pixels in different frames. Inputting motion trajectory curves can effectively improve the accuracy of the generated video.

[0050] Optionally, the moving subject includes one or more objects in the image used to represent the state of motion, such as Figure 2 As shown, the moving subject can be a black swan in the image. In one example, the moving subject can include various objects moving in a video generated from the image, such as a running lion, leaves swaying in the wind, etc.

[0051] Optionally, during the process of the first AI network acquiring feature information related to moving subjects in the image, image recognition technology can be used to identify moving subjects in the image that correspond to the video generation requirements, and then extract the corresponding feature information of the area where the moving subject is located. In the embodiments of this disclosure, such as Figure 4 As shown, the image data input to the first AI network also includes a moving subject mask. This mask can serve as a template to help the AI ​​network extract feature information at corresponding locations in the image as the second image feature of the moving subject. The moving subject mask can be obtained based on the description and drawing of the object being manipulated.

[0052] Optionally, in S102, second image features related to the moving subject in the image are obtained, including S102a to S102b: S102a, Obtain the second template corresponding to the moving subject.

[0053] S102b: Based on the second template, obtain the second image features of the region where the moving subject is located in the image.

[0054] Optionally, the second template can be generated by an AI model based on specific needs, or it can be designed and created by the user. For example, such as... Figures 2 to 4 As shown, the second template can be input into the first AI network in the form of a motion subject mask.

[0055] Optionally, when obtaining feature information of the moving subject in the image based on the second template, regions similar to the template can be searched in the image (e.g., by calculating the similarity between the template and various regions in the image to find the most matching position), and then feature information related to that region can be extracted. For example, such as... Figure 4 As shown, the first AI network may include several feature extraction modules. In one module, after convolution processing of the image and the moving subject template, the convolved moving subject template is encoded by an encoder (such as a Soft-Gated Encoder) using a Soft Gating network based on the Sigmoid function. The encoded information is then fused with the convolved image and downsampled as the input to the next module. After processing by several feature extraction modules, the second image features of the moving subject are output. The Soft Gating network can adaptively locate the focal region of the freeform shape. In the moving subject feature processing flow, a Soft Gating network can be used to adaptively locate the moving subject region by generating a soft subject prior map, and this prior map is used to separate the main complete features into two parts: the object and the background (as in the encoder and / or decoder processing in the following embodiments).

[0056] Optionally, optical flow refers to the motion of pixels in an image sequence, that is, the movement vector of each pixel between consecutive frames. Optical flow prediction can be performed on moving objects in an image, enabling accurate tracking of objects and estimation of their motion trajectories.

[0057] Optionally, during optical flow prediction, network layers at different image scales can be used for step-by-step processing. In this step-by-step processing, each layer's processing is based on the output of the previous layer. This means that in this embodiment, each layer extracts specific information or features from the previous layer and performs further processing on this basis, creating a correlation between optical flow information at different image scales. Furthermore, step-by-step processing can improve the performance of the first AI network. By extracting and utilizing features at different layers, the first AI network can more comprehensively understand the input data and make more accurate optical flow predictions.

[0058] Optionally, considering that optical flow results are typically used at deep semantic levels when used in hidden layer diffusion models, such as at a deep scale of [1 / 8, 1 / 16, 1 / 32, 1 / 64], embodiments of this disclosure design perform optical flow prediction step-by-step at different image scales to obtain optical flow information at each image scale as the basis data for subsequent video generation. For example, as... Figure 4 As shown, different image scales correspond to different processing levels, and step-by-step processing can generate progressively refined feature information.

[0059] This disclosure proposes an AI network for optical flow prediction. This AI network can be termed a motion pyramid generator with motion subject awareness. It can extract second image features of moving subjects from an image and combine these second image features with the first image features obtained from the image to perform optical flow prediction. During the optical flow prediction process, optical flow prediction can be performed step-by-step at different image scales, extracting deeper semantic information layer by layer to obtain optical flow information for each image scale.

[0060] In one feasible embodiment, such as Figure 3 As shown, this disclosure provides an optical flow prediction network that includes two parallel branches. The upper branch may include network structures such as an encoder and a propagation network (e.g., propagationnets), while the lower branch may include a first AI network. In one example, it is also possible to... Figure 3 The optical flow prediction network shown is considered as a first AI network.

[0061] Optionally, in S102, optical flow prediction is performed at different image scales based on the first image features and the second image features to obtain optical flow information for each image scale, including S102a to S102c: S102a: Based on the first image features and the second image features, feature information is extracted step by step through network layers of different image scales to obtain the first motion information of each image scale.

[0062] S102b: Obtain the second motion information of each image scale obtained by processing the first image features at different image scales.

[0063] S102c: For each image scale, optical flow prediction is performed based on the first motion information and the second motion information to obtain the optical flow information of the corresponding image scale.

[0064] Optional, such as Figure 3 As shown, the input to the upper branch can include sparse trajectory points and the input image. After passing through the sparse point encoder and the image encoder, the whole image feature f1 (such as the first image feature) can be obtained. Then, the second motion information corresponding to each image scale can be obtained through the propagation network and the propagation-based feature pyramid network. Among them, the feature of each image scale is obtained by directly downsampling and modulating the feature of the initial layer (such as the feature of the input image).

[0065] Optional, such as Figure 3As shown, the input to the lower-level branch can include the input image, the moving subject template, and the first image features. Through processing by the first AI network, the first motion information corresponding to each image scale can be obtained. Then, the first motion information and second motion information corresponding to each image scale can be fused, and prediction is performed by the prediction head to finally obtain the optical flow information corresponding to each image scale. For example, the motion information can include motion speed, motion region, motion category, etc., and the prediction head can predict the corresponding optical flow sequence based on the fused motion information.

[0066] In this embodiment of the disclosure, after the first AI network extracts deep semantic information layer by layer, it can be fused with the corresponding semantic information obtained from the propagation, and the optical flow result of the corresponding level can be predicted by the prediction head. The optical flow results of multiple levels correspond to the semantic information of each level, and motion pyramid information (such as optical flow information) from coarse to fine can be obtained as motion guidance for the subsequent generation network to generate video.

[0067] Optionally, for the first AI network, this disclosure provides a network structure with a U-shaped architecture (such as Unet), which consists of a symmetrical encoder-decoder structure. By combining the corresponding layers of the encoder and decoder through skip connections, high-resolution detail information can be effectively preserved, which helps to preserve details and process more accurately.

[0068] Optional, such as Figure 4 As shown, the first AI network includes a top-down connected multi-layer encoder and a bottom-up connected multi-layer decoder, with skip connections between encoders and decoders at corresponding levels. Different levels correspond to different image scales. In one example, four levels are included, described in a top-down direction: the first level can process images at 1 / 8 scale, the second level at 1 / 16 scale, the third level at 1 / 32 scale, and the fourth level at 1 / 64 scale.

[0069] Optionally, in step S102, optical flow prediction is performed at different image scales based on the first image features and the second image features to obtain optical flow information for each image scale, including steps A1 to A2: Step A1: The first image features and the second image features are downsampled step by step at each image scale by the encoder to obtain the output corresponding to each image scale.

[0070] Step A2: At each image scale, the decoder upsamples the output of the previous level and the output of the encoder at the same image scale level to obtain the optical flow information corresponding to each image scale.

[0071] Optionally, in the encoder processing, after outputting the first and second image features to the top-level encoder, the encoder performs feature extraction and image downsampling on the input feature information. The input of the next level encoder includes the input of the previous level encoder. As the image passes through the encoder, the size of its feature map gradually decreases, while the depth and complexity of the feature information gradually increase.

[0072] Optionally, in the decoder's processing, the output of the bottom-level encoder and the output of the corresponding level encoder are used as the input to the bottom-level decoder. The decoder upsamples the input feature information. The input to the next level decoder includes the output of the previous level decoder and the output of the corresponding level encoder. As the image passes through the decoder, the size of its feature map gradually increases.

[0073] Optionally, skip connections can concatenate or add feature maps from the encoder to the decoder to fuse information from different image scales.

[0074] In this embodiment, the first AI network employs the Unet framework, extracting features through progressive downsampling encoding and progressive upsampling decoding. A skip connection is used between the encoder and decoder to fuse semantic information from corresponding layers, thereby progressively refining the semantic information. The features output by each layer of the encoder are modulated with deep semantic information, resulting in more accurate semantic information. Optionally, the optical flow predictor also employs the Unet framework, capable of predicting motion information at each depth level that better represents the semantic level of that layer.

[0075] In one feasible embodiment, a U-net network structure based on a Motion Subject Attention Module (MSAM) is provided for the first AI network.

[0076] Considering that the region of a moving subject in a video changes over time, this embodiment of the disclosure divides MSAM into two branches: learning the template of the moving subject region and the features of the moving subject, respectively, thereby achieving shape adjustment of the moving subject region at various levels and feature modulation of the moving subject at various semantic levels. For example... Figure 5 As shown, the MSAM module takes the features of the moving subject and the overall features of the image (such as global image features, also known as global features) as inputs, and outputs the modulated features of the moving subject and the modulated overall features of the image.

[0077] Optionally, applying MSAM at each level of the semantic pyramid allows for adaptive adjustment of the moving subject region at each semantic level. This enables adaptive adjustment of the supervision information of the moving subject at different semantic levels, ensuring the accuracy of the shape and features of the moving subject region at the current semantic level and preserving more accurate motion information. After the overall image features at each semantic level are modulated by MSAM, the integrity and consistency of the overall image motion information can be guaranteed.

[0078] Optionally, the processing of each encoder and / or each decoder includes: performing spatiotemporal attention processing on the first input information and the second input information respectively to obtain the first output information and the second output information.

[0079] The encoder's first input information includes a first image feature or a modulated first image feature from the output of the previous level; the encoder's second input information includes a second image feature or a modulated second image feature from the output of the previous level.

[0080] The first input information of the decoder includes the modulated first image features in the output of the previous level and the modulated first image features in the output of the encoder at the same image scale level; the second input information of the decoder includes the modulated second image features in the output of the previous level and the modulated second image features in the output of the encoder at the same image scale level.

[0081] Optionally, spatial attention allows the network to focus on the moving subject region, while temporal attention can enhance temporal relevance. Since the shape of a moving subject changes during movement, spatial and temporal attention can be used to ensure smooth changes in the moving subject over time.

[0082] Optionally, both the encoder and decoder in the first AI network can use MSAM modules. For example... Figure 5 As shown, MSAM modules at different levels can be used to modulate feature information at different image scales.

[0083] In the embodiments disclosed herein, such as Figure 4 As shown, a soft-gated encoder can be used to aggregate the features of the moving subject at the same level. Gated convolution can provide learnable dynamic feature selection processing, thereby better handling freeform masks. Since the shape of the moving subject may be irregular, gated convolution can help the first AI network to better focus on the moving subject, which has freeform deformation in both space and time. Furthermore, by applying the MSAM designed in this embodiment at the pyramid semantic level and updating the object prior map at multiple image scales (the inputs at different image scale levels are different), the supervision information of the moving subject at different semantic levels can be adaptively adjusted to achieve more accurate motion results.

[0084] The following is combined with Figure 4 and Figure 5 The encoder and decoder will be explained separately.

[0085] In the encoder, the top-level encoder uses a first image feature as the first input information and a second image feature as the second input information. That is, it modulates the first and second image features to obtain a first output information (modulated first image feature) and a second output information (modulated first image feature). Encoders at other levels use the input of the encoder at the level above them as their input. The first input information includes the modulated first image feature from the output of the encoder at the level above, and the second input information includes the modulated second image feature from the output of the encoder at the level above.

[0086] In the decoder, each decoder takes the output of the previous level (which can be either an encoder or a decoder) and the encoder output at the same image scale level as input. The first input information includes modulated first image features from the output of the previous level and modulated first image features from the encoder output at the same image scale level. The second input information includes modulated second image features from the output of the previous level and modulated second image features from the encoder output at the same image scale level. Optionally, motion information corresponding to each image scale can be obtained by predicting based on the outputs of each decoder using a prediction head.

[0087] In this embodiment, when modulating the overall features of an image, the MSAM module can ensure smooth and continuous changes in space and time through spatiotemporal self-attention. It also ensures the integrity of the moving subject features and the overall image coordination by separating and processing the moving subject and background separately. Since the moving subject undergoes deformation during movement, this embodiment increases the transmission of moving subject features to generate an adaptively deformable template, thereby obtaining a more accurate moving subject region for modulating the overall features of the image.

[0088] Optionally, spatiotemporal attention processing is performed on the second input information (such as features of the moving subject) to obtain the second output information, including steps B1 to B3: Step B1: Perform spatiotemporal attention processing on the second input information to obtain the third image features corresponding to the moving subject and the fourth image features corresponding to the template of the moving subject.

[0089] Step B2: Transform the features of the fourth image to obtain the first template of the moving subject.

[0090] Step B3: Based on the first template, obtain the features corresponding to the moving subject from the third image features, and use them as the second output information after modulation.

[0091] Optional, such as Figure 6 As shown, the motion subject features (such as the second input information) are input into the MSAM module and then fused by the spatiotemporal attention module and the convolution module. The generated features consist of two parts. In the channel dimension, one half of the features is the modulated motion subject feature fs (such as the third image feature), and the other half is the modulated motion subject template feature fm (such as the fourth image feature). fm is adjusted to a floating-point number between 0 and 1 by sigmoid and serves as the motion subject template m (such as the first template). The motion subject features in fs are extracted using the template m and output as the modulated motion subject feature f2' (such as the second output information).

[0092] Optionally, spatiotemporal attention processing is performed on the first input information (such as the overall features of the image, also known as whole image features) to obtain the first output information, including steps C1 to C4: Step C1: Perform spatiotemporal attention processing on the first input information to obtain the fifth image feature that corresponds globally to the image.

[0093] Step C2: Based on the first template, obtain the sixth image feature corresponding to the moving subject from the fifth image feature.

[0094] Step C3: Based on the first template, obtain the seventh image feature of the non-moving subject part in the image from the fifth image features.

[0095] Step C4: Based on the sixth and seventh image features, obtain the first output information after modulation.

[0096] Optional, such as Figure 6 As shown, after the overall features (such as the first input information) are input into the MSAM module, they are fused by the spatiotemporal attention module and the convolution module to obtain the modulated overall features fa (such as the fifth image features). The moving subject template m (such as the first template) is used to extract the features fo of the moving subject part in the overall features fa (such as the sixth image features). The 1-m is used to extract the features fb of the background part (i.e. the non-moving subject part) in the overall features fa (such as the seventh image features). The feature extraction and fusion operation is performed on fo and fb to finally output the modulated overall features f1' (such as the first output information).

[0097] Optionally, since the moving subject and the background are independent but interactive, separating and recombining the features of the moving subject and the background can better maintain the integrity and independence of the moving subject.

[0098] In this embodiment, a spatiotemporal attention network is used to process the input features. This network simultaneously considers attention allocation across both temporal and spatial dimensions, adaptively adjusting the model's attention to different time points and spatial locations to improve processing accuracy and efficiency. Specifically, the spatial attention layer focuses on pixels and feature points at different spatial locations, adaptively adjusting the model's attention or weights based on their importance. Similarly, the temporal attention layer focuses on frame images or feature sequences across different time periods, adaptively adjusting the model's attention or weights based on the similarity or importance of these frames or sequences. The features input to the spatiotemporal attention-processed convolutional layer or other types of layers (such as fully connected layers) can then be further processed and used for decision-making.

[0099] In this embodiment, the motion pyramid generator with motion subject awareness employs a Unet-based structure, which can adaptively generate multi-scale coarse-to-fine dense motion information based on feature semantics. Furthermore, the MSAM module can be used as a base module, incorporating attention processing and feature gating to effectively enhance the integrity of the motion subject and make the motion smoother over time. Specifically, the MSAM module combines spatial and temporal attention, allowing the motion subject undergoing deformation along the time axis to learn better weights, and employs a soft-gating network (such as...) Figure 6 The network used in the convolutional layers adaptively generates soft masks in terms of deep semantics.

[0100] In this embodiment, to generate videos with smoother and more natural motion, a motion information extractor in the frequency domain (also known as a frequency domain motion pattern extractor or frequency domain motion pattern encoder) is proposed, and the extracted motion information is fed into a diffusion model, such as... Figure 2 As shown.

[0101] Optional, such as Figure 7 As shown, the input to the motion extractor (such as the second AI network) can be the optical flow sequences of different resolutions predicted in the previous step (such as optical flow information at different image scales). The motion information extractor can include multiple Fourier encoders (such as Fourier coding module 1 to Fourier coding module n). For each image scale optical flow sequence, it can be processed layer by layer, and finally, after Fourier transform and transformation at the corresponding multiple image scales, motion features at different image scales (such as the first frequency domain features) are obtained.

[0102] In a feasible embodiment, video generation based on optical flow information in step S103 includes steps D1 to D2: Step D1: Using the second AI network, based on optical flow information, obtain the first frequency domain features related to the motion of pixels in the image.

[0103] Step D2: Generate video based on the first frequency domain features through the third AI network.

[0104] Optionally, a motion information extractor (such as a second AI network) can be used to extract and learn motion pattern information (such as first frequency domain features) in the frequency domain. Motion patterns can be understood as the trajectories and manners of pixel movement between image frames. In the frequency domain, motion patterns can be extracted by analyzing changes in signal components at different frequencies, which reflect the dynamic differences between image frames.

[0105] Optional, such as Figure 8 As shown, the information obtained through the second AI network can be used as input to the video generation network (such as the third AI network), thereby enabling the use of motion information as guidance and ensuring the integrity and efficiency of motion guidance information.

[0106] In a feasible embodiment, step D1 involves obtaining first frequency domain features related to pixel motion in the image based on optical flow information, including steps D11 to D13: Step D11: Perform Fourier encoding on the optical flow information to obtain the first temporal features related to the motion of pixels in the image.

[0107] Step D12: Perform a Fourier transform on the first time-domain feature to obtain the second frequency-domain feature.

[0108] Step D13: Transform the second frequency domain features at each image scale to obtain the first frequency domain features for each image scale.

[0109] Optionally, the second AI network, also known as a frequency motion pattern encoder, can receive predicted optical flow results from multiple image scales as input, and all inputs can share the same Fourier encoder. For example, Figure 7 As shown, when using optical flow information from different image scales as input to the second AI network, the optical flow information is first encoded using a Fourier encoder to obtain the first temporal features related to the motion of pixels in the image. Then, the first temporal features can be mapped to the frequency domain using a Fourier transform (FFT) to obtain the second frequency domain features. Furthermore, considering that the number of channels varies at different stages of the video generation network, a single head can adaptively transform the output of the shared Fourier encoder to obtain the required feature dimensions for each stage.

[0110] Optionally, the execution order of steps D12 and D13 is not limited. The first time-domain feature can be transformed at each image scale to obtain the time-domain features of each image scale. Then, Fourier transform is performed on each time-domain feature and mapped to the frequency domain to obtain the frequency domain features of each image scale.

[0111] In this embodiment of the disclosure, motion pattern features (such as first frequency domain features) are obtained through a frequency domain motion pattern encoder. This encoder can encode optical flow into frequency domain motion features, where each component corresponds to a specific angle view in the entire input optical flow motion pattern. Through convolutional transformation in the frequency and time domains, the encoder can extract powerful motion pattern information (such as motion speed, motion range, motion category, etc.).

[0112] In a feasible embodiment, step D11 involves Fourier encoding of the optical flow information to obtain a first temporal feature related to the motion of pixels in the image, including steps D111 to D113: Step D111: Perform Fourier transform on the optical flow information to obtain the third frequency domain features; Step D112: Based on the third frequency domain features, obtain the fourth frequency domain features related to the motion of pixels in the image.

[0113] Step D113: Perform an inverse Fourier transform on the fourth frequency domain feature to obtain the first time domain feature.

[0114] Optionally, the Fourier encoder includes multiple Fourier encoding modules, such as... Figure 7 The Fourier coding modules 1 to n shown in the diagram transform the input features to the frequency domain through FFT in each module. Then, motion features in the frequency domain are extracted through a convolutional layer. Finally, the frequency domain features are converted to time domain features using inverse Fourier transform (IFFT).

[0115] Optional, such as Figure 7 As shown, each Fourier encoder module includes a reshape layer, which can be used in data preprocessing or post-processing stages to ensure that the data meets the input or output requirements of the convolutional layer. In one example, during the training of the second AI network, the reshape layer can reshape the input frequency domain features to a dimension suitable for processing by the convolutional neural network.

[0116] In this embodiment, addressing the problem in related technologies that lack optical flow information or provide incorrect optical flow information if no new pixels are found in the first image frame, a method is proposed to additionally utilize optical flow motion information. This method can directly extract the temporal changes from the optical flow and map them to the frequency domain to represent motion information, serving as additional motion supervision information until video generation. Optionally, in this embodiment, stable motion feature representations can be extracted across multiple channels. Furthermore, the proposed second AI network can share a Fourier encoder, reducing the model size.

[0117] In a feasible embodiment, step D2, which generates video based on the first frequency domain features, includes steps D21 to D22: Step D21: For each image scale, based on the first frequency domain feature and the eighth image feature obtained by encoding the image at the corresponding image scale, obtain the ninth image feature for the corresponding image scale.

[0118] Step D22: Generate video based on the ninth image features at each image scale.

[0119] Optional, such as Figure 8 As shown, in a third AI network (such as a diffusion model), the encoder of the network is formed by multiple I2V (Image-to-Video, used for image-to-video generation tasks). Adapting to the outputs of I2V modules at different stages or dimensions, the first frequency domain features of the corresponding image scale are input to the corresponding processing stage of the network for encoding. In one example, the third AI network includes an I2V encoder based on the Unet architecture.

[0120] Optional, such as Figure 8 As shown, to enhance the perception of static spatial features, the warping features obtained through Control Net (a neural network for enhancing image generation) can be fused with the output of the I2V module and the input through the motion information adapter to generate more natural and believable videos. Control Net is an independent control module added to the diffusion model. It can receive external control signals (such as image edges, depth maps, or other morphological information) and fuse them with the intermediate features of the diffusion model, thereby guiding the video generation process.

[0121] Optional, such as Figure 8 As shown, the I2V modules can be connected sequentially. With the addition of a motion information adapter, the input of each I2V module also includes the ninth image feature output from the previous image scale processing.

[0122] Optionally, when obtaining the ninth image features at different image scales, the output of the motion information extractor can be combined with the I2V decoder for decoding, and finally the generated video can be output through the VAE (Variational Autoencoder, a generative model) decoder.

[0123] In this embodiment of the disclosure, in order to make full use of the motion information in the predicted optical flow, it can be processed in the frequency domain by a frequency domain motion pattern encoder to extract smoother motion information. Then, these motion features are input into the motion pattern adapter of a video generation network (such as Stable Video Diffusion, SVD, applicable to diffusion-based video generation technology) to generate more natural and more credible video clips.

[0124] In a feasible embodiment, step D21 involves obtaining a ninth image feature at the corresponding image scale based on the first frequency domain feature and the eighth image feature encoded at the corresponding image scale, including the operations of steps D211 to D214: Step D211: Obtain the eighth image feature that converts the image into video at the corresponding image scale.

[0125] Step D212: Perform Fourier transform on the eighth image features to obtain the fifth frequency domain features.

[0126] Step D213: Perform cross-attention processing based on the first frequency domain features and the fifth frequency domain features to obtain the tenth image features.

[0127] Step D214: Perform an inverse Fourier transform on the tenth image feature to obtain the ninth image feature in the time domain.

[0128] Optional, such as Figure 9 As shown, the input to the motion information adapter includes features output from the I2V module (such as the eighth image feature) and motion pattern features (such as the first frequency domain feature). A Fourier transform (FFT) can be performed on the eighth image feature to obtain the fifth frequency domain feature. Based on this, cross-attention processing can be performed on the first and fifth frequency domain features. Since the processed features still belong to the frequency domain, an inverse Fourier transform can be used to obtain the ninth image feature in the time domain.

[0129] In this embodiment of the disclosure, cross-attention processing can solve the problem that even if the pixel values ​​in the first image frame are not transmitted, information such as the magnitude and speed of motion in time can still be learned, thereby generating a better video.

[0130] Optional, such as Figure 10As shown, the cross-attention between the first and fifth frequency domain features can be viewed as a weighted sum of motion features. This cross-attention processing allows each spatial location in the I2V feature map to perceive the predicted motion pattern in the frequency domain, rather than just static features.

[0131] In this embodiment, the motion information adapter (also known as the motion pattern adapter) is an additional module of the pre-trained I2V Unet. This adapter can be combined with the raw output of the I2V module using a 1x1 conv2D layer initialized to zero (Zero Init). The features output by the I2V module interact with the motion pattern features extracted in the frequency domain by the motion information extractor through cross-attention. In the frequency domain, the I2V Unet encoder can better perceive motion patterns to generate more reliable videos.

[0132] In one example, such as Figures 13a to 13b The example scene shown depicts a video clip of three people dancing. Figure 13a This represents the optical flow of the video in the frequency domain. These images show how the range and amplitude of each person's motion changes. Figure 13b It shows the frequency curves of specific positions of the dancers throughout the video clip. Although the amplitudes differ, it is clear that the movements of the three people are similar.

[0133] In one example, such as Figure 14 As shown, the example scenario is two videos of dogs moving at different speeds (e.g., 1xspeed and 4xspeed). Figure 14 The movement frequency curves of the two dogs are shown. Although they exhibit different patterns due to their different speeds, it is clear that they are making similar movements overall.

[0134] As seen in the two examples above, utilizing the frequency domain representation of optical flow allows for more efficient extraction of motion patterns from video clips. Integrating these frequency-based features into diffusion models (such as I2V U-Net) enhances their ability to integrate motion information, thereby improving their overall performance.

[0135] In one feasible embodiment, training the first AI network includes steps E1 through E5: Step E1: Obtain the sample image and the first sample motion information corresponding to the original image scale of the sample image.

[0136] Step E2: Based on the sample image, train the first AI network to obtain the first predicted motion information output by the first AI network that corresponds to the scale of the original sample image.

[0137] Step E3: Obtain the motion information of the second sample based on the sample image processed at the scale of the original image.

[0138] Step E4: Fuse the first predicted motion information and the second sample motion information to obtain the second predicted motion information.

[0139] Step E5: Adjust the network parameters of the first AI network based on the second predicted motion information and the first sample motion information.

[0140] Optional, such as Figure 3 As shown, to improve the accuracy of optical flow prediction results, optical flow prediction supervision at the original image scale is introduced during the training phase. During training, in addition to obtaining prediction information at different image scales through the prediction head, a fusion network is introduced to fuse the first predicted motion information predicted by the motion pyramid generator for moving subject perception with the second sample motion information obtained by processing sample images at the original image scale, resulting in the second predicted motion information (e.g., ...). Figure 3 Based on the optical flow 1 / 1 shown, loss calculation can be performed based on the second predicted motion information and the first sample motion information to adjust the network parameters of the first AI network.

[0141] In an interactive video generation method based on a diffusion model proposed in this disclosure, the video generation process includes drawing simple motion trajectory curves on an image and drawing the region of desired motion (such as a motion subject template). The motion pyramid generator with motion subject awareness designed in this disclosure can be used to generate precise, dense motion guidance. The frequency domain motion pattern extractor designed in this disclosure can be used to extract motion patterns, and the motion pattern features are injected into the video generation network through a motion pattern adapter.

[0142] In one application example, such as Figure 11 As shown, after the input image is processed, the motion trajectory (such as an upward arrow) of the moving subject (such as a flame) in the image can be drawn, and a moving subject template can be drawn. Based on this, the video generation method provided in this embodiment can generate a video based on the input image, the motion trajectory, and the moving subject template.

[0143] In one example, the video generation process includes: the object of operation specifies the location of motion in the input image, draws a simple motion trajectory curve, predicts the dense motion information of the pyramid (i.e. the optical flow value of each pixel in the image relative to the previous frame image) sequence based on the sparse control trajectory input by the object of operation, extracts the motion mode features in the frequency domain from the dense optical flow, and uses the pyramid motion information and the frequency domain motion mode features as aids to generate the final video frame sequence.

[0144] In another application example, such as Figure 12As shown, after the operating object inputs a first image, a sketch of a moving subject (such as an airplane) can be drawn on the first image. Then, a second image including the moving subject can be generated based on the sketch using image generation technology. The operating object can also draw the motion trajectory of the moving subject on the second image. Based on this, embodiments of this disclosure can obtain a moving subject template from the sketch drawn by the operating object, and combine the first image, the second image, and the motion trajectory to generate a video.

[0145] In this embodiment, more accurate semantic-level motion guidance information is generated. Specifically, a moving subject perception module is added during optical flow prediction, which can better handle the motion effects of moving subjects and non-moving subject regions (such as the background). Furthermore, optical flow information corresponding to semantic information at multiple image scales is directly used as guidance for features at the corresponding image scales, improving the accuracy and completeness of the motion guidance information. This results in higher reliability of the guidance information for the generation network and ultimately higher quality video. Figure 6 As shown, the embodiments of this disclosure can obtain accurate and more time-stable optical flow results. Based on these optical flow results, the generated video structure is more complete, and visual transitions are significantly improved. Furthermore, the use of motion guidance information is more effective, improving the quality of the generated video. Existing technologies implicitly guide the generation process by using motion information to deform pixels in the first frame, resulting in a lack of motion guidance information. Compared to existing technologies, the embodiments of this disclosure directly extract motion patterns from the motion guidance information in the frequency domain, explicitly using the motion information to guide the generation process of the network, resulting in a more complete and efficient use of motion information. After improving the accuracy of the optical flow results, the embodiments of this disclosure provide more accurate motion control for moving subjects, and the generated video better ensures the accuracy of motion and the consistency of the moving subject structure. Especially for parts not appearing in the first frame, relatively high-quality frame images can be generated.

[0146] This disclosure also provides an electronic device including a processor, and optionally, a transceiver and / or memory coupled to the processor, the processor being configured to perform the steps of the method provided in any optional embodiment of this disclosure.

[0147] Figure 15 The diagram shows a structural schematic of an electronic device to which this disclosure applies, such as... Figure 15 As shown, Figure 15The illustrated electronic device 4000 includes a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of this disclosure. Optionally, the electronic device may be a device with an interactive function module, a server, etc.

[0148] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with this disclosure. Processor 4001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0149] Bus 4002 may include a pathway for transmitting information between the aforementioned components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 4002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 15 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0150] The memory 4003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium capable of carrying or storing computer programs and capable of being read by a computer, without limitation herein.

[0151] The memory 4003 is used to store computer programs that execute embodiments of the present disclosure, and is controlled by the processor 4001 to execute them. The processor 4001 is used to execute the computer programs stored in the memory 4003 to implement the steps shown in the foregoing method embodiments.

[0152] This disclosure provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the steps and corresponding content of the aforementioned method embodiments.

[0153] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, can implement the steps and corresponding content of the aforementioned method embodiments.

[0154] The terms “first,” “second,” “third,” “fourth,” “1,” “2,” etc. (if present) in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in a sequence other than that shown in the figures or text.

[0155] It should be understood that although arrows indicate various operation steps in the flowcharts of the embodiments of this disclosure, the order in which these steps are implemented is not limited to the order indicated by the arrows. Unless explicitly stated herein, in some implementation scenarios of the embodiments of this disclosure, the implementation steps in each flowchart can be executed in other orders as required. Furthermore, some or all of the steps in each flowchart may include multiple sub-steps or multiple stages based on the actual implementation scenario. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage can also be executed at different times. In scenarios where execution times differ, the execution order of these sub-steps or stages can be flexibly configured as required, and the embodiments of this disclosure do not limit this.

[0156] The above text and accompanying drawings are provided as examples only to help the reader understand this disclosure. They are not intended and should not be construed as limiting the scope of this disclosure in any way. Although certain embodiments and examples have been provided, it will be apparent to those skilled in the art, based on the content disclosed herein, that changes can be made to the illustrated embodiments and examples, and other similar implementations based on the technical concept of this disclosure can be adopted without departing from the scope of this disclosure, and these modifications and modifications are also within the protection scope of the embodiments of this disclosure.

Claims

1. A method performed by an electronic device, characterized in that, include: Obtain the first image features that are globally relevant to the image; The first artificial intelligence (AI) network is used to acquire second image features related to the moving subject in the image. Based on the first and second image features, optical flow prediction is performed at different image scales to obtain optical flow information at each image scale. There is a correlation between the optical flow information at different image scales. Video is generated based on the optical flow information.

2. The method according to claim 1, characterized in that, The first AI network includes a top-down connected multi-layer encoder and a bottom-up connected multi-layer decoder, with skip connections between the encoder and decoder at corresponding levels, and different levels corresponding to different image scales; The step of predicting optical flow at different image scales based on the first image features and the second image features to obtain optical flow information at each image scale includes: The first image feature and the second image feature are downsampled step by step at each image scale by the encoder to obtain the output corresponding to each image scale respectively. At each image scale, the decoder upsamples the output of the previous level and the output of the encoder at the same image scale level to obtain the optical flow information corresponding to each image scale.

3. The method according to claim 2, characterized in that, The processing of each of the encoders and / or decoders includes: Spatiotemporal attention processing is performed on the first input information and the second input information respectively to obtain the first output information and the second output information; The first input information of the encoder includes the first image feature or the modulated first image feature in the output of the previous level; the second input information of the encoder includes the second image feature or the modulated second image feature in the output of the previous level. The first input information of the decoder includes the modulated first image features in the output of the previous level and the modulated first image features in the output of the encoder at the same image scale level; the second input information of the decoder includes the modulated second image features in the output of the previous level and the modulated second image features in the output of the encoder at the same image scale level.

4. The method according to claim 3, characterized in that, The second input information is subjected to spatiotemporal attention processing to obtain the second output information, including: Spatiotemporal attention processing is performed on the second input information to obtain a third image feature corresponding to the moving subject and a fourth image feature corresponding to the template of the moving subject; The fourth image features are transformed to obtain the first template of the moving subject; Based on the first template, features corresponding to the moving subject are obtained from the third image features and used as the second output information after modulation.

5. The method according to claim 4, characterized in that, Spatiotemporal attention processing is performed on the first input information to obtain the first output information, including: Spatiotemporal attention processing is performed on the first input information to obtain a fifth image feature that is globally corresponding to the image. Based on the first template, a sixth image feature corresponding to the moving subject is obtained from the fifth image feature; Based on the first template, the seventh image feature of the non-moving subject part in the image is obtained from the fifth image feature; Based on the sixth image feature and the seventh image feature, the first output information after modulation is obtained.

6. The method according to claim 1, characterized in that, The step of predicting optical flow at different image scales based on the first image features and the second image features to obtain optical flow information at each image scale includes: Based on the first image features and the second image features, feature information is extracted step by step through network layers of different image scales to obtain the first motion information of each image scale. Obtain second motion information for different image scales obtained by processing the first image features at different image scales; For each image scale, optical flow prediction is performed based on the first motion information and the second motion information to obtain the optical flow information for the corresponding image scale.

7. The method according to claim 1 or 6, characterized in that, The first image feature includes the image and / or feature information corresponding to the motion trajectory of the moving subject determined based on the image.

8. The method according to claim 1, characterized in that, The acquisition of second image features related to a moving subject in the image includes: Obtain a second template corresponding to the moving body; Based on the second template, a second image feature of the region where the moving subject is located is obtained in the image.

9. The method according to claim 1 or 6, characterized in that, The generation of video based on the optical flow information includes: Using a second AI network, based on the optical flow information, a first frequency domain feature related to the motion of pixels in the image is obtained; The video is generated based on the first frequency domain features using a third AI network.

10. The method according to claim 9, characterized in that, The step of obtaining the first frequency domain features related to the motion of pixels in the image based on the optical flow information includes: The optical flow information is Fourier encoded to obtain a first temporal feature related to the motion of pixels in the image; Perform a Fourier transform on the first time-domain feature to obtain the second frequency-domain feature; The second frequency domain feature is transformed at each image scale to obtain the first frequency domain feature for each image scale.

11. The method according to claim 10, characterized in that, The step of performing Fourier encoding on the optical flow information to obtain first temporal features related to the motion of pixels in the image includes: The optical flow information is subjected to Fourier transform to obtain the third frequency domain features; Based on the third frequency domain features, a fourth frequency domain feature related to the motion of pixels in the image is obtained; The fourth frequency domain feature is subjected to an inverse Fourier transform to obtain the first time domain feature.

12. The method according to claim 9, characterized in that, The process of generating video based on the first frequency domain features includes: For each image scale, a ninth image feature for the corresponding image scale is obtained based on the first frequency domain feature and the eighth image feature encoded by the image at the corresponding image scale. Videos are generated based on the ninth image features at each image scale.

13. The method according to claim 12, characterized in that, The step of obtaining the ninth image feature at the corresponding image scale based on the first frequency domain feature and the eighth image feature encoded at the corresponding image scale includes: Obtain the eighth image feature obtained by encoding the image at the corresponding image scale; Perform a Fourier transform on the eighth image feature to obtain the fifth frequency domain feature; Cross-attention processing is performed based on the first frequency domain feature and the fifth frequency domain feature to obtain the tenth image feature; Perform an inverse Fourier transform on the tenth image feature to obtain the ninth image feature in the time domain.

14. The method according to claim 1, characterized in that, The training of the first AI network includes: Acquire a sample image and first sample motion information corresponding to the original image scale of the sample image; Based on the sample image, the first AI network is trained to obtain the first predicted motion information output by the first AI network, which corresponds to the scale of the original sample image. Obtain second sample motion information based on the sample image processed at the original image scale; By fusing the first predicted motion information and the second sample motion information, a second predicted motion information is obtained; Based on the second predicted motion information and the first sample motion information, the network parameters of the first AI network are adjusted.

15. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 14.

Images