An inter-frame generation method, a method for training an image deformation model, and an electronic device

Abstract

Embodiments of the present application provide a frame insertion generation method, a method for training an image deformation model and an electronic device, which relate to the field of AI technology, and can reduce the computational overhead of the image deformation model when processing an image deformation task and improve the efficiency of the device in processing the image deformation task. The method can iteratively train a student-teacher network based on training samples, update the model parameters of the student-teacher network after the last iteration when the student-teacher network does not meet the convergence condition, and stop the iterative training when the student-teacher network meets the convergence condition, and use the student network after the last iteration as the image deformation model.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI), and more particularly to a frame interpolation generation method, a method for training an image deformation model, and an electronic device. Background Technology

[0002] Mobile devices such as smartphones and tablets can utilize image processing models based on deep neural networks (DNNs) to handle computer vision (CV) tasks. These CV tasks can include image warping, face tracking, expression transfer, and super-resolution reconstruction. Currently, image processing models can implement operations such as image warping, face tracking, and expression transfer using computationally intensive operators such as grid sampling.

[0003] However, due to the limited hardware resources of mobile devices, they cannot withstand the computational overhead of the operators in the above image processing model, which severely restricts the practical application of the image processing model on terminal devices. Summary of the Invention

[0004] This application provides a frame interpolation generation method, a method for training an image deformation model, and an electronic device, which can reduce the computational overhead of the image deformation model when processing image deformation tasks and improve the efficiency of the device in processing image deformation tasks.

[0005] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0006] Firstly, embodiments of this application provide a method for training an image deformation model. This method can be applied to electronic devices such as terminals or servers. The method includes: iteratively training a student-teacher network based on training samples; updating the model parameters of the student-teacher network after the most recent iteration when the student-teacher network does not meet the convergence condition; and stopping the iterative training when the student-teacher network meets the convergence condition, using the student network after the last iteration as the image deformation model. The aforementioned student-teacher network includes a teacher network and a student network. The teacher network is a model that deforms images through image deformation operations, and the student network is a model that deforms images through convolution operations. Furthermore, this application does not limit the structure of the student network, as long as the first deformed image and the second deformed image have the same size. The first deformed image is the image output by the teacher network, and the second deformed image is the image output by the student network.

[0007] In other words, this application transforms image deformation operations (e.g., operations implemented through Grid Sample) into a series of learnable convolution operations through knowledge distillation, enabling the student network to perform image deformation processing simply through convolution operations. This makes the trained image deformation module more adaptable to the model running environment of mobile devices (e.g., neural network processors), thereby reducing computational overhead and shortening runtime, effectively improving the efficiency of mobile devices in processing image deformation tasks.

[0008] In one embodiment provided in the first aspect, the training samples include training images, training optical flow, and target images. During any iteration of training, the training images and training optical flow can be input into the student-teacher network after the most recent iteration to obtain a first deformed image and a second deformed image. The model parameters of the student-teacher network after the most recent iteration are then updated according to a target loss function. The target loss function is associated with a first difference and a second difference, where the first difference is the difference between the first deformed image and the second deformed image, and the second difference is the difference between the second deformed image and the target image. Thus, during iterative training, the difference between the first deformed image and the second deformed image can be reduced, enabling the student network to learn the image deformation capabilities of the teacher network. Simultaneously, the difference between the second deformed image and the target image can be reduced, improving the student network's ability to obtain accurate image deformation results. In other words, adjusting the model parameters based on the target loss function enables the final image deformation model to possess the ability to accurately deform images.

[0009] In one embodiment provided in the first aspect, before updating the model parameters of the student-teacher network after the most recent iteration according to the target loss function, the method further includes: determining a first loss function based on a first deformed image and a second deformed image, determining a second loss function based on a second deformed image and a target image, and obtaining a target loss function by weighted summation of the first loss function and the second loss function.

[0010] In one embodiment provided in the first aspect, when updating the model parameters of the student-teacher network after the most recent iteration according to the target loss function, the electronic device may update the model parameters of the teacher network after the most recent iteration according to the target loss function if the actual number of iterations is less than or equal to the first expected number of iterations, where the actual number of iterations is the number of times the student-teacher network is iteratively trained; and if the actual number of iterations is greater than the first expected number of iterations and less than or equal to the second expected number of iterations, the electronic device may update the model parameters of the student network after the most recent iteration according to the target loss function if the actual number of iterations is greater than the first expected number of iterations and less than or equal to the second expected number of iterations, where the second expected number of iterations is greater than the first expected number of iterations.

[0011] In other words, the server can first keep the initial image deformation model parameters unchanged and update the deformation operation model parameters. After confirming that the deformation operation model has converged, the server then keeps the deformation operation model parameters unchanged and updates the initial image deformation model parameters again. In other words, adjusting the student network's model parameters after the teacher network has converged ensures that the image deformation capability learned by the student network is a more accurate representation of the converged teacher network's capability.

[0012] In one embodiment provided by the first aspect, when the actual number of iterations is less than or equal to the first expected number of iterations, the weight corresponding to the second loss function is the first weight; when the actual number of iterations is greater than the first expected number of iterations but less than the second expected number of iterations, the weight corresponding to the second loss function is the second weight, and the first weight is less than the second weight.

[0013] This approach allows for a greater proportion of the ground truth distillation loss (i.e., the second loss function) in the target loss function during the early stages of training, providing stronger supervision signals. Conversely, reducing the proportion of the second loss function in the target loss function during the later stages increases the proportion of the first loss function, which is more conducive to the student network (i.e., the initial image deformation model) developing its own feature representations. This progressive training method enables the student network to learn and generalize better and also reduces the risk of model overfitting to some extent.

[0014] In one implementation provided in the first aspect, the student-teacher network satisfies the convergence condition when the actual number of iterations equals the second expected number of iterations. Optionally, the student-teacher network can be determined to satisfy the convergence condition when the objective loss function is less than a threshold.

[0015] In one embodiment provided in the first aspect, the student network is used to downsample the training image n times to obtain image feature i, and to downsample the training optical flow n times to obtain optical flow feature i, i = 1 to n. The image feature i and the optical flow feature i have the same size, while the sizes of image feature 1 to image feature n and optical flow feature 1 to optical flow feature n are different. The student network is also used to upsample an intermediate image n n times to obtain a deformed image, where the intermediate image n is obtained based on the image feature n and the optical flow feature n. That is, the image deformation module can extract features at different scales to reflect the details of the training image and the training optical flow in different receptive fields. This makes the final deformed image also contain the local details and overall style of the training image, making it more realistic.

[0016] In one embodiment provided in the first aspect, the student network includes n downsampling groups and n upsampling modules. Each downsampling group includes two downsampling modules, which are used to downsample the training image and the training optical flow respectively. The n upsampling modules are used to upsample the intermediate image n n times to obtain the deformed image.

[0017] In one embodiment provided in the first aspect, the network structures of the n downsampling groups are different, while the network structures of the two downsampling modules in one downsampling group are the same. The different network structures of the n downsampling groups can yield image features and optical flow features at different scales, while the identical network structures of the two downsampling modules in one downsampling group ensure that the dimensions of image feature i and optical flow feature i are the same.

[0018] Secondly, embodiments of this application provide a frame interpolation generation method, which includes: acquiring a first image and a second image from a video to be interpolated, wherein the first image and the second image are two consecutive frames; and inputting the first image and the second image into a frame interpolation network to obtain an interpolated image. The frame interpolation network includes an image deformation model, which is used to deform the first image and the second image to obtain a deformed image. The interpolated image is obtained based on the deformed image. The image deformation model is obtained by iteratively training a student-teacher network, which includes a teacher network and a student network. The teacher network is a model that deforms the image through image deformation operations, and the student network is a model that deforms the image through convolution operations. The first deformed image and the second deformed image have the same size. The first deformed image is the image output by the teacher network, and the second deformed image is the image output by the student network. The image deformation model is a student network that satisfies a convergence condition.

[0019] In one embodiment provided in the second aspect, the process of training the student-teacher network includes: iteratively training the student-teacher network based on training samples; updating the model parameters of the student-teacher network after the most recent iteration when the student-teacher network does not meet the convergence condition; stopping the iterative training when the student-teacher network meets the convergence condition; and using the student network after the last iteration as the image deformation model.

[0020] In one embodiment provided in the second aspect, the training samples include training images, training optical flow, and a target image. The process of training the student-teacher network further includes: during any iteration of training, inputting the training images and training optical flow into the student-teacher network after the most recent iteration to obtain a first deformed image and a second deformed image. Updating the model parameters of the student-teacher network after the most recent iteration includes: updating the model parameters of the student-teacher network after the most recent iteration according to a target loss function, wherein the target loss function is associated with a first difference and a second difference, the first difference being the difference between the first deformed image and the second deformed image, and the second difference being the difference between the second deformed image and the target image.

[0021] In one embodiment provided in the second aspect, the frame interpolation network further includes an optical flow estimation module and a fusion module. The above-mentioned inputting the first image and the second image into the frame interpolation network to obtain the interpolated image includes: inputting the first image and the second image into the optical flow estimation module to obtain a first optical flow and a second optical flow, wherein the first optical flow is the optical flow from the first image to the interpolated image, and the second optical flow is the optical flow from the second image to the interpolated image; performing deformation processing on the first image based on the first optical flow using an image deformation model to obtain a third deformed image; performing deformation processing on the second image based on the second optical flow using an image deformation model to obtain a fourth deformed image; and fusing the third deformed image and the fourth deformed image using the fusion module to obtain the interpolated image.

[0022] In one embodiment provided in the second aspect, the frame interpolation network includes an optical flow estimation module, a pooling module, multiple image warping modules, and a feature fusion module. The above-mentioned inputting the first image and the second image into the frame interpolation network to obtain the interpolated image includes: inputting the first image and the second image into the optical flow estimation module to obtain a first optical flow and a second optical flow, wherein the first optical flow is the optical flow from the first image to the interpolated image, and the second optical flow is the optical flow from the second image to the interpolated image; performing pooling processing on the first image, the second image, the first optical flow, and the second optical flow through the pooling module to obtain multiple sets of feature maps, each set of feature maps including the first image, the second image, the first optical flow, and the second optical flow of the same size; inputting the multiple sets of feature maps into the multiple image warping modules respectively to obtain multiple fifth warped images, wherein the multiple fifth warped images have different sizes; and fusing the multiple fifth warped images through the feature fusion module to obtain the interpolated image.

[0023] In one embodiment provided in the second aspect, the image deformation module includes an image deformation model and a fusion module. For each image deformation module, the first image is deformed based on the first optical flow through the image deformation model to obtain a third deformed image, the second image is deformed based on the second optical flow through the image deformation model to obtain a fourth deformed image, and the third deformed image and the fourth deformed image are fused through the fusion module to obtain a fifth deformed image.

[0024] Thirdly, embodiments of this application also provide an electronic device, which includes: a memory and one or more processors; the memory and processors are coupled; the memory is used to store computer program code, the computer program code including computer instructions, which, when executed by the electronic device, cause the electronic device to perform the methods as described in the first aspect, the second aspect, and any one of their embodiments.

[0025] Fourthly, embodiments of this application also provide a computer-readable storage medium storing computer instructions that, when executed in an electronic device, cause the electronic device to perform the methods described in the first aspect, the second aspect, and any one of their embodiments.

[0026] Fifthly, embodiments of this application provide a computer program product including computer instructions. When the computer instructions are executed on an electronic device, they cause the electronic device to perform the methods described in the first aspect, the second aspect, and any of their embodiments.

[0027] The technical effects of any of the design methods in aspects two through five can be found in the technical effects of different design methods in aspect one, and will not be repeated here. Attached Figure Description

[0028] Figure 1 This is a flowchart of the processing of a frame interpolation network;

[0029] Figure 2 A pie chart showing the runtime of each operator of the frame interpolation network when running on a mobile phone;

[0030] Figure 3 A schematic diagram of the AI ​​system provided in this application;

[0031] Figure 4 This is a flowchart illustrating a frame interpolation generation method provided in an embodiment of this application;

[0032] Figure 5 A network structure diagram of a frame interpolation network provided in an embodiment of this application;

[0033] Figure 6 A network structure diagram of another frame interpolation network provided in this application embodiment;

[0034] Figure 7 A network structure diagram of an image deformation model provided in an embodiment of this application;

[0035] Figure 8 A network structure diagram of another image deformation model provided in this application embodiment;

[0036] Figure 9This is a flowchart illustrating a method for training an image deformation model according to an embodiment of this application;

[0037] Figure 10 This is a schematic diagram of the structure of a chip system provided in an embodiment of this application;

[0038] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0039] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this embodiment, unless otherwise stated, "a plurality of" means two or more.

[0040] In this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being better or more advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner for ease of understanding.

[0041] To ensure clarity and brevity in the description of the following embodiments, a brief introduction to the relevant technologies is given first.

[0042] In typical technical solutions, image processing models can perform image processing tasks by combining convolutional layers and operators. Taking image morphing models as an example, they can be applied to video playback scenarios. In video playback scenarios, the terminal device can take two adjacent image frames from the video stream and input them into the image morphing model to obtain the target image frame. Then, the target image frame is inserted as an intermediate frame between the two image frames to increase the video frame rate and reduce motion blur.

[0043] Please see Figure 1 This provides a flowchart of a frame interpolation network for related technologies. For example... Figure 1 The frame interpolation network includes at least an optical flow estimation module, an image warping module, and a fusion module. Specifically, inputting image 1 and image 2 into the optical flow estimation module yields optical flow 1 and optical flow 2. Then, inputting optical flow 1 and image 1 into the image warping module yields deformed image 1. Similarly, inputting optical flow 2 and image 2 into the image warping module yields deformed image 2. Finally, inputting deformed image 1 and deformed image 2 into the fusion module yields the target image.

[0044] The optical flow estimation module may include convolutional layers, rectified linear units (ReLU), and slice operators. The image warping module may include the grid sampling operator, a commonly used function for frame warping, which can directly sample the input frame (e.g., image 1 and image 2) based on the sampling grid in the optical flow map (e.g., optical flow 1 and optical flow 2 mentioned above). The fusion module may include the contact operator, which can be used to stitch deformed image 1 and deformed image 2 together.

[0045] The computational overhead of the frame interpolation network described above is minimal for devices with strong computing power, such as personal computers (PCs). However, when ported to mobile devices like smartphones, it is typically processed by the device's Neural Processing Unit (NPU). The computational load of the frame interpolation network places a significant burden on the NPU, resulting in longer runtimes for the network on mobile devices. Please refer to [link to relevant documentation]. Figure 2 This is a pie chart showing the runtime of each operator in the frame interpolation network running on a mobile phone. For example... Figure 2 As shown, the runtime of Grid Sample accounts for 45.7% of the total runtime of the frame interpolation network, the runtime of Slice accounts for 15.2%, the runtime of Contact accounts for 12.4%, and the runtime of other operators accounts for 26.7%. This demonstrates that Grid Sample has a high computational overhead, causing its runtime to significantly impact the runtime of the frame interpolation network, which limits its application on mobile devices.

[0046] To at least address the aforementioned problems, embodiments of this application provide a frame interpolation generation method, a method for training an image deformation model, and an electronic device. These methods can transfer complex image deformation computation capabilities to a fully convolutional image processing model through knowledge distillation, enabling the image processing model to achieve image deformation simply through convolution operations. Since convolution operations have lower computational overhead than image deformation operations, the image processing model provided in this application has lower computational overhead and runtime, making it easier to port to mobile devices.

[0047] like Figure 3 As shown, Figure 3 A schematic diagram of the structure of the AI ​​system provided in this application. Figure 3 As shown, the AI ​​system includes a data center and multiple terminals (such as...). Figure 3As shown in terminals 111 and 112, the data center can communicate with the terminals via a network, which can be the Internet or other networks. This network can include one or more network devices, such as routers or switches.

[0048] A data center includes one or more servers, such as Figure 3 The server 120 shown is, for example, an application server that supports application services, which can provide video services, image services, game services, and other AI processing services based on video or images. In an alternative scenario, server 120 refers to a server cluster with multiple servers deployed. This server cluster may have racks, and the racks can establish communication between the multiple servers via wired connections, such as Universal Serial Bus (USB) or Peripheral Component Interconnect Express (PCIe) high-speed buses.

[0049] Server 120 can also acquire data from terminals, perform AI processing on the data, and send the AI ​​processing results to the corresponding terminals. This AI processing can refer to using AI models to perform tasks such as image deformation, object recognition, target detection, and image classification, or it can refer to acquiring AI models that meet specific requirements based on samples collected by the terminals.

[0050] in addition, Figure 3 The data center shown may also include other physical devices with AI processing capabilities, such as mobile phones, tablets, or other devices.

[0051] A terminal can also be called a terminal device, user equipment (UE), mobile station (MS), or mobile terminal (MT). A terminal can be a mobile phone (such as...). Figure 3 The terminal 111 shown has mobile payment function and facial recognition payment device, and camera device with data (such as image or video) acquisition and processing function (such as...). Figure 3The terminal shown (112) can also be a tablet computer, a computer with wireless transceiver capabilities, a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, a wireless terminal in transportation safety, a wireless terminal in a smart city, a wireless terminal in a smart home, and so on. The embodiments of this application do not limit the specific technology or device form used in the terminal device.

[0052] It is worth noting that the terminal can access AI models stored on the server 120, and then use these AI models to perform various tasks on the information contained in the image. For example, the AI ​​model can be the aforementioned image processing model, or a model that performs operations such as target detection, object recognition, or classification on the data.

[0053] Figure 3 These are merely illustrative images and should not be construed as limiting the scope of this application. The embodiments of this application can be used in scenarios such as frame interpolation, face tracking, expression transfer, and super-resolution reconstruction for terminals and servers, and no specific limitations are set herein.

[0054] Below Figure 3 Based on the AI ​​system shown, this embodiment provides a frame interpolation generation method. Please refer to [link to relevant documentation]. Figure 4 This is a flowchart illustrating a frame interpolation generation method provided in this embodiment. This frame interpolation generation method can be applied to... Figure 3 The AI ​​system shown can be trained by a terminal or a server. Here, we will take the execution of the frame interpolation generation method provided in this embodiment by a terminal as an example.

[0055] like Figure 4 As shown, this embodiment provides a frame interpolation generation method including S401 to S402.

[0056] S401, the terminal acquires image 1 and image 2.

[0057] In this embodiment, Image 1 and Image 2 can be two adjacent frames in the video to be interpolated. This application uses Image 1 as the preceding frame of Image 2 as an example. In an optional implementation, Image 1 can also be referred to as the first image, and Image 2 as the second image.

[0058] S402, the terminal inputs Image 1 and Image 2 into the frame interpolation network to obtain the interpolated image.

[0059] In this embodiment of the application, the frame interpolation network is used to obtain the interpolated image based on image 1 and image 2.

[0060] Please see Figure 5 This is a network structure diagram of a frame interpolation network provided in an embodiment of this application. Figure 5 As shown, the frame interpolation network includes an optical flow estimation module and an image deformation module. The optical flow estimation module estimates the optical flow 1 from image 1 to the interpolated image, and the optical flow 2 from image 2 to the interpolated image. When determining optical flow 1 and optical flow 2, the terminal can input image 1 and image 2 into the optical flow estimation module. The module first calculates the intermediate optical flow 1 from image 1 to image 2, and the intermediate optical flow 2 from image 2 to image 1. Then, based on an interpolation method, it obtains optical flow 1 and optical flow 2 from the intermediate optical flow 1 and intermediate optical flow 2. In an optional implementation, optical flow 1 can also be referred to as the first optical flow, and optical flow 2 can also be referred to as the second optical flow.

[0061] The image warping module can be used to obtain interpolated images based on image 1, image 2, optical flow 1, and optical flow 2.

[0062] Please continue reading. Figure 5 The image deformation module includes image deformation model 1, image deformation model 2, and an image fusion module. The optical flow estimation module inputs optical flow 1 into image deformation model 1 and optical flow 2 into image deformation model 2. Image deformation model 1 deforms image 1 based on optical flow 1 to obtain deformed image 1 (also called the third deformed image), and image deformation model 2 deforms image 2 based on optical flow 2 to obtain deformed image 2 (also called the fourth deformed image). Deformed image 1 is the image obtained by moving pixels in image 1 based on optical flow 1, and deformed image 2 is the image obtained by moving pixels in image 2 based on optical flow 2.

[0063] Then, image deformation model 1 can input deformed image 1 into the image fusion module, and image deformation model 2 can input deformed image 2 into the image fusion module. The image fusion module can fuse deformed image 1 and deformed image 2 to obtain the interpolated image.

[0064] It should be noted that both image deformation model 1 and image deformation model 2 described above are image deformation models with the same structure and parameters. In the embodiments of this application, the image deformation model is composed of convolutional layers. The image deformation model can achieve image distortion and deformation by performing convolution operations on the input image (such as image 1 or image 2 described above).

[0065] Understandably, since an NPU is a processor specifically designed for AI and machine learning tasks, especially optimized for neural network computations, it primarily accelerates the inference and training processes of deep learning models, particularly excelling in handling neural networks such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs). In other words, NPUs are better suited for handling convolutional operations than image morphing operations. Therefore, running an image morphing model composed of convolutional layers on an NPU is more efficient, with less computation and runtime compared to performing image morphing operations on the NPU itself, making these image morphing models and frame interpolation networks easier to port to mobile devices.

[0066] Please see Figure 6 This is a network structure diagram of another frame interpolation network provided in an embodiment of this application. Figure 6 As shown, the frame interpolation network may include an optical flow estimation module, a pooling module, multiple image deformation modules, and a feature fusion module. The pooling module performs pooling operations on the input to obtain multiple outputs of different sizes, and the feature fusion module fuses multiple inputs of different sizes to obtain the output.

[0067] Please continue reading. Figure 6 First, the terminal can input Images 1 and 2 into the optical flow estimation module and the pooling module. The optical flow estimation module can obtain Optical Flow 1 and Optical Flow 2 from Images 1 and 2, and then input Optical Flow 1 and Optical Flow 2 into the pooling module. The pooling module can then perform pooling operations of different sizes on the inputs (including Optical Flow 1, Optical Flow 2, Images 1 and 2) to obtain multiple sets of feature maps. Each set of feature maps includes Optical Flow 1, Optical Flow 2, Images 1 and 2 of the same size, while the multiple sets of feature maps have different sizes. The pooling module can then input these multiple sets of feature maps into multiple image deformation modules to obtain deformed images j, where j = 1 to m, and m is the number of image deformation modules. Finally, the multiple image deformation modules can input the obtained deformed images j into the feature fusion module, which performs fusion processing on the multiple deformed images j to obtain the interpolated image.

[0068] It can be seen that, Figure 6 The frame interpolation network shown can acquire features at different scales through pooling modules, and obtain deformed images j at different scales based on these features. Finally, the deformed images j at different scales are fused to obtain the interpolated image. This network can simulate the multi-scale characteristics of image deformation operations and capture both the overall and local details of the image, making the output of the frame interpolation network (i.e., the interpolated image) more accurate. At the same time, the structure of the above frame interpolation network is simple, resulting in less computation for the NPU and effectively saving runtime.

[0069] The previous section introduced that the image deformation model is composed of convolutional layers. The following section, with reference to the accompanying diagram, introduces a possible structure of the image deformation model.

[0070] Please see Figure 7 This is a network structure diagram of an image deformation model provided in an embodiment of this application. Figure 7 As shown, the image deformation model may include convolutional layer 1, convolutional layer 2, convolutional layer 3, convolutional layer 4, convolutional layer 5, and convolutional layer 6. The terminal can input image 1 into convolutional layer 1 to obtain features. Figure 1 Optical flow 1 is input into convolutional layer 2 to obtain features. Figure 2 Then, the features are fused. Figure 1 and characteristics Figure 2 Obtain features Figure 3 Next, the features Figure 3 Features are obtained by inputting them into convolutional layer 3 and convolutional layer 4 respectively. Figure 4 and characteristics Figure 5 and integrate features Figure 4 and characteristics Figure 5 The obtained features Figure 6 Then the features Figure 6 Features are obtained by inputting them into convolutional layer 5 and convolutional layer 6 respectively. Figure 7 and characteristics Figure 8 Finally, the features are merged. Figure 7 and characteristics Figure 8 The deformed image 1 is obtained.

[0071] In another alternative implementation, the image deformation model can downsample image 1 and optical flow 1 n times respectively, then fuse the downsampled results of image 1 and optical flow 1, and upsample the fused result n times to obtain deformed image 1.

[0072] Please see Figure 8 This is a network structure diagram of another image deformation model provided in an embodiment of this application. Figure 8 As shown, the image deformation model may include 2n downsampling modules and n upsampling modules. Downsampling module 1 performs a first downsampling process on image 1 to obtain feature map FM1; downsampling module 3 performs a second downsampling process on feature map FM1 to obtain feature map FM2; and so on, using downsampling modules 2n-3 to downsample feature maps FM1 and FM2. n-2 The feature map FM is obtained by performing the (n-1)th downsampling process. n-1 The feature map is FMed using the downsampling module 2n-1. n-1The nth downsampling process is performed to obtain feature map FMn. Additionally, downsampling module 2 can obtain feature map FL1 from optical flow 1; downsampling module 4 then performs a second downsampling process on feature map FL1 to obtain feature map FL2; and so on, using downsampling module 2n-2 to downsample feature map FL... n-2 The feature map FL is obtained by performing the (n-1)th downsampling process. n-1 The feature map FL is processed using the downsampling module 2n. n-1 The nth downsampling process yields the feature map FLn. The feature map FS is obtained by adding the feature map FMn and the feature map FLn. n Feature Map FM n-1 and feature map FL n-1 The feature map FS can be obtained by adding them together. n-1 ...and so on, adding feature map FM2 and feature map FL2 yields feature map FS2, and adding feature map FM1 and feature map FL1 yields feature map FS1. Then, upsampling module 1 can process feature map FS... n The first upsampling process is performed to obtain the feature map FT. n-1 Upsampling module 2 can process feature maps FS n-1 and feature map FT n-1 The fusion result is then subjected to a second upsampling process to obtain the feature map FT. n-2 ...and so on. Upsampling module n-1 can perform upsampling processing on the fusion result of feature map FS2 and feature map FT2 for the (n-1)th time to obtain feature map FT1. Finally, upsampling module n performs upsampling processing on the fusion result of feature map FS1 and feature map FT1 for the nth time to obtain deformed image 1.

[0073] In one optional implementation, the feature map obtained by downsampling image 1 can also be called image feature, the feature map obtained by downsampling optical flow 1 can also be called optical flow feature, and the feature map obtained by fusing image feature and optical flow feature can also be called intermediate image (i.e., image feature 1).

[0074] In other words, the image deformation model can downsample image 1 n times to obtain image feature i, and downsample optical flow 1 n times to obtain optical flow feature i. The size of image feature i and optical flow feature i are the same, the sizes of image feature 1 to image feature n are different, the sizes of optical flow feature 1 to optical flow feature n are different, and i = 1 to n.

[0075] For example, feature map FM1 (i.e., image feature 1) and feature map FL1 (i.e., optical flow feature 1) have the same size. n-1 (i.e., image features n-1) and feature map FL n-1(i.e., optical flow feature n-1) has the same size, and feature maps FM1, FM2 and FM3 are the same. n The sizes are different, feature map FL1, feature map FL2 to feature map FL n They are different sizes.

[0076] In this embodiment, the 2n downsampling modules can be divided into n downsampling groups. Each downsampling group includes two downsampling modules, which are used to downsample image 1 and optical flow 1, respectively. Furthermore, the network structures of the two downsampling modules in a group are identical to obtain image features and optical flow features of the same size. For example, Figure 8 In this model, downsampling module 1 and downsampling module 2 are grouped together, meaning they have the same network structure, and consequently, feature map FM1 and feature map FL1 have the same size. Similarly, downsampling module 2i-1 and downsampling module 2i are grouped together, with the same network structure and feature map FM1 having the same size. i and feature map FL i The dimensions are the same, i = 1 to n.

[0077] Furthermore, the n downsampling modules have different network structures to extract features of image 1 and optical flow 1 at different resolutions. For example, downsampling module 1 and downsampling module 3 belong to different groups and have different network structures. Similarly, downsampling module 4 and downsampling module 2n belong to different groups and have different network structures. Optionally, the n downsampling modules can be composed of multiple residual networks with different network structures.

[0078] Understandably, Figure 8 The image deformation model shown can extract features at different scales to reflect the details of the training image and training optical flow in different receptive fields; therefore, the deformed image obtained by fusing features at different scales also contains the local details and overall style of the training image, which is more in line with the real situation.

[0079] It should be noted that the structure of the image deformation model described above is for illustrative purposes only. Other structures of the image deformation model may also exist, which will not be elaborated here.

[0080] After obtaining the interpolated image, the terminal can display the interpolated image between image 1 and image 2. In this way, when displaying video, the terminal can insert interpolated images between two consecutive frames of the video, which can increase the video frame rate, reduce motion blur, improve video quality, and provide users with a smoother visual experience.

[0081] The following is combined Figure 9 The method for training the above image deformation model is described in detail. Please refer to [link / reference]. Figure 9This is a flowchart illustrating a method for training an image deformation model, which can be applied to... Figure 3 The AI ​​system shown can be trained by a terminal or a server. Here, the method of training the image deformation model provided in this embodiment is described using server 120 as an example.

[0082] like Figure 9 As shown, this embodiment provides a method for training an image deformation model, including steps S901 to S904.

[0083] S901, the server iteratively trains the student-teacher network based on training samples.

[0084] In this embodiment, the training samples include training images, training optical flow, and target images. The target image is the deformed image expected to be obtained by deforming the training image based on the training optical flow.

[0085] For example, the training image can be image 1 as described above, the training optical flow can be optical flow 1 as described above, and the target image can be the deformed image 1 as described above. As another example, the training image can be image 2 as described above, the training optical flow can be optical flow 2 as described above, and the target image can be the deformed image 2 as described above.

[0086] Please continue reading. Figure 9 The student-teacher network consists of a deformation operation model and an initial image deformation model. The deformation operation model is the teacher network, and the initial image deformation model is the student network.

[0087] In this embodiment, the deformation operation model refers to a model that deforms an image through image deformation operations. Image deformation operations, for example, involve sampling and interpolating the input image using Grid Sample. The initial image deformation model consists of convolutional layers and is a model that deforms the image through convolution operations. It should be noted that this embodiment does not limit the structure of the initial image deformation model, as long as the size of the image output by the initial image deformation model is the same as the size of the image output by the deformation operation model. For example, the network structure of the initial image deformation model can be as follows: Figure 7 As shown.

[0088] The above deformation operation model is a teacher network, and the initial image deformation model is a student network. The initial image deformation model can learn the image distortion and deformation capabilities of the deformation operation model through knowledge distillation.

[0089] During an iterative training process, the server can input the training image and training optical flow into the deformation operation model to obtain the first deformed image, and input the training image and training optical flow into the initial image deformation model to obtain the second deformed image.

[0090] Both the first and second deformed images refer to deformed images obtained from the training image under the influence of training optical flow. The difference between the first and second deformed images lies in their generation methods. The first deformed image is obtained by performing calculations based on the training image and training optical flow using Grid Sample, while the second deformed image is obtained by performing a convolution operation on the training image and training optical flow. Furthermore, the first and second deformed images have the same size. This same image size includes, but is not limited to, having the same length, width, and number of channels.

[0091] It should be noted that the resolution of the training samples used in each iteration of training is different. During the iterative training of the student-teacher network based on the training samples, the server can train the student-teacher network based on the training samples in order of increasing resolution. This helps the student-teacher network to learn features at more scales and adapt to inputs at different scales.

[0092] S902, the server determines whether the student-teacher network has converged.

[0093] If the student-teacher network does not converge, the server executes S903; if the student-teacher network converges, the server executes S904.

[0094] In the embodiments of this application, student-teacher network convergence means that both the initial image deformation model (i.e., student network) and the deformation operation model (i.e., teacher network) converge.

[0095] In one optional implementation, the server pre-sets a desired number of iterations. When the actual number of iterations reaches the desired number, the server determines that the student-teacher network has converged. Specifically, each iteration is performed by inputting training samples into the student-teacher network, and the actual iteration count is incremented by 1.

[0096] In another alternative implementation, the server pre-sets a desired number of iterations 1 and a desired number of iterations 2, where the desired number of iterations 2 is greater than the desired number of iterations 1. The server can determine that the deformation operation model has converged when the actual number of iterations reaches the desired number of iterations 1, and determine that the initial image deformation model has converged when the actual number of iterations reaches the desired number of iterations 2.

[0097] In another alternative implementation, the server can determine whether the student-teacher network has converged using a target loss function. For example, the student-teacher network is determined to be converged when the value of the target loss function is less than a threshold, and not converged when the value of the target loss function is greater than or equal to the threshold. The details of the target loss function are explained below and will not be repeated here.

[0098] S903, the server updates the model parameters of the student-teacher network based on the first loss function and the second loss function.

[0099] In this embodiment, the server can perform phased training on the deformation operation model and the initial image deformation model in the student-teacher network. Optionally, the server can first fix the model parameters of the initial image deformation model and update the model parameters of the deformation operation model. After determining that the deformation operation model has converged, the server then fixes the model parameters of the deformation operation model and updates the model parameters of the initial image deformation model.

[0100] In one alternative implementation, the server can, if the actual number of iterations is less than or equal to the expected number of iterations 1, keep the model parameters of the initial image deformation model unchanged and update the model parameters of the deformation operation model; if the actual number of iterations is greater than the expected number of iterations 1 but less than or equal to the expected number of iterations 2, keep the model parameters of the deformation operation model unchanged and update the model parameters of the initial image deformation model.

[0101] In this embodiment of the application, when the server updates the model parameters of the student-teacher network based on the first loss function and the second loss function, it can first determine the first loss function based on the second deformed image and the first deformed image, determine the second loss function based on the second deformed image and the target image, then perform a weighted summation of the first loss function and the second loss function to obtain the target loss function, and then update the model parameters of the student-teacher network based on the target loss function.

[0102] The first loss function is the direct supervision loss, which reflects the difference between the second deformed image and the first deformed image; specifically, it can be the Euclidean distance between the two images. The second loss function is the truth distillation loss, which reflects the difference between the second deformed image and the target image; specifically, it can be the Euclidean distance between the two images.

[0103] In this embodiment of the application, the target loss function, the first loss function, and the second loss function satisfy formula (1):

[0104] L total =λ1L direct +λ2L gt (1)

[0105] Among them, L total Let L be the target loss function. direct Let L be the first loss function. gt Let λ1 be the first preset weight and λ2 be the second preset weight.

[0106] In one optional implementation, the first weight and the second weight are variable. Optionally, during the period when the server updates the model parameters of the deformation operation model while keeping the model parameters of the initial image deformation model unchanged, the value of the second weight can be a first value; during the period when the server updates the model parameters of the initial image deformation model while keeping the model parameters of the deformation operation model unchanged, the value of the second weight can be a second value, and the first value is greater than the second value.

[0107] For example, when the actual number of iterations is less than or equal to the expected number of iterations 1, the value of the second weight can be the first value; when the actual number of iterations is greater than the expected number of iterations 1 but less than or equal to the expected number of iterations 2, the value of the second weight can be the second value.

[0108] This approach allows for a greater proportion of the ground truth distillation loss (i.e., the second loss function) in the target loss function during the early stages of training, providing stronger supervision signals. Conversely, reducing the proportion of the second loss function in the target loss function during the later stages increases the proportion of the first loss function, which is more conducive to the student network (i.e., the initial image deformation model) developing its own feature representations. This progressive training method enables the student network to learn and generalize better and also reduces the risk of model overfitting to some extent.

[0109] In an alternative implementation, the Euclidean distance can also be other parameters that can characterize the differences between two images, such as the Van der Waals distance.

[0110] S904, the server obtains the image deformation model.

[0111] The image deformation model is the initial image deformation model trained to convergence.

[0112] As can be seen, this application combines truth distillation loss and direct supervision loss to adjust the model parameters. On the one hand, this allows the student network to learn the image deformation capabilities of the teacher network; on the other hand, it makes the output of the student network closer to the target, resulting in a final image deformation model with accurate image deformation capabilities. Furthermore, since the initial image deformation model is composed of convolutional layers, it has high computational efficiency when running on an NPU. In other words, the image deformation model trained using the above method achieves a balance between high performance and high-quality deformation effects.

[0113] Furthermore, the image deformation model provided in this application does not require additional storage space to cache intermediate variables (such as the feature maps mentioned above) when performing deformation processing on images, thus saving memory resources.

[0114] It should be noted that the method described above for training image deformation models can also be applied to the training process of other image processing models. Other image processing models include image classification models, super-resolution models, and style transfer models, differing only in their network structures and the need to adjust the training samples according to the specific model being trained. For example, when training an image classification model, training samples can include training images and labels; when training a super-resolution model, training samples can include low-resolution training images and high-resolution target images; and style transfer models can include two training images with different styles, etc. No specific limitations are imposed here.

[0115] This application also provides a chip system, such as... Figure 10 As shown, the chip system 1000 includes at least one processor 1001 and at least one interface circuit 1002. The processor 1001 and the interface circuit 1002 are interconnected via lines. For example, the interface circuit 1002 can be used to receive signals from other devices (e.g., the memory of an electronic device). As another example, the interface circuit 1002 can be used to send signals to other devices (e.g., the processor 1001). Exemplarily, the interface circuit 1002 can read instructions stored in memory and send those instructions to the processor 1001. When the instructions are executed by the processor 1001, the electronic device or server can perform the steps in the above embodiments. Of course, the chip system may also include other discrete devices, and this application embodiment does not specifically limit this.

[0116] This application also provides an electronic device; please refer to [link / reference]. Figure 11 This is a schematic diagram of the structure of the electronic device provided in an embodiment of this application. Figure 11 As shown, the electronic device 1100 may include a processor 1110 and a memory 1120.

[0117] The processor 1110 can be used to read and execute computer-readable instructions. Specifically, the processor 1110 mainly includes a controller, an arithmetic logic unit (ALU), and registers. The controller is primarily responsible for instruction decoding and issuing control data for the operations corresponding to the instructions. The ALU is primarily responsible for storing register operands and intermediate operation results temporarily stored during instruction execution. Specifically, the hardware architecture of the processor 1110 can be an application-specific integrated circuit (ASIC) architecture, a microprocessor without interlocked pipelined stages (MIPS) architecture, an advanced reduced instruction set machine (RISC) machine (ARM) architecture, or a network processor (NP) architecture, etc., without specific limitations.

[0118] Memory 1120 is coupled to processor 1110 and is used to store various software programs (such as the compiler described above) and / or multiple sets of instructions. In specific implementations, memory 1120 may include high-speed random access memory or non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 1120 may store an operating system, such as uCOS, VxWorks, RTLinux, or other embedded operating systems.

[0119] It is understood that the structure illustrated in the embodiments of this application does not constitute a specific limitation on the electronic device 1100. In other embodiments, the electronic device 1100 may include more than Figure 11 It may contain more or fewer components, or combine some components, or separate some components, or arrange the components differently. The components shown in the diagram may be implemented in hardware, software, or a combination of software and hardware.

[0120] This embodiment also provides a computer-readable storage medium storing computer instructions that, when executed on an electronic device, cause the electronic device to perform the various functions or steps described in the above method embodiments.

[0121] This embodiment also provides a computer program product that, when run on a computer, causes the computer to perform the various functions or steps in the above method embodiments.

[0122] In addition, embodiments of this application also provide an apparatus, which may specifically be a chip, component or module. The apparatus may include a connected processor and a memory. The memory is used to store computer execution instructions. When the apparatus is running, the processor can execute the computer execution instructions stored in the memory to cause the chip to perform the various functions or steps performed by the mobile phone in the above method embodiments.

[0123] In this embodiment, the electronic device, communication system, computer-readable storage medium, computer program product or chip are all used to execute the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0124] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0125] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0126] The unit described as a separate component may or may not be physically separate. The component shown as a unit can be one physical unit or multiple physical units, that is, it can be located in one place or distributed in multiple different places. Some or all of the units can be selected to achieve the purpose of the solution in this embodiment according to actual needs.

[0127] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0128] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially or in other words, the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0129] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this application without departing from the spirit and scope of the technical solutions of this application.

Claims

1. A method of training an image warping model, the method comprising: The method includes: The student-teacher network is iteratively trained based on training samples; wherein the student-teacher network includes a teacher network and a student network, the teacher network is a model that deforms images through image deformation operations, and the student network is a model that deforms images through convolution operations, the first deformed image and the second deformed image are of the same size, the first deformed image is the image output by the teacher network, and the second deformed image is the image output by the student network; If the student-teacher network does not meet the convergence condition, update the model parameters of the student-teacher network after the most recent iteration. When the student-teacher network meets the convergence condition, the iterative training stops, and the student network after the last iteration is used as the image deformation model.

2. The method of claim 1, wherein, The training samples include training images, training optical flow, and target images; the method further includes: During any iteration of training, the training image and the training optical flow are input into the student-teacher network after the most recent iteration to obtain the first deformed image and the second deformed image; The updated model parameters of the student-teacher network after the most recent iteration include: The model parameters of the student-teacher network after the most recent iteration are updated according to the target loss function, which is associated with a first difference and a second difference. The first difference is the difference between the first deformed image and the second deformed image, and the second difference is the difference between the second deformed image and the target image.

3. The method of claim 2, wherein, The method further includes: A first loss function is determined based on the first deformed image and the second deformed image; A second loss function is determined based on the second deformed image and the target image; The target loss function is obtained by weighted summation of the first loss function and the second loss function.

4. The method according to claim 2 or 3, characterized in that, The step of updating the model parameters of the student-teacher network after the most recent iteration based on the target loss function includes: If the actual number of iterations is less than or equal to the first expected number of iterations, the model parameters of the teacher network after the most recent iteration are updated according to the target loss function, where the actual number of iterations is the number of times the student-teacher network is trained iteratively. If the actual number of iterations is greater than the first expected number of iterations and less than or equal to the second expected number of iterations, the model parameters of the student network after the most recent iteration are updated according to the target loss function, wherein the second expected number of iterations is greater than the first expected number of iterations.

5. The method of claim 4, wherein, When the actual number of iterations is less than or equal to the first expected number of iterations, the weight corresponding to the second loss function is the first weight. When the actual number of iterations is greater than the first expected number of iterations but less than the second expected number of iterations, the weight corresponding to the second loss function is the second weight, and the first weight is less than the second weight.

6. The method of claim 4, wherein, The student-teacher network satisfies the convergence condition when the actual number of iterations equals the second expected number of iterations.

7. The method according to any one of claims 1 to 6, characterized in that, The student network is used to downsample the training image n times to obtain image feature i, and to downsample the training optical flow n times to obtain optical flow feature i, i = 1 to n. The image feature i and the optical flow feature i have the same size, the image feature 1 to the image feature n have different sizes, and the optical flow feature 1 to the optical flow feature n have different sizes. The student network is also used to upsample the intermediate image n n times to obtain a deformed image, wherein the intermediate image n is obtained based on image features n and optical flow features n.

8. The method of claim 7, wherein, The student network includes n downsampling groups and n upsampling modules. Each downsampling group includes two downsampling modules, and the two downsampling modules in a downsampling group are used to downsample the training image and the training optical flow, respectively. The n upsampling modules are used to upsample the intermediate image n n times to obtain the deformed image.

9. The method of claim 8, wherein, The network structures of the n downsampling groups are different, while the network structures of the two downsampling modules in a downsampling group are the same.

10. A method for generating an interframe, characterized by, The method includes: Obtain a first image and a second image from the video to be interpolated, wherein the first image and the second image are two consecutive frames; The first image and the second image are input into a frame interpolation network to obtain an interpolated image. The frame interpolation network includes an image deformation model, which is used to deform the first image and the second image to obtain a deformed image. The interpolated image is obtained based on the deformed image. The image deformation model is obtained by iteratively training a student-teacher network, which includes a teacher network and a student network. The teacher network is a model that deforms the image through image deformation operations, and the student network is a model that deforms the image through convolution operations. The first deformed image and the second deformed image have the same size. The first deformed image is the image output by the teacher network, and the second deformed image is the image output by the student network. The image deformation model is a student network that satisfies the convergence condition.

11. The method of claim 10, wherein, The process of training the student-teacher network includes: The student-teacher network is iteratively trained based on the training samples; If the student-teacher network does not meet the convergence condition, update the model parameters of the student-teacher network after the most recent iteration. When the student-teacher network meets the convergence condition, the iterative training stops, and the student network after the last iteration is used as the image deformation model.

12. The method of claim 11, wherein, The training samples include training images, training optical flow, and target images. The process of training the student-teacher network further includes: During any iteration of training, the training image and the training optical flow are input into the student-teacher network after the most recent iteration to obtain the first deformed image and the second deformed image; The updated model parameters of the student-teacher network after the most recent iteration include: The model parameters of the student-teacher network after the most recent iteration are updated according to the target loss function, which is associated with a first difference and a second difference. The first difference is the difference between the first deformed image and the second deformed image, and the second difference is the difference between the second deformed image and the target image.

13. The method according to any one of claims 10-12, characterized in that, The frame interpolation network further includes an optical flow estimation module and an image fusion module. The step of inputting the first image and the second image into the frame interpolation network to obtain the interpolated image includes: The first image and the second image are input into the optical flow estimation module to obtain the first optical flow and the second optical flow. The first optical flow is the optical flow from the first image to the interpolated image, and the second optical flow is the optical flow from the second image to the interpolated image. The first image is deformed based on the first optical flow using the image deformation model to obtain a third deformed image. The second image is deformed based on the second optical flow using the image deformation model to obtain a fourth deformed image; The interpolated image is obtained by fusing the third deformed image and the fourth deformed image through the image fusion module.

14. The method of any of claims 10-12, wherein, The frame interpolation network includes an optical flow estimation module, a pooling module, multiple image deformation modules, and a feature fusion module; The step of inputting the first image and the second image into the frame interpolation network to obtain the interpolated image includes: The first image and the second image are input into the optical flow estimation module to obtain the first optical flow and the second optical flow. The first optical flow is the optical flow from the first image to the interpolated image, and the second optical flow is the optical flow from the second image to the interpolated image. The pooling module performs pooling processing on the first image, the second image, the first optical flow, and the second optical flow to obtain multiple sets of feature maps. Each set of feature maps includes the first image, the second image, the first optical flow, and the second optical flow, all of the same size. The multiple sets of feature maps are respectively input into the multiple image deformation modules to obtain multiple fifth deformation images, and the multiple fifth deformation images have different sizes; The interpolated image is obtained by fusing the multiple fifth deformed images through the feature fusion module.

15. The method according to claim 14, characterized in that, The image deformation module includes the image deformation model and the image fusion module. The multiple sets of feature maps are input into the multiple image deformation modules respectively to obtain multiple fifth deformed images, including: For each image deformation module, the first image is deformed based on the first optical flow using the image deformation model to obtain a third deformed image; The second image is deformed based on the second optical flow using the image deformation model to obtain a fourth deformed image; The fifth deformed image is obtained by fusing the third deformed image and the fourth deformed image through the image fusion module.

16. An electronic device, comprising: The electronic device includes: a memory and one or more processors; the memory and the processors are coupled; the memory is used to store computer program code, the computer program code including computer instructions, which, when executed by the electronic device, cause the electronic device to perform the method as described in any one of claims 1-15.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed in an electronic device, cause the electronic device to perform the method as described in any one of claims 1-15.

Images