Video processing method, device, readable storage medium and electronic device

By determining the target image and performing unsequential stitching processing, the problem of stuttering and inability to adapt to dynamic interaction during the video stitching process is solved, and smooth video playback and barrier-free communication are achieved in scenarios such as online teaching.

CN112200739BActive Publication Date: 2024-11-15BEIJING DA MI TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202011062278.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-09-30
Publication Date
2024-11-15
Estimated Expiration
2040-09-30

AI Technical Summary

Technical Problem

The existing technology has a sense of lag and is unable to adapt to dynamic interactions during the video splicing process, resulting in poor user experience, especially in application scenarios such as online teaching, which cannot achieve barrier-free communication with students.

Method used

By acquiring at least two image sequences, a transition image set is determined, and the target image is determined based on the set, and an unsequential splicing of the image sequence is performed to reduce lags during alternate video playback, and smooth splicing is achieved in dynamic interactive scenes.

Benefits of technology

It realizes unsequential splicing between multiple videos, reduces lag during alternate video playback, improves user experience, and can achieve barrier-free communication between teachers and students in dynamic interaction scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112200739B_ABST
    Figure CN112200739B_ABST
Patent Text Reader

Abstract

The embodiments of the present invention disclose a method, device, readable storage medium and electronic device for video processing. The embodiments of the present invention obtain at least two image sequences; determine a transition image set according to the at least two image sequences, wherein the transition image set includes multiple transition images, and the transition image is an image at a predetermined position in the beginning or the end of the image sequence; determine a target image according to the transition image set; and perform splicing processing on the at least two image sequences based on the target image to obtain a target image sequence. Through the above method, any two image sequences are spliced ​​through the target image without determining the sequence of the image segments, thereby realizing non-sequential splicing between multiple videos and reducing the freeze during the alternating playback of videos.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of video processing, and in particular to a video processing method, device, readable storage medium and electronic device. Background Art

[0002] With the development of Internet applications, online teaching, intelligent customer service, etc. are increasingly used in people's daily lives. In the face of different application scenarios, sometimes it is necessary to pre-record multiple videos and then splice them to get a long video. For example: in the online teaching recorded class scenario, it is necessary to splice multiple segmented recorded class clips into a complete class video.

[0003] In the process of video splicing in the prior art, a sense of lag may be caused due to insufficient smoothness. For example, direct splicing may cause problems such as teleportation and flashing between the first video and the second video, resulting in a poor user experience. In addition, the splicing order of videos in the prior art is fixed and cannot adapt to dynamic interactive situations. For example, there may be dynamic interactive links in online teaching courses, and students may ask multiple questions in no order. How to splice multiple videos pre-recorded by teachers so as to achieve barrier-free communication with students during the dynamic interaction process is a problem that needs to be solved at present. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a method, device, readable storage medium and electronic device for video processing, which realize non-sequential splicing between multiple videos and reduce the jamming during the alternating playback of videos.

[0005] In a first aspect, an embodiment of the present invention provides a method for video processing, the method comprising:

[0006] acquiring at least two image sequences;

[0007] Determine a transition image set according to the at least two image sequences, wherein the transition image set includes a plurality of transition images, and the transition images are images located at predetermined positions at the beginning or the end of the image sequence;

[0008] Determine a target image according to the transition image set;

[0009] The at least two image sequences are spliced ​​based on the target image to obtain a target image sequence.

[0010] In one embodiment, determining the target image according to the transition image set specifically includes:

[0011] In the transition image set, an image with the smallest difference from the remaining images is determined as the target image.

[0012] In one embodiment, the predetermined position is a starting position or an ending position of the image sequence.

[0013] In one embodiment, the at least two image sequences include a first image sequence and a second image sequence, and the splicing order is that the first image sequence is in front and the second image sequence is in the back;

[0014] The performing stitching processing on the at least two image sequences based on the target image to obtain the target image sequence specifically includes:

[0015] replacing the image corresponding to the end position of the first image sequence and the image corresponding to the start position of the second image sequence with the target image;

[0016] The replaced first image sequence and the replaced second image sequence are spliced ​​into a target image sequence.

[0017] In one embodiment, the method further comprises:

[0018] Smoothing the last part of the image sequence of the set duration in the replaced first image sequence;

[0019] The image sequence of the beginning part of the set time length in the replaced second image sequence is smoothed.

[0020] In one embodiment, the smoothing of the last image sequence of the set duration in the replaced first image sequence specifically includes:

[0021] Determine a timestamp of a first image, where the first image is an image in the first image sequence after replacement whose timestamp is earlier than that of the target image;

[0022] Determining a similarity between the first image and the target image;

[0023] In response to the similarity satisfying a predetermined similarity condition, based on a pre-trained first model, determining a corresponding forward optical flow field vector and a reverse optical flow field vector according to the first image and the target image, the forward optical flow field vector being used to characterize the forward optical flow of the first image and the target image, and the reverse optical flow field vector being used to characterize the reverse optical flow of the first image and the target image;

[0024] Based on a pre-trained second model, at least one intermediate image is determined according to the forward optical flow field vector and the reverse optical flow field vector, where the intermediate image is an image between the first image and the target image.

[0025] In one embodiment, determining the similarity between the first image and the target image specifically includes:

[0026] The optical flow similarity between the first image and the target image is calculated as the similarity.

[0027] In one embodiment, the determining at least one intermediate image based on the pre-trained second model according to the forward optical flow field vector and the reverse optical flow field vector specifically includes:

[0028] Determine a first approximate optical flow field vector and a second approximate optical flow field vector according to the forward optical flow and the reverse optical flow, wherein the first approximate optical flow field vector is used to characterize the forward approximate optical flow at an intermediate moment, and the second approximate optical flow field vector is used to characterize the reverse approximate optical flow at the intermediate moment;

[0029] determining a first interpolation function according to the first image and the first approximate optical flow field vector;

[0030] Determining a second interpolation function according to the target image and the second approximate optical flow field vector;

[0031] Taking the first image, the target image, the first approximate optical flow field vector, the second approximate optical flow field vector, the first interpolation function and the second interpolation function as inputs of the second model, determining a forward visible graph, a reverse visible graph, a first increment and a second increment at the intermediate moment, wherein the first increment is used to represent the increment of the first approximate optical flow field vector at the intermediate moment, and the second increment is used to represent the increment of the second approximate optical flow field vector at the intermediate moment;

[0032] The intermediate image corresponding to the intermediate moment is determined according to the first image, the target image, the forward visible image, the reverse visible image, the first increment, and the second increment.

[0033] In one embodiment, the first model and the second model are trained in the following manner:

[0034] Acquire a plurality of image groups, wherein the image groups include a first image sample, a target image sample, and at least one intermediate image sample, wherein the first image sample, the target image sample, and the intermediate image sample are different images in the same image sequence;

[0035] Taking the first image samples and the target image samples in each of the image groups as input and the corresponding intermediate image samples as training targets, the first model and the second model are trained simultaneously until the loss functions of the first model and the second model converge.

[0036] In one embodiment, the loss function is used to characterize the reconstruction loss, semantic loss, distortion loss and smoothness loss of the first model and the second model.

[0037] In one embodiment, the method further comprises:

[0038] A target video is determined according to the target image sequence and the corresponding audio sequence.

[0039] In a second aspect, an embodiment of the present invention provides a video processing device, the device comprising:

[0040] An acquisition unit, used for acquiring at least two image sequences;

[0041] a determining unit, configured to determine a transition image set according to the at least two image sequences, wherein the transition image set includes a plurality of transition images, and the transition images are images located at predetermined positions at the beginning or the end of the image sequence;

[0042] The determining unit is further used to determine the target image according to the transition image set;

[0043] A processing unit is used to perform splicing processing on the at least two image sequences based on the target image to obtain a target image sequence.

[0044] In one embodiment, the determining unit is specifically configured to:

[0045] In the transition image set, an image with the smallest difference from the remaining images is determined as the target image.

[0046] In one embodiment, the predetermined position is a starting position or an ending position of the image sequence.

[0047] In one embodiment, the at least two image sequences include a first image sequence and a second image sequence, and the splicing order is that the first image sequence is in front and the second image sequence is in the back;

[0048] The processing unit is specifically used for:

[0049] replacing the image corresponding to the end position of the first image sequence and the image corresponding to the start position of the second image sequence with the target image;

[0050] The replaced first image sequence and the replaced second image sequence are spliced ​​into a target image sequence.

[0051] In one embodiment, the processing unit is further used to perform smoothing processing on the last part of the image sequence of the set duration in the replaced first image sequence;

[0052] The image sequence of the beginning part of the set time length in the replaced second image sequence is smoothed.

[0053] In one embodiment, the processing unit is specifically configured to:

[0054] Determine a timestamp of a first image, where the first image is an image in the first image sequence after replacement whose timestamp is earlier than that of the target image;

[0055] Determining a similarity between the first image and the target image;

[0056] In response to the similarity satisfying a predetermined similarity condition, based on a pre-trained first model, determining a corresponding forward optical flow field vector and a reverse optical flow field vector according to the first image and the target image, the forward optical flow field vector being used to characterize the forward optical flow of the first image and the target image, and the reverse optical flow field vector being used to characterize the reverse optical flow of the first image and the target image;

[0057] Based on a pre-trained second model, at least one intermediate image is determined according to the forward optical flow field vector and the reverse optical flow field vector, where the intermediate image is an image between the first image and the target image.

[0058] In one embodiment, the processing unit is specifically configured to:

[0059] The optical flow similarity between the first image and the target image is calculated as the similarity.

[0060] In one embodiment, the determining unit is further configured to:

[0061] A target video is determined according to the target image sequence and the corresponding audio sequence.

[0062] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement a method as described in the first aspect or any possible embodiment of the first aspect.

[0063] In a fourth aspect, an embodiment of the present invention provides an electronic device, comprising a memory and a processor, the memory being used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement a method as described in the first aspect or any possible embodiment of the first aspect.

[0064] The embodiment of the present invention obtains at least two image sequences; determines a transition image set according to the at least two image sequences, wherein the transition image set includes multiple transition images, and the transition image is an image at a predetermined position in the beginning or the end of the image sequence; determines a target image according to the transition image set; and splices the at least two image sequences based on the target image to obtain a target image sequence. Through the above method, any two image sequences are spliced ​​through the target image without determining the sequence of the image sequences, thereby realizing non-sequential splicing between multiple videos and reducing the freeze during the alternating playback of videos. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0066] Figure 1 is a flow chart of a video processing method according to an embodiment of the present invention;

[0067] Figure 2 is a schematic diagram of a target image sequence structure according to an embodiment of the present invention;

[0068] Figure 3 is a flow chart of a video processing method according to an embodiment of the present invention;

[0069] Figure 4 is a schematic diagram of a first image sequence structure according to an embodiment of the present invention;

[0070] Figure 5 is a flow chart of a method for determining an intermediate image according to an embodiment of the present invention;

[0071] Figure 6 is a schematic diagram of an optical flow field at an intermediate moment in an embodiment of the present invention;

[0072] Figure 7 is a schematic diagram of image interpolation according to an embodiment of the present invention;

[0073] Figure 8 is a schematic structural diagram of a first image sequence and a second image sequence according to an embodiment of the present invention;

[0074] Fig. 9 is a flow chart of training a first model and a second model according to an embodiment of the present invention;

[0075] Fig.10 is a schematic diagram of a video processing device according to an embodiment of the present invention;

[0076] Fig.11 is a schematic diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0077] The present invention is described below based on the embodiments, but the present invention is not limited to these embodiments. In the detailed description of the present invention below, some specific details are described in detail. It is possible for a person skilled in the art to fully understand the present invention without the description of these details. In order to avoid confusing the essence of the present invention, known methods, processes, procedures, components and circuits are not described in detail.

[0078] In addition, persons of ordinary skill in the art will appreciate that the drawings provided herein are for illustration purposes and are not necessarily drawn to scale.

[0079] Unless the context clearly requires otherwise, the words "include", "comprising" and similar words throughout the application should be interpreted as including rather than exclusive or exhaustive; that is, the meaning is "including but not limited to".

[0080] In the description of the present invention, it should be understood that the terms "first", "second", etc. are used for descriptive purposes only and cannot be understood as indicating or implying relative importance. In addition, in the description of the present invention, unless otherwise specified, the meaning of "plurality" is two or more.

[0081] In the prior art, in the application scenarios of online teaching, some course teachers need to record the course content in advance, for example, by recording videos through image acquisition devices such as cameras, mobile phones with camera functions, tablet computers, etc. Assuming that the teacher has recorded in advance the videos required for dynamic interactive classes, the characteristic of dynamic interactive classes is that the order in which students ask questions is changing. Therefore, a short video needs to be recorded separately for each question, and then, according to the order in which students ask questions, multiple short videos recorded separately are spliced ​​to obtain a complete course video, so as to achieve complete interaction between teachers and students; since the order in which students ask questions is changing, and the splicing order of videos in the prior art is fixed, it cannot adapt to the situation of dynamic interactive classes; and in the process of playing the complete course video, the smoothness between each video segment is low, that is, the overlap between the last frame image of the previous video segment and the first frame image of the adjacent next video segment is not High, resulting in a relatively abrupt image transition during the alternating playback of two adjacent videos, thus presenting a teleport or freeze effect, resulting in a poor viewing experience for users; in the existing splicing technology, in order to solve the above problem, image frames are inserted between short videos to achieve smooth processing, and the image frames to be inserted are usually determined based on the previous video and the next video, so the splicing order is fixed. For example: to splice three videos A, B, and C, it is necessary to determine the order of A, B, and C in advance. Assuming that the order of A, B, and C is determined to be A first, then B, and finally C, the image frames to be inserted between A and B and between B and C are determined according to the order of the three videos to achieve splicing, that is, only fixed-order splicing can be achieved; in summary, how to splice multiple videos pre-recorded by teachers so that barrier-free communication with students can be achieved during dynamic interaction and user experience is a problem that needs to be solved at present.

[0082] In the embodiment of the present invention, any two image sequences are spliced ​​together through the target image without determining the order of the image segments, thereby realizing non-sequential splicing between multiple videos and reducing the jamming during the alternating playback of videos. Since non-sequential splicing is possible, it is possible to solve the problem of barrier-free communication between teachers when the order in which students ask questions changes during online teaching, that is, according to the questions raised by the students, the short video (image sequence) corresponding to the question is determined, and then the short videos corresponding to each question are spliced ​​according to the order in which the questions are asked. From the students' perspective, the teacher's answers to the questions are smooth, which improves the students' user experience.

[0083] In the embodiment of the present invention, Figure 1 FIG. 1 is a flow chart of a video processing method according to an embodiment of the present invention. Figure 1 As shown, the specific steps include:

[0084] Step S100: Acquire at least two image sequences.

[0085] In one or more embodiments, the image sequence may also be referred to as a video clip, a short video, a video, etc., which is not limited in the embodiments of the present invention.

[0086] For example, assume that the image sequence is a teacher's teaching short video, and each video segment is used to answer a question of a student. A student may ask multiple questions, so the teacher needs to record multiple teaching video segments.

[0087] Step S101: determining a transition image set according to the at least two image sequences, wherein the transition image set includes a plurality of transition images, and the transition image is an image located at a predetermined position at the beginning or the end of the image sequence.

[0088] In one or more embodiments, at least one transition image is acquired in each image sequence, wherein the transition image may be an image corresponding to the starting position of the image sequence, may be an image corresponding to the ending position of the image sequence, or may be an image in the beginning part of the image sequence, for example, at least one of the images within 0.5 seconds from the beginning of the image sequence, or may be an image in the last part of the image sequence, for example, at least one of the images within 0.5 seconds from the end of the image sequence.

[0089] For example, assume that there are three image sequences, and an image is obtained at the starting position and the ending position of each image sequence in the three image sequences. The three image sequences are image sequence 1, image sequence 2 and image sequence 3. The image at the starting position of image sequence 1 is image A, and the image at the ending position is image B. The image at the starting position of image sequence 2 is image C, and the image at the ending position is image D. The image at the starting position of image sequence 3 is image E, and the image at the ending position is image F. Image A, image B, image C, image D, image E and image F form a transition image set.

[0090] Step S102: determining a target image according to the transition image set.

[0091] In one or more embodiments, determining the target image according to the transition image set specifically includes: determining, in the transition image set, an image having the smallest difference from the remaining images as the target image.

[0092] In one or more embodiments, it is assumed that there are three transition images in the transition image set, and the contents of the transition images are all images of a person's upper body, wherein the person's head in transition image 1 is tilted to the left, the person's head in transition image 2 is centered, and the person's head in transition image 3 is tilted to the right. Therefore, among the three transition images, transition image 2 is the image with the smallest difference from the other transition images, and transition image 2 is determined as the target image. The above embodiments are merely illustrative illustrations of the minimum difference, which is determined based on actual conditions, and the embodiments of the present invention do not limit it.

[0093] For example, in step S101 , the transition image set is composed of the image A, image B, image C, image D, image E and image F. According to the principle of minimum image difference, image C is determined as the target image.

[0094] Step S103: performing splicing processing on the at least two image sequences based on the target image to obtain a target image sequence.

[0095] In one or more embodiments, when the at least two image sequences are stitched together based on the target image, the order of the image sequences is not limited. If there are two image sequences, namely a first image sequence and a second image sequence, the stitching order may be that the first image sequence comes first and the second image sequence comes second; it may also be that the second image sequence comes first and the first image sequence comes second.

[0096] In one or more embodiments, when the first image sequence is in front and the second image sequence is in the back, the splicing processing of the at least two image sequences based on the target image to obtain the target image sequence specifically includes: replacing the image corresponding to the end position of the first image sequence and the image corresponding to the starting position of the second image sequence with the target image; and splicing the replaced first image sequence and the replaced second image sequence into the target image sequence.

[0097] For example, in the above step S102, image C is determined as the target image. First, the splicing order of image sequence 1, image sequence 2 and image sequence 3 is determined, including: image sequence 1 to image sequence 2 to image sequence 3; image sequence 2 to image sequence 1 to image sequence 3; image sequence 3 to image sequence 2 to image sequence 1; image sequence 3 to image sequence 1 to image sequence 2, etc. Assuming that the splicing order is image sequence 2 to image sequence 1 to image sequence 3, the specific example is as follows: Figure 2As shown, the image D at the end position of image 2 is replaced by image C, the image A at the start position of image sequence 1 and the image B at the end position are both replaced by image C, and the image sequence E at the start position of image sequence 3 is replaced by image C; then the replaced image sequences are spliced ​​to obtain the target image sequence.

[0098] In the embodiment of the present invention, through the above-mentioned method, there is no need to consider the order of the image sequence. The target image can be determined first, and then the image sequence can be arbitrarily stitched according to the target image. This solves the problem in the prior art that it is necessary to first determine the position of the image sequence, and then determine the transition image between the image sequences according to the position of the image sequence, which makes it impossible to achieve arbitrary stitching and requires determining multiple transition images, resulting in complex calculations.

[0099] In the implementation of the present invention, the above scheme realizes arbitrary splicing of images, but because the target image replaces the image at the starting position or the image at the ending position in the image sequence, it will cause freezes or blinks when playing the beginning or the last part of the image sequence. Therefore, it is necessary to smooth the images at the beginning or the last part of the image sequence.

[0100] In one or more embodiments, the first image sequence is in front and the second image sequence is in the back, and after the at least two image sequences are spliced ​​based on the target image to obtain the target image sequence, the method further includes: smoothing the image sequence of the last part of the set duration in the replaced first image sequence; and smoothing the image sequence of the beginning part of the set duration in the replaced second image sequence.

[0101] In one or more embodiments, the last part of the image sequence of the set duration in the replaced first image sequence is smoothed, such as Figure 3 As shown, the specific steps include:

[0102] Step S300: determining a timestamp of a first image, where the first image is an image in the first image sequence after replacement whose timestamp is earlier than that of the target image.

[0103] For example, the timestamp of the first image is 0.5 seconds (S) before the timestamp of the target image in the first image sequence. Figure 4 shown.

[0104] Step S301: Determine the similarity between the first image and the target image.

[0105] In one or more embodiments, the determination of the similarity between the first image and the target image specifically includes: calculating the optical flow similarity between the first image and the target image as the similarity; the optical flow is the projection of a spatial (i.e., three-dimensional) moving object on a pixel observation plane (i.e., a two-dimensional image sequence), which is generated according to the relative speed of the object and the camera, and reflects the movement direction and speed of the image pixels corresponding to the object in a very short period of time. The essence of optical flow is a two-dimensional vector field, each vector represents the displacement of the image pixel in the scene from the previous frame to the next frame, in this embodiment, that is, the displacement from the first image to the target image; specifically, the total variation algorithm (Total Variation L1-regulation, TVL1) based on L1 norm regularization can be selected to calculate the optical flow similarity between the first image and the second image. TVL1 is a cross-platform computer vision and machine learning software library (OpenCV) function based on open source release. It converts the first image and the target image into grayscale images and outputs an optical flow field image. Each pixel value in the optical flow field image is used to characterize the displacement of the corresponding image pixel in the x and y directions during the process of moving from the position in the first image to the position in the target image, that is, the optical flow similarity is used to characterize the displacement of the image pixel.

[0106] Optionally, other optical flow similarity algorithms may be used to determine the optical flow similarity between the first image and the target image, such as the total variation algorithm based on L2 norm regularization (Total Variation L2-regulation, TVL2), the Horn-Schunck method, etc., which are not specifically limited in this embodiment.

[0107] Step S302, in response to the similarity satisfying a predetermined similarity condition, based on a pre-trained first model, determine the corresponding forward optical flow field vector and the reverse optical flow field vector according to the first image and the target image, the forward optical flow field vector is used to characterize the forward optical flow of the first image and the target image, and the reverse optical flow field vector is used to characterize the reverse optical flow of the first image and the target image.

[0108] In one or more embodiments, since the optical flow similarity is used to characterize the displacement of image pixels, when the displacement is too large, the optical flow similarity is low, and the difference between the first image and the target image is large; when the displacement is too small, the optical flow similarity is high, and the difference between the first image and the target image is small. Therefore, it is necessary to insert an intermediate image between the first image and the target image to achieve a smooth transition, and the predetermined similarity condition is that the optical flow similarity is greater than a first threshold and less than a second threshold.

[0109] In one or more embodiments, the first model is used to calculate optical flow, and specifically can be a convolutional neural network (CNN); CNN is a type of feedforward neural network that includes convolution calculations and has a deep structure, and is one of the representative algorithms for deep learning. CNN is constructed by imitating the visual perception mechanism of organisms, and can perform supervised learning and unsupervised learning. The convolution kernel parameter sharing within the hidden layer and the sparsity of the inter-layer connection enable CNN to learn grid-like topology features such as pixels and audio with a smaller amount of computation, with a stable effect and no additional feature engineering requirements for the data. More specifically, the first model can be a Cascade Region-Convolutional Neural Network (Cascade R-CNN).

[0110] Step S303: Based on the pre-trained second model, determine at least one intermediate image according to the forward optical flow field vector and the reverse optical flow field vector, where the intermediate image is an image between the first image and the target image.

[0111] In one embodiment, the pre-trained second model is used to determine at least one intermediate image according to the forward optical flow field vector and the reverse optical flow field vector, such as Figure 5 As shown, the specific steps include:

[0112] S500. Determine a first approximate optical flow field vector and a second approximate optical flow field vector according to the forward optical flow and the reverse optical flow, wherein the first approximate optical flow field vector is used to characterize the forward approximate optical flow at an intermediate moment, and the second approximate optical flow field vector is used to characterize the reverse approximate optical flow at the intermediate moment.

[0113] In one or more embodiments, the intermediate moment is the moment corresponding to the intermediate image between the first image and the target image. For example, the first image is represented by I0, the target image is represented by I1, and the image acquisition period is calculated as 0.1 seconds, and the first image is the image 0.5 seconds before the first image sequence, and the target image is the last image in the first image sequence. For example, there is a total of 0.3 seconds between the first image and the target image, so the intermediate moment can take three values ​​of 0.4, 0.3, and 0.2.

[0114] Since the intermediate image is unknown, the forward optical flow field vector and the reverse optical flow field vector at the intermediate moment are difficult to obtain. Figure 6 is a schematic diagram of the optical flow field at the intermediate moment. Figure 6As shown, each point represents a pixel, pixels in the same column correspond to the same moment, and pixels in the same row correspond to the same position. For pixel 21 at T=t (i.e., the middle moment), the optical flow field vector corresponding to pixel 22 can be approximated by the pixel at the same position at T=0 (i.e., the moment corresponding to the first image), i.e., the forward optical flow corresponding to pixel 22, and the pixel at the same position at T=1 (i.e., the moment corresponding to the target image), i.e., the reverse optical flow corresponding to pixel 23. Specifically, the first approximate optical flow field vector can be determined based on the forward optical flow corresponding to pixel 22. And determine the first approximate optical flow field vector according to the forward optical flow corresponding to pixel 23

[0115] Specifically, the first approximate optical flow field vector It can be determined by the following formula:

[0116]

[0117] Among them, F 0→1 (p) is used to represent the forward optical flow field vector of the pixel at the same position as the pixel at the intermediate time at time T = 0, F 1→0 (p) is used to characterize the reverse optical flow field vector of the pixel at the same position as the pixel at the intermediate time at time T=1.

[0118] Similarly, the second approximate optical flow vector It can be determined by the following formula:

[0119]

[0120] Since the above approximation method works better in a smooth area, but not so well near a motion boundary, because the motion near the motion boundary is not locally smooth and artifacts may be generated, the server may determine a first interpolation function based on the first image and the first approximate optical flow field vector, and determine a second interpolation function based on the target image and the second approximate optical flow field vector.

[0121] S501. Determine a first interpolation function according to the first image and the first approximate optical flow field vector.

[0122] S502: Determine a second interpolation function according to the target image and the second approximate optical flow field vector.

[0123] In one or more embodiments, the first interpolation function and the second interpolation function are both bilinear interpolation functions.

[0124] In one or more embodiments, the server may determine an intermediate image according to the first image, the first approximate optical flow field vector, the target image, and the second approximate optical flow field vector, and then respectively calculate the intermediate image based on the first interpolation function. and the second interpolation function Bilinear interpolation is performed on parts of the intermediate image to obtain the missing elements in the intermediate image.

[0125] In one or more embodiments, Figure 7 4 is a schematic diagram of image interpolation according to an embodiment of the present invention. Figure 7 Taking the first interpolation function as an example, it is easy to understand that the second interpolation function is determined in a similar manner to the first interpolation function. At time T=0, pixel 71, pixel 72, pixel 73 and pixel 74 are four adjacent pixels, and their position relationship is as follows: Figure 7 As shown in the upper left side; at time T=t, pixel 71, pixel 72, pixel 73 and pixel 74 have moved and are not adjacent to each other, and the position relationship is as follows Figure 7 As shown in the upper right side. When calculating the missing pixels between pixel 71, pixel 72, pixel 73 and pixel 74 at time T=t, such as pixel 75, the server can use the direction from pixel 71 to pixel 72 as the positive direction of the X axis and the direction from pixel 71 to pixel 73 as the positive direction of the Y axis, and then set the coordinates of pixel 71, pixel 72, pixel 73 and pixel 74 at time T=t to (0,0), (1,0), (0,1) and (1,1), respectively, and then determine the pixel value corresponding to pixel 75 based on the distance between pixel 75 and pixel 71, pixel 72, pixel 73 and pixel 74, that is, x, 1-x, y and 1-y (where x and y are both real numbers greater than 0 and less than 1). Specifically, the first interpolation function It can be expressed by the following formula:

[0126] f(x,y)≈f(0,0)*(1-x)(1-y)+f(1,0)*x(1-y)+f(0,1)*(1-x)y+f(1,1)xy;

[0127] Among them, (x, y) is the coordinate value of pixel 75, f(x, y) is the pixel value of pixel 75, f(0,0) is the pixel value of pixel 71, f(1,0) is the pixel value of pixel 72, f(0,1) is the pixel value of pixel 73, and f(1,1) is the pixel value of pixel 74.

[0128] S503, taking the first image, the target image, the first approximate optical flow field vector, the second approximate optical flow field vector, the first interpolation function and the second interpolation function as inputs of the second model, determining a forward visible graph, a reverse visible graph, a first increment and a second increment at the intermediate moment, wherein the first increment is used to characterize the increment of the first approximate optical flow field vector at the intermediate moment, and the second increment is used to characterize the increment of the second approximate optical flow field vector at the intermediate moment.

[0129] In one or more embodiments, after determining the first approximate optical flow field vector, the second approximate optical flow field vector, the first interpolation function, and the second interpolation function, the server can use the first image, the first approximate optical flow field vector, the first interpolation function, the second image, the second approximate optical flow field vector, and the second interpolation function as inputs, and obtain the forward visible graph, the reverse visible graph, the first increment, and the second increment at the target time based on the second model, so as to perform residual correction on the image to be corrected based on the second model. The forward visible graph and the reverse visible graph are used to characterize the visibility of the target image, the first increment is used to characterize the increment of the first approximate optical flow field vector at the target time, and the second increment is used to characterize the increment of the second approximate optical flow field vector at the target time.

[0130] In one or more embodiments, the second model is used to correct the optical flow (i.e., the forward optical flow field vector and the reverse optical flow field vector), and can also be a CNN, specifically a Cascade R-CNN. Similar to the first model, the main architecture of the second model can also be U-Net.

[0131] S504: Determine the intermediate image corresponding to the intermediate moment according to the first image, the target image, the forward visible image, the reverse visible image, the first increment, and the second increment.

[0132] In one or more embodiments, it is assumed that three intermediate images are required between the first image and the target image, the timestamp of the first image is 0.5s in the countdown, the timestamp of the target image is 0.1s in the countdown, and the corresponding intermediate images are inserted at the intermediate moments 0.4s, 0.3s and 0.2s respectively, so that the first image smoothly transitions to the target image.

[0133] Similarly, the smoothing of the image sequence of the starting part of the set duration in the replaced second image sequence specifically includes: determining the timestamp of the second image, the second image being an image in the replaced second image sequence whose timestamp is later than that of the target image; determining the similarity between the second image and the target image; in response to the similarity satisfying a predetermined similarity condition, determining corresponding forward optical flow field vectors and reverse optical flow field vectors according to the second image and the target image based on a pre-trained first model, the forward optical flow field vectors being used to characterize the forward optical flow of the second image and the target image, and the reverse optical flow field vectors being used to characterize the reverse optical flow of the second image and the target image; determining at least one intermediate image based on the forward optical flow field vectors and the reverse optical flow field vectors based on the pre-trained second model, the intermediate image being an image between the second image and the target image.

[0134] To sum up, after the replaced first image sequence and the replaced second image sequence are spliced, the image sequence of the last part of the set length in the replaced first image sequence is smoothed, that is, the images of the last part of the first image sequence are replaced; and the image sequence of the beginning part of the set length in the replaced second image sequence is smoothed, that is, the images of the beginning part of the second image sequence are replaced; and finally the target image sequence is obtained.

[0135] For example, Figure 8 As shown, the last 0.5s image of the first image sequence is replaced by a combination of a smoothed intermediate image (also called a smoothed frame) and a target image (also called a standard sample frame); the first 0.5s image of the second image sequence is replaced by a combination of a smoothed intermediate image (also called a smoothed frame) and a target image (also called a standard sample frame).

[0136] In one or more embodiments, after the target image sequence is spliced, the target video is determined based on the target image sequence and the corresponding audio sequence; specifically, the audio sequence is sorted according to the order of each image sequence in the target image sequence, and the sorted audio sequences are spliced ​​into a target audio sequence; then the target image sequence and the target audio sequence are synthesized to output a video file.

[0137] In one or more embodiments, the training method of the first model and the second model is an unsupervised training method, and the server can train the first model and the second model simultaneously. Fig. 9 is a flow chart for training the first model and the second model. Fig. 9 As shown, the first model and the second model can be trained in the following way:

[0138] Step S900: Acquire multiple image groups, wherein the image groups include a first image sample, a target image sample, and at least one intermediate image sample, and the first image sample, the target image sample, and the intermediate image sample are different images in the same image sequence.

[0139] In one or more embodiments, the intermediate image sample is an intermediate image sample whose time stamp is between the first image sample and the target image sample.

[0140] For example, the image sequence as a sample includes image P1, image P2, ..., image P(m-1) and image Pm (where m is a predetermined integer greater than 1), and image P1, image P2, ..., image P(m-1) and image Pm are arranged in time axis order. The server can determine image P1 as the first image sample, image Pm as the target image sample, and at least one image from image P2 to image P(m-1) as an intermediate image sample; after determining the intermediate image sample, the server can determine the time corresponding to the intermediate image sample according to the number of intermediate image samples. For example, if the number of intermediate image samples is 3, the server can determine the time corresponding to the intermediate image sample at the front of the time axis order as 0.4s, and the time corresponding to the intermediate image sample at the back of the time axis order as 0.2s.

[0141] Step S901: Taking the first image samples and the target image samples in each of the image groups as input and the corresponding intermediate image samples as training targets, the first model and the second model are trained simultaneously until the loss functions of the first model and the second model converge.

[0142] In one or more embodiments, during the training process of the first model and the second model, the server can use the first image sample and the target image sample in each image group as input and a corresponding intermediate image sample as a training target to train the first model and the second model simultaneously until the loss functions of the first model and the second model converge; for example, the image group G1 includes the first image P1, the target image P2, the intermediate image P31, the intermediate image P32 and the intermediate image P33. The server can use the first image P1 and the target image P2 as input and the intermediate image P31, the intermediate image P32 and the intermediate image P33 as training targets to train the first model and the second model simultaneously.

[0143] In one or more embodiments, the loss function common to the first model and the second model consists of four parts, which are used to characterize the reconstruction loss, semantic loss, distortion loss, and smoothness loss of the first model and the second model, respectively. Among them, the reconstruction loss is used to characterize the quality of the reconstruction of the intermediate image, and the pixel value is usually in the range of [0,255]; the semantic loss is used to retain the predicted details and enhance the clarity of the target image; the distortion loss is used to measure the optical flow quality of the target image; and the smoothness loss is used to promote similar optical flows between adjacent pixels. Specifically, the loss function I can be expressed by the following formula:

[0144] I=λ r I r +λ p I p +λ w Iw +λ s I s ;

[0145] Among them, λ r To rebuild the loss r The corresponding weight, λ p is the semantic loss I p The corresponding weight, λ w is the distortion loss I w The corresponding weight, λ s is the smoothing loss I s The corresponding weight. Optionally, you can r Set to 0.8 and λ p Set to 0.005, and w Set to 0.4 and λ s Set to 1. Among them, the reconstruction loss I r It can be expressed by the following formula:

[0146]

[0147] Among them, i is used to represent the i-th intermediate image sample, It is used to represent the intermediate image sample predicted at time t, I t Used to represent the actual intermediate image sample at time t.

[0148] Semantic Loss I p It can be expressed by the following formula:

[0149]

[0150] Among them, Φ is used to represent the conv4_3 features of the VGG16 model pre-trained on the ImageNet dataset.

[0151] Distortion Loss I w It can be expressed by the following formula:

[0152]

[0153] Smoothing Loss I s It can be expressed by the following formula:

[0154]

[0155] Therefore, when the loss function converges, it can be considered that the first model and the second model are trained, and an intermediate image with higher smoothness can be obtained based on the first image and the target image.

[0156] Fig.10 FIG. 1 is a schematic diagram of a video processing device according to an embodiment of the present invention. Fig.10 As shown, the device of this embodiment includes an acquisition unit 1001 , a determination unit 1002 and a processing unit 1003 .

[0157] Among them, the acquisition unit is used to acquire at least two image sequences; the determination unit is used to determine a transition image set based on the at least two image sequences, wherein the transition image set includes multiple transition images, and the transition image is an image located at a predetermined position at the beginning or the end of the image sequence; the determination unit is also used to determine a target image based on the transition image set; the processing unit is used to splice the at least two image sequences based on the target image to obtain a target image sequence.

[0158] Furthermore, the determining unit is specifically configured to:

[0159] In the transition image set, an image with the smallest difference from the remaining images is determined as the target image.

[0160] Furthermore, the predetermined position is a starting position or an ending position of the image sequence.

[0161] Furthermore, the at least two image sequences include a first image sequence and a second image sequence, and the splicing order is that the first image sequence is in front and the second image sequence is in the back;

[0162] The processing unit is specifically configured to: replace the image corresponding to the end position of the first image sequence and the image corresponding to the start position of the second image sequence with the target image;

[0163] The replaced first image sequence and the replaced second image sequence are spliced ​​into a target image sequence.

[0164] Furthermore, the processing unit is further used to perform smoothing processing on the last part of the image sequence of the set duration in the replaced first image sequence;

[0165] The image sequence of the beginning part of the set time length in the replaced second image sequence is smoothed.

[0166] Furthermore, the processing unit is specifically used for:

[0167] Determine a timestamp of a first image, where the first image is an image in the first image sequence after replacement whose timestamp is earlier than that of the target image;

[0168] Determining a similarity between the first image and the target image;

[0169] In response to the similarity satisfying a predetermined similarity condition, based on a pre-trained first model, determining a corresponding forward optical flow field vector and a reverse optical flow field vector according to the first image and the target image, the forward optical flow field vector being used to characterize the forward optical flow of the first image and the target image, and the reverse optical flow field vector being used to characterize the reverse optical flow of the first image and the target image;

[0170] Based on a pre-trained second model, at least one intermediate image is determined according to the forward optical flow field vector and the reverse optical flow field vector, where the intermediate image is an image between the first image and the target image.

[0171] Furthermore, the processing unit is specifically used for:

[0172] The optical flow similarity between the first image and the target image is calculated as the similarity.

[0173] Furthermore, the determining unit is further configured to:

[0174] A target video is determined according to the target image sequence and the corresponding audio sequence.

[0175] Fig.11 is a schematic diagram of an electronic device according to an embodiment of the present invention. Fig.11 The electronic device shown is a general video processing device, which includes a general computer hardware structure, which at least includes a processor 1101 and a memory 1102. The processor 1101 and the memory 1102 are connected via a bus 1103. The memory 1102 is suitable for storing instructions or programs executable by the processor 1101. The processor 1101 can be an independent microprocessor or a collection of one or more microprocessors. Thus, the processor 1101 executes the instructions stored in the memory 1102, thereby executing the method flow of the embodiment of the present invention as described above to realize the processing of data and the control of other devices. The bus 1103 connects the above-mentioned multiple components together, and at the same time connects the above-mentioned components to the display controller 1104 and the display device and the input / output (I / O) device 1105. The input / output (I / O) device 1105 can be a mouse, a keyboard, a modem, a network interface, a touch input device, a somatosensory input device, a printer, and other devices known in the art. Typically, the input / output device 1105 is connected to the system through an input / output (I / O) controller 1106.

[0176] As will be appreciated by those skilled in the art, various aspects of the embodiments of the present invention may be implemented as a system, method or computer program product. Therefore, various aspects of the embodiments of the present invention may take the form of a complete hardware implementation, a complete software implementation (including firmware, resident software, microcode, etc.), or an implementation that combines software aspects with hardware aspects, which may generally be referred to herein as a "circuit," "module," or "system." In addition, various aspects of the embodiments of the present invention may take the form of a computer program product implemented in one or more computer-readable media having a computer-readable program code implemented thereon.

[0177] Any combination of one or more computer-readable media can be utilized. Computer-readable media can be computer-readable signal media or computer-readable storage media. Computer-readable storage media can be, for example, (but not limited to) electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or apparatuses, or any suitable combination of the foregoing. More specific examples (non-exhaustive enumeration) of computer-readable storage media will include the following: an electrical connection with one or more wires, a portable computer floppy disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of an embodiment of the present invention, a computer-readable storage medium can be any tangible medium that can contain or store a program used by an instruction execution system, device or apparatus or a program used in conjunction with an instruction execution system, device or apparatus.

[0178] A computer readable signal medium may include a propagated data signal having a computer readable program code implemented therein, such as in baseband or as part of a carrier wave. Such propagated signals may take any of a variety of forms, including but not limited to electromagnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any of the following computer readable media: not a computer readable storage medium, and may communicate, propagate, or transmit a program used by or in conjunction with an instruction execution system, device, or apparatus.

[0179] Program code embodied on a computer readable medium may be transmitted using any appropriate medium including, but not limited to, wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0180] The computer program code for performing operations for various aspects of the embodiments of the present invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Smalltalk, C++, etc., and conventional process programming languages ​​such as "C" programming language or similar programming languages. The program code can be executed completely on the user's computer, partially on the user's computer as a stand-alone software package; partially on the user's computer and partially on a remote computer; or completely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer through any type of network including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., by using the Internet of an Internet service provider).

[0181] The flowchart legend and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present invention described above describe various aspects of the embodiment of the present invention.It will be understood that each block of the flowchart legend and / or block diagram and the combination of the blocks in the flowchart legend and / or block diagram can be implemented by computer program instructions.These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine so that the instructions (executed by the processor of the computer or other programmable data processing device) create a device for implementing the function / action specified in the flowchart and / or block diagram block or block.

[0182] These computer program instructions may also be stored in a computer-readable medium that can direct a computer, other programmable data processing device, or other apparatus to operate in a particular manner, so that the instructions stored in the computer-readable medium produce an article of manufacture including instructions for implementing the functions / actions specified in the flowchart and / or block diagram blocks or blocks.

[0183] The computer program instructions may also be loaded onto a computer, other programmable data processing device or other apparatus so that a series of operable steps are performed on the computer, other programmable device or other apparatus to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide a process for implementing the functions / actions specified in the flowchart and / or block diagram block or blocks.

[0184] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A video processing method, characterized in that: The method includes: acquiring at least two image sequences; Determine a transition image set according to the at least two image sequences, wherein the transition image set includes a plurality of transition images, and the transition images are images located at predetermined positions at the beginning or the end of the image sequence; Determine a target image according to the transition image set; Replacing the image corresponding to the end position of the first image sequence and the image corresponding to the start position of the second image sequence with the target image; splicing the replaced first image sequence and the replaced second image sequence, and performing smoothing on the spliced ​​portion to obtain a target image sequence; The smoothing process includes: When the similarity between a first image earlier than the target image in the replaced first image sequence and the target image satisfies a predetermined similarity condition, based on a pre-trained first model, a corresponding forward optical flow field vector and a reverse optical flow field vector are determined according to the first image and the target image, the forward optical flow field vector is used to characterize the forward optical flow of the first image and the target image, and the reverse optical flow field vector is used to characterize the reverse optical flow of the first image and the target image; Based on a pre-trained second model, determining at least one intermediate image according to the forward optical flow field vector and the reverse optical flow field vector, wherein the intermediate image is an image between the first image and the target image; Wherein, training the first model and the second model comprises: In an unsupervised manner, the first model and the second model are trained simultaneously using an image group of the first image, the target image, and the intermediate image as training data until the loss functions of the first model and the second model converge; Wherein, the determining at least one intermediate image based on the pre-trained second model according to the forward optical flow field vector and the reverse optical flow field vector comprises: Determine a first approximate optical flow field vector and a second approximate optical flow field vector according to the forward optical flow and the reverse optical flow, wherein the first approximate optical flow field vector is used to characterize the forward approximate optical flow at an intermediate moment, and the second approximate optical flow field vector is used to characterize the reverse approximate optical flow at the intermediate moment; determining a first interpolation function according to the first image and the first approximate optical flow field vector; Determining a second interpolation function according to the target image and the second approximate optical flow field vector; Taking the first image, the target image, the first approximate optical flow field vector, the second approximate optical flow field vector, the first interpolation function and the second interpolation function as inputs of the second model, determining a forward visible graph, a reverse visible graph, a first increment and a second increment at the intermediate moment, wherein the first increment is used to represent the increment of the first approximate optical flow field vector at the intermediate moment, and the second increment is used to represent the increment of the second approximate optical flow field vector at the intermediate moment; The intermediate image corresponding to the intermediate moment is determined according to the first image, the target image, the forward visible image, the reverse visible image, the first increment, and the second increment.

2. The method according to claim 1, characterized in that The determining of the target image according to the transition image set specifically includes: In the transition image set, an image with the smallest difference from the remaining images is determined as the target image.

3. The method according to claim 1, characterized in that The predetermined position is the starting position or the ending position of the image sequence.

4. The method according to claim 3, characterized in that The at least two image sequences include a first image sequence and a second image sequence, and the splicing order is that the first image sequence is in front and the second image sequence is in the back; The step of splicing the replaced first image sequence and the replaced second image sequence and performing smoothing on the spliced ​​portion to obtain the target image sequence specifically includes: Smoothing the last part of the image sequence of the set duration in the replaced first image sequence; The image sequence of the beginning part of the set time length in the replaced second image sequence is smoothed.

5. The method according to claim 4, characterized in that The step of smoothing the last part of the image sequence of the set duration in the replaced first image sequence specifically includes: Determine a timestamp of a first image, where the first image is an image in the first image sequence after replacement whose timestamp is earlier than that of the target image; A similarity between the first image and the target image is determined.

6. The method according to claim 5, characterized in that The determining the similarity between the first image and the target image specifically includes: The optical flow similarity between the first image and the target image is calculated as the similarity.

7. The method according to claim 1, characterized in that The method further comprises: A target video is determined according to the target image sequence and the corresponding audio sequence.

8. A video processing device, characterized in that: The device includes: An acquisition unit, used for acquiring at least two image sequences; a determining unit, configured to determine a transition image set according to the at least two image sequences, wherein the transition image set includes a plurality of transition images, and the transition images are images located at predetermined positions at the beginning or the end of the image sequence; The determining unit is further used to determine the target image according to the transition image set; A processing unit, configured to replace the image corresponding to the end position of the first image sequence and the image corresponding to the start position of the second image sequence with the target image; splicing the replaced first image sequence and the replaced second image sequence, and performing smoothing on the spliced ​​portion to obtain the target image sequence; The smoothing process includes: When the similarity between a first image earlier than the target image in the replaced first image sequence and the target image satisfies a predetermined similarity condition, based on a pre-trained first model, a corresponding forward optical flow field vector and a reverse optical flow field vector are determined according to the first image and the target image, the forward optical flow field vector is used to characterize the forward optical flow of the first image and the target image, and the reverse optical flow field vector is used to characterize the reverse optical flow of the first image and the target image; Based on a pre-trained second model, determining at least one intermediate image according to the forward optical flow field vector and the reverse optical flow field vector, wherein the intermediate image is an image between the first image and the target image; Wherein, training the first model and the second model comprises: In an unsupervised manner, the first model and the second model are trained simultaneously using an image group of the first image, the target image, and the intermediate image as training data until the loss functions of the first model and the second model converge; Wherein, the determining at least one intermediate image based on the pre-trained second model according to the forward optical flow field vector and the reverse optical flow field vector comprises: Determine a first approximate optical flow field vector and a second approximate optical flow field vector according to the forward optical flow and the reverse optical flow, wherein the first approximate optical flow field vector is used to characterize the forward approximate optical flow at an intermediate moment, and the second approximate optical flow field vector is used to characterize the reverse approximate optical flow at the intermediate moment; determining a first interpolation function according to the first image and the first approximate optical flow field vector; Determining a second interpolation function according to the target image and the second approximate optical flow field vector; Taking the first image, the target image, the first approximate optical flow field vector, the second approximate optical flow field vector, the first interpolation function and the second interpolation function as inputs of the second model, determining a forward visible graph, a reverse visible graph, a first increment and a second increment at the intermediate moment, wherein the first increment is used to represent the increment of the first approximate optical flow field vector at the intermediate moment, and the second increment is used to represent the increment of the second approximate optical flow field vector at the intermediate moment; The intermediate image corresponding to the intermediate moment is determined according to the first image, the target image, the forward visible image, the reverse visible image, the first increment, and the second increment.

9. A computer-readable storage medium storing computer program instructions, characterized in that: The computer program instructions implement the method according to any one of claims 1 to 7 when executed by a processor.

10. An electronic device comprising a memory and a processor, characterized in that: The memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method according to any one of claims 1-7.