Model training method and device

Abstract

The embodiment of the invention provides a model training method and device, computer equipment, a computer readable storage medium and a computer program product, and belongs to the field of video processing. The model training method comprises the following steps: acquiring a first video frame, a second video frame, a third video frame and an optical flow mask graph; wherein the first video frame, the second video frame and the third video frame are continuous video frames which are sequentially distributed and are all provided with target texts, and the optical flow mask graph is used for indicating the positions of the target texts on frame pictures; taking the first video frame, the third video frame and the optical flow mask graph as model input, and outputting a predicted video frame through the frame insertion model; and adjusting the frame insertion model according to the predicted video frame and the second video frame. According to the technical scheme provided by the embodiment of the invention, the static characteristic of the text foreground can be more accurately maintained when the predicted video frame is generated, the deformation of the text caused by background motion is reduced, and the accuracy and visual effect of frame insertion can be improved.

Description

Technical Field

[0001] This application relates to the field of video processing technology, and in particular to a model training method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] With the rapid development of internet technology, the production and dissemination of video content have experienced explosive growth. Whether in movies, TV series, short online videos, or live streaming, video quality and viewing experience are crucial. Video frame interpolation refers to adding several frames between every two consecutive frames in a video sequence. In practice, video frame interpolation can be performed in various ways, such as analyzing consecutive frames to generate optical flow to determine the inter-frame mapping relationship and thus synthesizing intermediate frames—this is frame interpolation based on an interpolation model. However, frame interpolation based on an interpolation model results in static subtitles on the screen, while when objects in the background move, the text in the subtitle area becomes distorted and deformed along with the background movement, affecting the quality of the video frame interpolation.

[0003] It should be noted that the above content is not necessarily prior art, nor is it intended to limit the scope of patent protection of this application. Summary of the Invention

[0004] This application provides a model training method, apparatus, computer device, computer-readable storage medium, and computer program product to solve or alleviate one or more of the technical problems mentioned above.

[0005] One aspect of this application provides a model training method, the method comprising: Acquire a first video frame, a second video frame, a third video frame, and an optical flow mask; wherein the first video frame, the second video frame, and the third video frame are sequentially distributed consecutive video frames and each contains target text, and the optical flow mask is used to indicate the position of the target text on the frame; The first video frame, the third video frame, and the optical flow mask are used as model inputs, and the predicted video frame is output through the frame interpolation model. The frame interpolation model is adjusted based on the predicted video frame and the second video frame.

[0006] Optionally, acquiring the first video frame, the second video frame, the third video frame, and the optical flow mask includes: The optical flow mask is generated based on the target text, wherein the target text is located at a preset text position in the optical flow mask; Acquire consecutive first, second, and third initial video frames; According to the preset text position, the target text is added to the corresponding positions of the first initial video frame, the second initial video frame and the third initial video frame respectively to obtain the first video frame, the second video frame and the third video frame.

[0007] Optionally, generating the optical flow mask map based on the target text includes: Obtain an initial mask image, which includes multiple pixels, each pixel corresponding to a mask value, and the mask values ​​corresponding to the multiple pixels are all zero values; Based on the target text and the preset text position, determine multiple text pixels from the multiple pixels; The mask values ​​corresponding to the multiple text pixels are adjusted from the zero value to a preset value to obtain the optical flow mask map.

[0008] Optionally, the frame interpolation model includes an optical flow prediction module; Using the first video frame, the third video frame, and the optical flow mask as model inputs, the interpolation model outputs predicted video frames, including: Based on the first video frame and the third video frame, the optical flow prediction module generates a first optical flow map and a second optical flow map; wherein, the first optical flow map is used to describe the optical flow result between the first video frame and the predicted video frame, and the second optical flow map is used to describe the optical flow result between the third video frame and the predicted video frame. Based on the first optical flow map, the second optical flow map, and the optical flow mask map, a first shielded optical flow map and a second shielded optical flow map are determined; wherein, the first shielded optical flow map corresponds to the first optical flow map, and the second shielded optical flow map corresponds to the second optical flow map; The predicted video frame is generated using the frame interpolation model based on the first and second shielded optical flow maps.

[0009] Optionally, the first optical flow map includes a first target pixel, the optical flow mask map includes a second target pixel corresponding to the first target pixel, and the first masking optical flow map includes a third target pixel corresponding to the first target pixel; Determining the first shielded optical flow map and the second shielded optical flow map based on the first optical flow map, the second optical flow map, and the optical flow mask map includes: Determine the target mask value of the second target pixel and the optical flow value of the first target pixel; Determine the difference between the preset value and the target mask value; The optical flow value of the third target pixel is determined based on the difference and the optical flow value of the first target pixel.

[0010] Optionally, the first video frame, the predicted video frame, and the third video frame all include target pixels corresponding to the same element; Based on the first and second shielded optical flow maps, the predicted video frames are generated using the frame interpolation model, including: Based on the first video frame and the second video frame, a target weight value for the target pixel is determined. The target weight value is used to describe the correlation between the position of the target pixel in the predicted video frame and the optical flow value of the target pixel in the first video frame. Based on the first shielding optical flow map, the first optical flow value of the target pixel in the first video frame is determined, and based on the first optical flow value and the target weight value, the first prediction result is determined; Based on the second shielded optical flow map, the second optical flow value of the target pixel in the third video frame is determined, and based on the second optical flow value and the supplementary weight value, the second prediction result is determined, wherein the sum of the target weight value and the supplementary weight value is 1; Based on the first prediction result and the second prediction result, the position of the target pixel in the predicted video frame is determined.

[0011] Another aspect of this application provides a frame interpolation method, the method comprising: Acquire adjacent first target video frames and second target video frames; The first target video frame and the second target video frame are used as model inputs, and the target predicted video frame is output through a frame interpolation model; the frame interpolation model is trained using the model training method of any one of claims 1 to 6; and The target predicted video frame is inserted between the first target video frame and the second target video frame.

[0012] Another aspect of this application provides a model training apparatus, the apparatus comprising: The acquisition module is used to acquire a first video frame, a second video frame, a third video frame, and an optical flow mask; wherein the first video frame, the second video frame, and the third video frame are sequentially distributed consecutive video frames and each has target text, and the optical flow mask is used to indicate the position of the target text on the frame. The output module is used to take the first video frame, the third video frame and the optical flow mask as model input, and output the predicted video frame through the frame interpolation model; An adjustment module is used to adjust the frame interpolation model based on the predicted video frame and the second video frame.

[0013] Another aspect of this application provides a frame interpolation apparatus, the apparatus comprising: The acquisition module is used to acquire adjacent first target video frames and second target video frames. The output module is used to take the first target video frame and the second target video frame as model input, and output the target predicted video frame through the frame interpolation model; the frame interpolation model is trained by the model training method of the above embodiment; and An insertion module is used to insert the target predicted video frame between the first target video frame and the second target video frame.

[0014] Another aspect of this application provides a computer device, including: At least one processor; and A memory that is communicatively connected to the at least one processor; Wherein: the memory stores instructions that can be executed by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method as described above.

[0015] Another aspect of this application provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the method described above.

[0016] Another aspect of this application provides a computer program product including a computer program that, when executed by a processor, implements the method described above.

[0017] The embodiments of this application employing the above technical solution may include the following advantages: by using an optical flow mask map to indicate the position of the target text on the frame, the frame interpolation model can selectively distinguish between static text foreground and moving background during training, thereby more accurately maintaining the static characteristics of the text foreground when generating predicted video frames, reducing the deformation of the text caused by background movement, and improving the accuracy and visual effect of frame interpolation. Attached Figure Description

[0018] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0019] Figure 1 The diagram illustrates the operating environment of the model training method according to Embodiment 1 of this application. Figure 2 A flowchart illustrating the model training method according to Embodiment 1 of this application is shown schematically. Figure 3 Schematic illustration Figure 2 Flowchart of the sub-steps in step S200; Figure 4 Schematic illustration Figure 3 Flowchart of the sub-steps in step S300; Figure 5 Schematic illustration Figure 2 Flowchart of the sub-steps in step S202; Figure 6 Schematic illustration Figure 5 Flowchart of the sub-steps in step S502; Figure 7 Schematic illustration Figure 5 Flowchart of the sub-steps in step S504; Figure 8 A flowchart illustrating the frame interpolation method according to Embodiment 2 of this application is shown schematically; Figure 9 The illustration shows an exemplary application flowchart of the model training method and frame interpolation method according to embodiments of this application; Figure 10 This illustration shows the text deformation generated when using an interpolation model to predict intermediate frames. Figure 11 A schematic diagram of a first video frame (or a second video frame, or a third video frame) is shown. Figure 12 A schematic diagram of an optical flow mask is shown. Figure 13 A block diagram of a model training apparatus according to Embodiment 3 of this application is shown schematically; Figure 14 A block diagram of a frame interpolation device according to Embodiment 4 of this application is schematically shown; and Figure 15 A schematic diagram of the hardware architecture of a computer device according to Embodiment 5 of this application is shown. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0021] It should be noted that the descriptions involving "first," "second," etc., in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0022] It should be noted that, in any stage of this application involving the collection, storage, use, transmission, and processing of data, each stage strictly adheres to the laws, regulations, industry standards, and regulatory requirements of the data source, usage location, and relevant countries and regions to ensure the legality and compliance of data activities. In the collection stage, the purpose, method, and scope of collection are clearly communicated to the data subject in a prominent manner. Collection is conducted only after obtaining the data subject's legal authorization, ensuring that the collection process follows the "minimum necessary" principle and does not exceed the scope of data collection. In the storage stage, storage periods are limited, and data is promptly deleted or anonymized / encrypted after the storage purpose is achieved. In the usage stage, a strict data security protection mechanism is implemented, using field-level desensitization technology and processing the original data according to preset desensitization rules. For different types of data, multiple desensitization strategies, such as data generalization, data anonymization, and data encryption, are employed to effectively mitigate the risk of sensitive information leakage and ensure that all data used is securely processed and desensitized, comprehensively protecting the rights and interests of data subjects and data security. In the transmission and processing stages, the confidentiality and security of data are ensured during transmission and processing.

[0023] In the description of this application, it should be understood that the numerical labels before the steps do not indicate the order of the steps, but are only used to facilitate the description of this application and to distinguish each step, and therefore should not be construed as a limitation of this application.

[0024] First, a definition of the terminology used in this application is provided: Video frame interpolation: a technique that inserts extra frames between adjacent frames of an original video. It generates intermediate frames to fill in time gaps by analyzing motion information between consecutive frames.

[0025] Optical flow is a technique used to describe the motion of pixels in an image sequence between adjacent frames. It estimates the motion vector of each pixel in the time dimension by analyzing the brightness changes of pixels.

[0026] Mask: A tool in image processing and computer vision, such as a two-dimensional matrix of the same size as the image, used to label the attributes or features of different regions in an image.

[0027] Motion detection: A computer vision technique used to identify and analyze the motion of objects in a video sequence by comparing pixel changes between consecutive frames to detect which areas are moving.

[0028] Scene transition detection: A technique in video processing used to identify transition points between different scenes in a video.

[0029] Bidirectional optical flow: a technique for video processing and analysis that simultaneously estimates optical flow fields in two directions: the optical flow field from the current frame to the previous frame (forward optical flow) and the optical flow field from the previous frame to the current frame (backward optical flow).

[0030] Optical flow correction: a technique for optimizing and adjusting the optical flow field obtained in the initial calculation process during optical flow estimation. By correcting errors or unreasonable parts in the optical flow field, it better reflects the actual movement of pixels between video frames.

[0031] The Horn-Schunck method is a global optical flow estimation algorithm used to calculate the motion vectors of pixels in an image sequence. Based on the assumptions of constant brightness and smoothness, it estimates the optical flow field by minimizing the error of the optical flow constraint equation.

[0032] The Lucas-Kanade method is a local optical flow estimation algorithm used to calculate the motion of pixels in an image sequence. Based on the assumptions of constant brightness and small motion, it estimates the motion vector of each pixel by solving the optical flow constraint equation within a local window.

[0033] FlowNet: A deep learning model architecture for optical flow estimation that takes continuous image pairs as input and outputs pixel-level optical flow fields by constructing a convolutional neural network.

[0034] PWC-Net (Pyramid, Warping, and Cost-Volume Network): A deep learning model for optical flow estimation. It constructs a pyramid structure to process image information at different scales, utilizes the propagation and update mechanism of optical flow, and combines cost volume to calculate the matching relationship between pixels, thereby estimating the motion information of objects in the image sequence.

[0035] Secondly, to facilitate understanding of the technical solutions provided in the embodiments of this application by those skilled in the art, the relevant technologies are described below: Video frame interpolation refers to adding several frames between every two consecutive frames in a video sequence. This can shorten the display time between frames, improving the frame rate and smoothness of the video. In practice, video frame interpolation can be performed in various ways. For example, directly copying the previous or next frame as the interpolation frame, or applying a double-exposure-like blurring process to the two frames to obtain an intermediate frame, or analyzing and modeling the two frames to generate optical flow and thus obtain a linear mapping relationship between frames, ultimately combining them to create an intermediate frame—this is frame interpolation based on an interpolation model. Although frame interpolation based on an interpolation model has a much better effect than the previous two methods, when there are static subtitles in the frame and objects in the background are moving, the text in the subtitle area will be distorted with the background movement, such as… Figure 10 As shown, when the legs are swinging, the subtitles in front of the legs cannot be correctly synthesized by the frame interpolation model.

[0036] Therefore, this application provides a model training technology solution. In this technology solution, (1) the static characteristics of the text foreground can be maintained more accurately when generating predicted video frames, reducing the deformation of the text caused by background movement, which can improve the accuracy and visual effect of frame interpolation; (2) the mask value of the text pixel is adjusted from zero to a preset value to obtain an optical flow mask map, so that the frame interpolation model can identify the location of the target text; (3) the optical flow value of the third target pixel is determined according to the difference between the target mask value and the preset value, and the optical flow value of the pixel corresponding to the target text can be changed to 0, so that each pixel corresponding to the target text remains static during the frame interpolation process and does not move. See below for details.

[0037] Finally, for ease of understanding, an exemplary operating environment is provided below.

[0038] like Figure 1 As shown in the diagram, the operating environment includes: server 2 and clients (4A, 4B, ..., 4N).

[0039] Server 2 can connect to clients (4A, 4B, ..., 4N) via the network.

[0040] Server 2 can be a single server, a server cluster, or a cloud computing service center.

[0041] Server 2 can provide video frame interpolation services, model training services, etc. to the client.

[0042] Server 2 can be located in a data center, such as a single location, or distributed across different geographical locations (e.g., multiple locations). Server 2 can provide services via a network. The network includes various network devices such as routers, switches, multiplexers, hubs, modems, bridges, repeaters, firewalls, proxy devices, and / or similar devices. The network can include physical links, such as coaxial cable links, twisted-pair cable links, fiber optic links, or combinations thereof, or wireless links, such as cellular links, satellite links, Wi-Fi links, etc.

[0043] Clients (4A, 4B, ..., 4N) can be configured to access the content and services of server 2. Clients (4A, 4B, ..., 4N) can include electronic devices with built-in or external display panels, such as mobile devices, tablets, laptops, workstations, virtual reality devices, gaming devices, digital streaming media devices, vehicle terminals, smart TVs, set-top boxes, etc., and can also include virtualized computing instances. Virtualized computing instances can include virtual machines, such as simulations of computer systems, operating systems, servers, etc. The computing device can load the virtual machine based on the virtual image and / or other data defining specific software (e.g., operating system, dedicated applications, servers) used for simulation. As the demand for different types of processing services changes, different virtual machines can be loaded and / or terminated on one or more computing devices.

[0044] A client (4A, 4B, ..., 4N) can be associated with one or more users. A single user can also use one or more of the clients (4A, 4B, ..., 4N) to access server 2. Clients (4A, 4B, ..., 4N) can travel to various locations and use different networks to access server 2.

[0045] The client (4A, 4B, ..., 4N) may include an interface. This interface may include a touchpad, touchscreen, mouse, keyboard, or other sensing elements. For example, the input element may be configured to receive user commands that cause the client (4A, 4B, ..., 4N) to perform various operations, such as uploading video frames or inputting target text.

[0046] It should be noted that the above-mentioned equipment is exemplary, and the number and type of equipment can be adjusted in different scenarios or according to different needs.

[0047] The technical solutions of this application are described below through multiple embodiments, using server 2 or client 4A as the execution entity. It should be understood that these embodiments can be implemented in many different forms and should not be construed as being limited to the embodiments described herein.

[0048] Example 1 In this embodiment, server 2 can be used as the execution entity.

[0049] Figure 2 A flowchart illustrating a model training method according to Embodiment 1 of this application is shown schematically.

[0050] like Figure 2 As shown, the model training method may include steps S200~S204, wherein: Step S200: Obtain a first video frame, a second video frame, a third video frame, and an optical flow mask; wherein the first video frame, the second video frame, and the third video frame are sequentially distributed consecutive video frames and each has target text, and the optical flow mask is used to indicate the position of the target text on the frame. Step S202: The first video frame, the third video frame, and the optical flow mask are used as model inputs, and the predicted video frame is output through the frame interpolation model. Step S204: Adjust the frame interpolation model based on the predicted video frame and the second video frame.

[0051] The model training method provided in this embodiment uses an optical flow mask map to indicate the position of the target text on the frame, so that the frame interpolation model can selectively distinguish between static text foreground and moving background during the training process. This allows the model to more accurately maintain the static characteristics of the text foreground when generating predicted video frames, reduce the deformation of the text caused by background movement, and improve the accuracy and visual effect of frame interpolation.

[0052] The following combination Figure 2 The steps in steps S200 to S204, as well as other optional steps, are described in detail.

[0053] Step S200 The process involves acquiring a first video frame, a second video frame, a third video frame, and an optical flow mask. The first video frame, the second video frame, and the third video frame are sequentially distributed consecutive video frames, each containing target text. The optical flow mask is used to indicate the position of the target text on the frame.

[0054] The first, second, and third video frames can be three consecutive video frames extracted from the same video file, such as the first, second, and third frames at the beginning of a video. In some embodiments, scene switching or motion detection algorithms can be used to identify key video segments with significant scene changes or object movement, and consecutive video frames can be extracted from these key video segments. The target text can be subtitles, watermarks, or advertisements in the video. The target text can be located at the bottom, top, or other random positions of the frame. In some embodiments, the optical flow mask can be generated by identifying text regions in the first video frame (or the second or third video frame) through manual annotation or text detection algorithms. In other embodiments, the optical flow mask can also be generated according to certain rules based on pre-generated target text and text positions. The first video frame (or the second or third video frame) can be as follows: Figure 11 As shown, this includes the background figures and the foreground target text, "Frame Interpolation Model Training Method for Removing Text Jitter." The optical flow mask image can be obtained as follows: Figure 12 As shown, the image includes a black background and white text, with the text content and position identical to the first video frame.

[0055] As mentioned earlier, the video frames and optical flow mask maps required for model training can be obtained in various ways. The following provides another exemplary method for obtaining them.

[0056] In optional embodiments, such as Figure 3 As shown, step S200 includes: S300, Generate the optical flow mask based on the target text, wherein the target text is located at a preset text position in the optical flow mask.

[0057] S302, acquire consecutive first initial video frames, second initial video frames, and third initial video frames.

[0058] S304, according to the preset text position, add the target text to the corresponding positions of the first initial video frame, the second initial video frame and the third initial video frame respectively to obtain the first video frame, the second video frame and the third video frame.

[0059] In some embodiments, the target text can be custom-inputted using a text editor, and the content, font, size, color, etc., of the text can be specified during input. In other embodiments, the target text can be pre-generated text randomly selected from a preset text library. The preset text position can be fixed, such as a specific area like the upper left corner, lower right corner, or center of the frame, or it can be randomly distributed across the frame according to a certain probability. The target text can be added to each initial video frame through different methods such as direct overlay, semi-transparent overlay, or adaptive fusion based on the background content. In some embodiments, after adding the target text, the text clarity, readability, and contrast with the background of each video frame can be evaluated, and the qualified video frames can be used for model training.

[0060] In this embodiment, by obtaining corresponding training data (i.e., each video frame and optical flow mask map) through the target text and preset text positions, the consistency and accuracy between training data can be improved, thereby improving the training effect of the frame interpolation model and improving the accuracy of video frame interpolation.

[0061] As mentioned earlier, optical flow mask maps can be generated in various ways. The following provides an exemplary generation method.

[0062] In optional embodiments, such as Figure 4 As shown, step S300 includes: S400, Obtain an initial mask image, the initial mask image includes multiple pixels, each pixel corresponds to a mask value, and the mask values ​​corresponding to the multiple pixels are all zero values.

[0063] S402, determine multiple text pixels from the multiple pixels based on the target text and the preset text position.

[0064] S404, adjust the mask values ​​corresponding to the plurality of text pixels from the zero value to a preset value to obtain the optical flow mask map.

[0065] The initial mask can be a pre-created all-zero mask template or an all-zero mask of the same size as the video frame, generated in real time using an array or matrix library. In some embodiments, the preset value can be a fixed value (such as "1") or a dynamic value determined based on the text features of the target text or the image features of the initial video frame. The preset value can also be dynamically adjusted based on the model training results. For example, if it is found during training that the frame interpolation model does not perform well in recognizing certain types of text pixels, the preset value used during training with these types of text pixels can be appropriately increased. In other embodiments, different text portions of the same optical flow mask can use different preset values. For example, a higher value can be set for text that needs to be highlighted, while a lower value can be set for less important text.

[0066] In this embodiment, the mask value of the text pixels is adjusted from zero to a preset value to obtain an optical flow mask map, so that the frame interpolation model can identify the location of the target text, thereby more accurately distinguishing between static text foreground and moving background during the training process and improving the accuracy of frame interpolation.

[0067] Step S202 The first video frame, the third video frame, and the optical flow mask are used as model inputs, and the interpolation model outputs predicted video frames.

[0068] Before inputting each video frame and optical flow mask into the interpolation model, the temporal correspondence between the first and third video frames, as well as the size or resolution matching degree between the optical flow mask and the video frames, can be verified using a frame synchronization verification method. In some embodiments, the interpolation model can output multiple video frames at once, and a predicted video frame can be selected from the multiple output video frames based on the image quality of each video frame. In other embodiments, the input data can be processed and preliminary predicted video frames can be generated using multiple interpolation models with different structures or parameters, and then fused using a fusion network to obtain the final predicted video frame. In some embodiments, the predicted video frame can be obtained by estimating optical flow from both the forward and backward directions using a bidirectional optical flow estimation network, and the accuracy of optical flow estimation can be improved through consistency constraints. In other embodiments, the optical flow field can be corrected and adjusted using the optical flow mask using an optical flow correction network, or the optical flow field can be optimized multiple times using the optical flow mask using an iterative correction algorithm.

[0069] In this embodiment, an optical flow mask is used to indicate the location of the target text. Figure 1Inputting the text into the frame interpolation model to generate predicted video frames enables the model to accurately identify target text in images, thereby distinguishing between static text foreground and moving background, reducing abnormal phenomena such as blurring and jittering of target text in predicted video frames, and improving the accuracy and visual effect of frame interpolation.

[0070] As mentioned earlier, frame interpolation models can generate predicted video frames in various ways. The following provides an exemplary generation method.

[0071] In an optional embodiment, the frame interpolation model includes an optical flow prediction module, such as... Figure 5 As shown, step S202 includes: S500, based on the first video frame and the third video frame, the optical flow prediction module generates a first optical flow map and a second optical flow map; wherein, the first optical flow map is used to describe the optical flow result between the first video frame and the predicted video frame, and the second optical flow map is used to describe the optical flow result between the third video frame and the predicted video frame.

[0072] S502, determine a first shielded optical flow map and a second shielded optical flow map based on the first optical flow map, the second optical flow map and the optical flow mask map; wherein, the first shielded optical flow map corresponds to the first optical flow map and the second shielded optical flow map corresponds to the second optical flow map.

[0073] S504, Based on the first shielded optical flow map and the second shielded optical flow map, the predicted video frame is generated using the frame interpolation model.

[0074] The first and second optical flow maps can be generated using optical flow algorithms (such as the Horn-Schunck method, Lucas-Kanade method, etc.) or deep learning-based optical flow algorithms (such as FlowNet, PWC-Net, etc.). In some embodiments, the optical flow results of corresponding pixels in the first (or second) optical flow map can be processed according to the mask value of each pixel in the mask map to obtain the first masked optical flow map (or second masked optical flow map). In other embodiments, connected component analysis can be performed on the optical flow mask map first to identify the static text foreground region and the moving background region, and then the optical flow results of all pixels in the corresponding static text foreground region in the first (or second) optical flow map can be uniformly masked to obtain the first masked optical flow map (or second masked optical flow map). In some embodiments, the video frame can be predicted by first mapping the pixels in the first video frame to the corresponding positions in the predicted video frame according to the first masked optical flow map, then mapping the corresponding pixels in the third video frame to the same position in the predicted video frame according to the second masked optical flow map, and finally fusing the two mapping results. In other embodiments, different weights can be assigned to the two mapping results from the two video frames based on the distance of the predicted video frame to the first and third video frames on the time axis.

[0075] In this embodiment, the optical flow map and the second optical flow map are adjusted using an optical flow mask map. The optical flow of the text region is masked in the corresponding masked optical flow map. This allows the target text to remain static or stable within the video frame while the moving background can move normally, improving the smoothness and visual quality of the interpolated video. There are various ways to synthesize the first and second masked optical flow maps. An exemplary synthesis method is provided below.

[0076] In an optional embodiment, the first optical flow map includes a first target pixel, the optical flow mask map includes a second target pixel corresponding to the first target pixel, and the first masking optical flow map includes a third target pixel corresponding to the first target pixel.

[0077] like Figure 6 As shown, step S502 includes: S600, determine the target mask value of the second target pixel and the optical flow value of the first target pixel.

[0078] S602, determine the difference between the preset value and the target mask value.

[0079] S604, determine the optical flow value of the third target pixel based on the difference and the optical flow value of the first target pixel.

[0080] For example, if the first target pixel is a pixel on the character "去" in the first video frame, the target mask value of the corresponding second target pixel in the optical flow mask map should be a preset value (such as "1"). At this time, the difference between the target mask value and the preset value is 0, so the optical flow value of the third target pixel is also 0.

[0081] For another example, if the first target pixel is a pixel outside the text area in the first video frame, the target mask value of the corresponding second target pixel in the optical flow mask map should be 0. At this time, the difference between the target mask value and the preset value (such as "1") is the preset value, so the optical flow value of the third target pixel is also a preset multiple of the optical flow value of the first target pixel (for example, when the preset value is "1", the optical flow value of the third target pixel is the optical flow value of the first target pixel). In some embodiments, the range of the optical flow value of the adjusted third target pixel can also be set. For example, the magnitude of the optical flow value cannot exceed a certain threshold. The optical flow value of the third target pixel can be calculated by the following formula: f'_t->0=(1 - text_mask)×f_t->0; where f'_t->0 is the optical flow value of the third target pixel, 1 is the preset value, text_mask is the target mask value of the second target pixel, and f_t->0 is the optical flow value of the first target pixel.

[0082] In this embodiment, determining the optical flow value of the third target pixel according to the difference between the target mask value and the preset value can change the optical flow values of the pixels corresponding to the target text to 0, so that the pixels corresponding to the target text remain stationary during the frame interpolation process and do not move, reducing abnormal phenomena such as text jitter or blurring, and improving the accuracy and quality of video frame interpolation.

[0083] As mentioned above, there are multiple ways to generate the predicted video frame. The following provides an exemplary generation method.

[0084] In an alternative embodiment, the first video frame, the predicted video frame, and the third video frame all include target pixels corresponding to the same element.

[0085] As Figure 7 shown, step S504 includes: S700, determining the target weight value of the target pixel according to the first video frame and the second video frame, where the target weight value is used to describe the association degree between the position of the target pixel in the predicted video frame and the optical flow value of the target pixel in the first video frame.

[0086] S702, based on the first shielding optical flow map, determine the first optical flow value of the target pixel in the first video frame, and based on the first optical flow value and the target weight value, determine the first prediction result.

[0087] S704, based on the second shielded optical flow map, determine the second optical flow value of the target pixel in the third video frame, and based on the second optical flow value and the supplementary weight value, determine the second prediction result, wherein the sum of the target weight value and the supplementary weight value is 1.

[0088] S706, Based on the first prediction result and the second prediction result, determine the position of the target pixel in the predicted video frame.

[0089] In some embodiments, a prediction algorithm can be used to assess the extent to which the position of a target pixel in a predicted video frame is derived from the optical flow motion of a target pixel in a first video frame, thereby determining the target weight value. In other embodiments, the target weight value can also be determined based on motion characteristics of the target pixel in the video frame sequence, such as its direction of motion, velocity, and acceleration. The target weight value can also be adjusted based on the stability of the target pixel in the video sequence, such as whether it is a stationary object or part of a moving object. The predicted video frame can be calculated using the following formula: I'_t=f_map×warp(I_0, f'_t->0)+(1-f_map)×warp(I_1, f'_t->1); Where I'_t is the predicted video frame, f_map is the weight matrix composed of the target weight values ​​of each pixel in the predicted video frame, warp(I_0, f'_t->0) means that according to the first shielded optical flow map f'_t->0, the pixels of the previous frame I_0 are mapped to the position of the predicted video frame I'_t, and warp(I_1, f'_t->1) means that according to the second shielded optical flow map f'_t->1, the pixels of the next frame I_1 are mapped to the position of the predicted video frame I'_t.

[0090] In this embodiment, the position of the target pixel in the predicted video frame is determined according to the optical flow results of the target pixel in the first video frame and the third video frame, based on the corresponding target weight value. This fully considers the motion of the target pixel in the preceding and following frames, thereby improving the accuracy and quality of the predicted video frame.

[0091] Step S204 The interpolation model is adjusted based on the predicted video frame and the second video frame.

[0092] In some embodiments, the loss between the predicted video frame and the second video frame can be calculated using a pixel-level loss function, a perceptual loss function, or an adversarial loss function, thereby updating the gradient of the frame interpolation model for adjustment. In some embodiments, multiple loss functions can be combined for adjustment, and the weights of different loss functions can be dynamically adjusted based on error changes or other metrics during training. In other embodiments, the loss functions for different regions can be weighted according to their importance or characteristics within the video frame. For example, regions containing target text can be given higher weights, while background or other less important regions can have their weights appropriately reduced.

[0093] In this embodiment, the interpolation model is adjusted in reverse based on the comparison result (loss value) by comparing the predicted video frame with the actual second video frame. The predicted video frame is generated based on the optical flow mask map used to indicate the location of the target text. By introducing supervision information of the text optical flow mask, the interpolation model can be guided to accurately distinguish between static text regions and moving background images, thereby improving the accuracy and quality of video interpolation.

[0094] Example 2 In this embodiment, client 4A can be used as the execution subject.

[0095] Figure 8 A flowchart illustrating the frame interpolation method according to Embodiment 2 of this application is shown schematically.

[0096] like Figure 8 As shown, the frame interpolation method may include steps S800~S804, wherein: S800, acquire adjacent first target video frames and second target video frames.

[0097] S802, the first target video frame and the second target video frame are used as model inputs, and the target predicted video frame is output through the frame interpolation model; the frame interpolation model is trained by the model training method in Example 1.

[0098] S804, the target predicted video frame is inserted between the first target video frame and the second target video frame.

[0099] In this embodiment, the frame interpolation model trained according to the model training method of Embodiment 1 is used to perform frame interpolation, which can accurately identify the text region in adjacent frames, thereby accurately distinguishing between static foreground text and moving background images, reducing abnormal phenomena such as text jitter and blurring, and improving the accuracy and visual effect of video frame interpolation.

[0100] To make this application easier to understand, the following is combined with... Figure 9 An example application is provided. Wherein: Training phase: S11. Generate an optical flow mask using a mask generator. The bottom edge of the optical flow mask has a subtitle "Text". The mask value of the subtitle area of ​​the optical flow mask is 1, and the mask value of the area other than the subtitle area is 0. S12. Extract the first frame F1, the second frame F2, and the third frame F3 from the beginning of video A; S13. Add the same subtitle Text at the same position as the subtitle Text in the optical flow mask in F1, F2 and F3 to obtain a new first frame NF1, a new second frame NF2 and a new third frame NF3 with the added subtitle; S14. Input NF1, NF3 and the optical flow mask into the frame interpolation model M1; S15. Using the optical flow prediction module in the frame interpolation model, the first optical flow map corresponding to NF1 and the second optical flow map corresponding to NF3 are predicted based on NF1 and NF3. S16. Adjust the first optical flow map and the second optical flow map according to the optical flow mask map to obtain the corresponding first shielded optical flow map and the second shielded optical flow map; For example, there is a pixel C on the subtitle Text in NF1 and NF3. Its corresponding optical flow values ​​in the first optical flow map and the second optical flow map are Flow1 and Flow2, respectively. The mask value of pixel C in the optical flow mask map is 1. The difference between the mask value and the preset value (1) is 0. Therefore, its corresponding optical flow value in the first shielded optical flow map and the second shielded optical flow map is also 0. S17. Based on NF1 and NF3, determine the optical flow weight map Fmap, which includes the target weight value of each pixel in the predicted video frame. S18. Generate a predicted video frame PF based on the target weight value of each pixel and the corresponding optical flow value in the first and second shielded optical flow maps. For example, if the target weight value of target pixel D is 0.8, and its corresponding optical flow values ​​in the first and second shielded optical flow maps are Flow1 and Flow2 respectively, then the position prediction result of target pixel D in PF is SP=0.8×Flow1+(1-0.8)×Flow2==0.8×Flow1+0.2×Flow2; S19. Calculate the model loss based on the predicted video frame PF and NF2, and update the model gradient of the interpolation model M1.

[0101] Repeat steps S11-S19 to obtain the trained frame interpolation model M2.

[0102] Usage phase: S21. Input adjacent video frames F3 and F4 with subtitles into the frame interpolation model M2, and output the predicted intermediate frame PF2 through the frame interpolation model M2. S22. Insert the predicted intermediate frame PF2 between F3 and F4.

[0103] Example 3 Figure 13 The diagram schematically illustrates a model training apparatus according to Embodiment 3 of this application. This apparatus can be divided into one or more program modules. One or more program modules are stored in a storage medium and executed by one or more processors to complete the embodiments of this application. The program module referred to in the embodiments of this application refers to a series of computer program instruction segments capable of performing a specific function. The following description will specifically introduce the function of each program module in this embodiment. Figure 13 As shown, the device 1000 may include: an acquisition module 1100, an output module 1200, and an adjustment module 1300, wherein: The acquisition module 1100 is used to acquire a first video frame, a second video frame, a third video frame, and an optical flow mask; wherein the first video frame, the second video frame, and the third video frame are sequentially distributed consecutive video frames and each has target text, and the optical flow mask is used to indicate the position of the target text on the frame. Output module 1200 is used to take the first video frame, the third video frame and the optical flow mask as model input, and output the predicted video frame through the frame interpolation model; The adjustment module 1300 is used to adjust the frame interpolation model based on the predicted video frame and the second video frame.

[0104] As an optional embodiment, the acquisition module 1100 is further configured to: The optical flow mask is generated based on the target text, wherein the target text is located at a preset text position in the optical flow mask; Acquire consecutive first, second, and third initial video frames; According to the preset text position, the target text is added to the corresponding positions of the first initial video frame, the second initial video frame and the third initial video frame respectively to obtain the first video frame, the second video frame and the third video frame.

[0105] As an optional embodiment, the device 1000 further includes a mask image generation module, used for: Obtain an initial mask image, which includes multiple pixels, each pixel corresponding to a mask value, and the mask values ​​corresponding to the multiple pixels are all zero values; Based on the target text and the preset text position, determine multiple text pixels from the multiple pixels; The mask values ​​corresponding to the multiple text pixels are adjusted from the zero value to a preset value to obtain the optical flow mask map.

[0106] As an optional embodiment, the frame interpolation model includes an optical flow prediction module, and the output module 1200 is further configured to: Based on the first video frame and the third video frame, the optical flow prediction module generates a first optical flow map and a second optical flow map; wherein, the first optical flow map is used to describe the optical flow result between the first video frame and the predicted video frame, and the second optical flow map is used to describe the optical flow result between the third video frame and the predicted video frame. Based on the first optical flow map, the second optical flow map, and the optical flow mask map, a first shielded optical flow map and a second shielded optical flow map are determined; wherein, the first shielded optical flow map corresponds to the first optical flow map, and the second shielded optical flow map corresponds to the second optical flow map; The predicted video frame is generated using the frame interpolation model based on the first and second shielded optical flow maps.

[0107] As an optional embodiment, the first optical flow map includes a first target pixel, the optical flow mask map includes a second target pixel corresponding to the first target pixel, the first shielding optical flow map includes a third target pixel corresponding to the first target pixel, and the device 1000 further includes a shielding optical flow map determining module, used for: Determine the target mask value of the second target pixel and the optical flow value of the first target pixel; Determine the difference between the preset value and the target mask value; The optical flow value of the third target pixel is determined based on the difference and the optical flow value of the first target pixel.

[0108] As an optional embodiment, the first video frame, the predicted video frame, and the third video frame all include target pixels corresponding to the same element. The device 1000 further includes a predicted video frame generation module, used for: Based on the first video frame and the second video frame, a target weight value for the target pixel is determined. The target weight value is used to describe the correlation between the position of the target pixel in the predicted video frame and the motion result of the target pixel in the first video frame. Based on the first shielding optical flow map, a first motion result of the target pixel in the first video frame is determined, and a first prediction result is determined based on the first motion result and the target weight value. Based on the second shielded optical flow map, the second motion result of the target pixel in the third video frame is determined, and based on the second motion result and the supplementary weight value, the second prediction result is determined, wherein the sum of the target weight value and the supplementary weight value is 1; Based on the first prediction result and the second prediction result, the position of the target pixel in the predicted video frame is determined.

[0109] Example 4 Figure 14 The diagram schematically illustrates a frame interpolation apparatus according to Embodiment 4 of this application. This apparatus can be divided into one or more program modules. One or more program modules are stored in a storage medium and executed by one or more processors to complete the embodiments of this application. The program modules referred to in the embodiments of this application are a series of computer program instruction segments capable of performing specific functions. The following description will specifically introduce the functions of each program module in this embodiment. For example... Figure 14 As shown, the device 2000 may include: an acquisition module 2100, an output module 2200, and an insertion module 2300, wherein: The acquisition module 2100 is used to acquire adjacent first target video frames and second target video frames; Output module 2200 is used to take the first target video frame and the second target video frame as model input, and output the target predicted video frame through a frame interpolation model; the frame interpolation model is trained using the model training method of Embodiment 1; and The insertion module 2300 is used to insert the target predicted video frame between the first target video frame and the second target video frame.

[0110] Example 5 Figure 15 This illustration schematically shows a hardware architecture diagram of a computer device 10000 suitable for implementing a model training method according to Embodiment 5 of this application. In some embodiments, the computer device 10000 may be a terminal device such as a smartphone, wearable device, tablet computer, personal computer, in-vehicle terminal, game console, virtual device, workbench, digital assistant, set-top box, robot, etc. In other embodiments, the computer device 10000 may be a rack server, blade server, tower server, or cabinet server (including independent servers or server clusters composed of multiple servers), etc. Figure 15 As shown, the computer device 10000 includes, but is not limited to: a memory 10010, a processor 10020, and a network interface 10030 that can communicate and be linked with each other via a system bus. Wherein: The memory 10010 includes at least one type of computer-readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 10010 may be an internal storage module of a computer device 10000, such as the hard disk or memory of the computer device 10000. In other embodiments, the memory 10010 may also be an external storage device of the computer device 10000, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 10000. Of course, the memory 10010 may also include both the internal storage module and the external storage device of the computer device 10000. In this embodiment, the memory 10010 is typically used to store the operating system and various application software installed on the computer device 10000, such as program code for model training methods. In addition, the memory 10010 can also be used to temporarily store various types of data that have been output or will be output.

[0111] In some embodiments, processor 10020 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other chip. Processor 10020 is typically used to control the overall operation of computer device 10000, such as performing control and processing related to data interaction or communication with computer device 10000. In this embodiment, processor 10020 is used to run program code stored in memory 10010 or process data.

[0112] Network interface 10030 may include a wireless network interface or a wired network interface, which is typically used to establish a communication link between computer device 10000 and other computer devices. For example, network interface 10030 is used to connect computer device 10000 to an external terminal via a network, establishing a data transmission channel and communication link between computer device 10000 and the external terminal. The network may be an intranet, the Internet, Global System for Mobile Communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, or other wireless or wired networks.

[0113] It should be pointed out that, Figure 15 Only computer devices with components 10010-10030 are shown; however, it should be understood that it is not required to implement all of the shown components, and more or fewer components may be implemented instead.

[0114] In this embodiment, the model training method stored in memory 10010 can also be divided into one or more program modules and executed by one or more processors (such as processor 10020) to complete the embodiment of this application.

[0115] Example 6 This application also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the steps of the model training method in the embodiments.

[0116] In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the computer-readable storage medium can be an internal storage unit of a computer device, such as the hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device. Of course, the computer-readable storage medium can also include both the internal storage unit and the external storage device of the computer device. In this embodiment, the computer-readable storage medium is typically used to store the operating system and various application software installed on the computer device, such as the program code of the model training method in the embodiment. In addition, the computer-readable storage medium can also be used to temporarily store various types of data that have been output or will be output.

[0117] Example 7 This application also provides a computer program product, including a computer program that, when executed by a processor, implements the methods described in the above embodiments.

[0118] Obviously, those skilled in the art should understand that the modules or steps of the embodiments of this application described above can be implemented using general-purpose computer devices. They can be centralized on a single computer device or distributed across a network of multiple computer devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computer device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of this application are not limited to any particular combination of hardware and software.

[0119] It should be noted that the above are merely preferred embodiments of this application and do not limit the scope of patent protection of this application. Any equivalent structural or procedural changes made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of this application.

Claims

1. A model training method, characterized in that, The method includes: Acquire a first video frame, a second video frame, a third video frame, and an optical flow mask; wherein the first video frame, the second video frame, and the third video frame are sequentially distributed consecutive video frames and each contains target text, and the optical flow mask is used to indicate the position of the target text on the frame; The first video frame, the third video frame, and the optical flow mask are used as model inputs, and the predicted video frame is output through the frame interpolation model. The frame interpolation model is adjusted based on the predicted video frame and the second video frame.

2. The method of claim 1, wherein, Obtain the first video frame, the second video frame, the third video frame, and the optical flow mask, including: The optical flow mask is generated based on the target text, wherein the target text is located at a preset text position in the optical flow mask; Acquire consecutive first, second, and third initial video frames; According to the preset text position, the target text is added to the corresponding positions of the first initial video frame, the second initial video frame and the third initial video frame respectively to obtain the first video frame, the second video frame and the third video frame.

3. The method of claim 2, wherein, Generating the optical flow mask map based on the target text includes: Obtain an initial mask image, which includes multiple pixels, each pixel corresponding to a mask value, and the mask values ​​corresponding to the multiple pixels are all zero values; Based on the target text and the preset text position, determine multiple text pixels from the multiple pixels; The mask values ​​corresponding to the multiple text pixels are adjusted from the zero value to a preset value to obtain the optical flow mask map.

4. The method according to claim 3, wherein the frame interpolation model includes an optical flow prediction module; Using the first video frame, the third video frame, and the optical flow mask as model inputs, the interpolation model outputs predicted video frames, including: Based on the first video frame and the third video frame, the optical flow prediction module generates a first optical flow map and a second optical flow map; wherein, the first optical flow map is used to describe the optical flow result between the first video frame and the predicted video frame, and the second optical flow map is used to describe the optical flow result between the third video frame and the predicted video frame. Based on the first optical flow map, the second optical flow map, and the optical flow mask map, a first shielded optical flow map and a second shielded optical flow map are determined; wherein, the first shielded optical flow map corresponds to the first optical flow map, and the second shielded optical flow map corresponds to the second optical flow map; The predicted video frame is generated using the frame interpolation model based on the first and second shielded optical flow maps.

5. The method of claim 4, wherein, The first optical flow map includes a first target pixel, the optical flow mask map includes a second target pixel corresponding to the first target pixel, and the first masking optical flow map includes a third target pixel corresponding to the first target pixel; Determining the first shielded optical flow map and the second shielded optical flow map based on the first optical flow map, the second optical flow map, and the optical flow mask map includes: Determine the target mask value of the second target pixel and the optical flow value of the first target pixel; Determine the difference between the preset value and the target mask value; The optical flow value of the third target pixel is determined based on the difference and the optical flow value of the first target pixel.

6. The method of claim 4, wherein, The first video frame, the predicted video frame, and the third video frame all include target pixels corresponding to the same element; Based on the first and second shielded optical flow maps, the predicted video frames are generated using the frame interpolation model, including: Based on the first video frame and the second video frame, a target weight value for the target pixel is determined. The target weight value is used to describe the correlation between the position of the target pixel in the predicted video frame and the optical flow value of the target pixel in the first video frame. Based on the first shielding optical flow map, the first optical flow value of the target pixel in the first video frame is determined, and based on the first optical flow value and the target weight value, the first prediction result is determined; Based on the second shielded optical flow map, the second optical flow value of the target pixel in the third video frame is determined, and based on the second optical flow value and the supplementary weight value, the second prediction result is determined, wherein the sum of the target weight value and the supplementary weight value is 1; Based on the first prediction result and the second prediction result, the position of the target pixel in the predicted video frame is determined.

7. A method of stuffing frames, characterized by The method includes: Acquire adjacent first target video frames and second target video frames; The first target video frame and the second target video frame are used as model inputs, and the target predicted video frame is output through a frame interpolation model; the frame interpolation model is trained using the model training method of any one of claims 1 to 6; and The target predicted video frame is inserted between the first target video frame and the second target video frame.

8. A model training apparatus, comprising: The device includes: The acquisition module is used to acquire a first video frame, a second video frame, a third video frame, and an optical flow mask; wherein the first video frame, the second video frame, and the third video frame are sequentially distributed consecutive video frames and each has target text, and the optical flow mask is used to indicate the position of the target text on the frame. The output module is used to take the first video frame, the third video frame and the optical flow mask as model inputs, and output the predicted video frame through the frame interpolation model; An adjustment module is used to adjust the frame interpolation model based on the predicted video frame and the second video frame.

9. A frame interpolation device, characterized in that, The device includes: The acquisition module is used to acquire adjacent first target video frames and second target video frames. The output module is configured to take the first target video frame and the second target video frame as model inputs, and output a target predicted video frame through a frame interpolation model; the frame interpolation model is trained using the model training method of any one of claims 1 to 6; and An insertion module is used to insert the target predicted video frame between the first target video frame and the second target video frame.

10. A computer device, characterized in that, include: At least one processor; and A memory communicatively connected to the at least one processor; wherein: The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 7.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method as described in any one of claims 1 to 7.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method described in claims 1 to 7.

Images