Video processing method and device, equipment and storage medium

Abstract

Embodiments of the present application provide a video processing method, device and equipment, and a storage medium, which are applied to at least the field of artificial intelligence and the field of video, and the method comprises the following steps: taking an initial frame and an ending frame of a target video as a prediction starting frame respectively, determining a forward prediction frame sequence corresponding to the initial frame and a reverse prediction frame sequence corresponding to the ending frame by using an autoregressive prediction mode; performing prediction frame fusion processing on the forward prediction frame sequence and the reverse prediction frame sequence according to a timestamp corresponding to each forward prediction frame in the forward prediction frame sequence and a timestamp corresponding to each reverse prediction frame in the reverse prediction frame sequence, to obtain a prediction frame sequence; and performing video frame splicing processing on the initial frame, the prediction frame sequence and the ending frame, to obtain the target video. Through the present application, the target frame deviation problem in the key frame interpolation task can be solved, the animation quality of the generated target video can be improved, the workload of video post-processing can be reduced, and the computing resources can be greatly saved.

Description

Technical Field

[0001] This application relates to the field of Internet technology, and includes, but is not limited to, a video processing method, apparatus, device, and storage medium. Background Technology

[0002] Keyframe interpolation (in-betweening) is widely used in film production and animation video fields such as video games. However, manual interpolation is extremely time-consuming, significantly increasing the time cost of animation video production. Some data-driven methods based on deep learning can effectively replace manual interpolation, thereby significantly saving human resources and accelerating the animation video production process.

[0003] In related technologies, deep learning-based methods are affected by error accumulation, and the generated transition animation will gradually deviate from the target frame. The deviation problem is usually solved by post-processing, such as mixing the last frame or several frames of the generated transition animation with the provided target frame or several frames after the target frame.

[0004] However, the methods in related technologies, after mixing the video frames of the animation, still result in a deviation from the target frame because the target frame has changed. Furthermore, these technologies cannot guarantee that the mixed result will be a natural animated video. The mixed result may have animation quality issues such as slippage and drift, which require additional post-processing to resolve. Summary of the Invention

[0005] This application provides a video processing method, apparatus, device, and storage medium, which are applied to at least the fields of artificial intelligence and video. It can solve the target frame deviation problem in keyframe interpolation tasks, improve the animation quality of the generated target video, reduce the workload of video post-processing, and greatly save computing resources.

[0006] The technical solution of this application embodiment is implemented as follows:

[0007] This application provides a video processing method, the method comprising:

[0008] Obtain the initial frame and the end frame of the target video; using the initial frame and the end frame as prediction start frames respectively, determine the forward prediction frame sequence corresponding to the initial frame and the backward prediction frame sequence corresponding to the end frame using an autoregressive prediction method; perform prediction frame fusion processing on the forward prediction frame sequence and the backward prediction frame sequence according to the timestamp corresponding to each forward prediction frame in the forward prediction frame sequence and the timestamp corresponding to each backward prediction frame in the backward prediction frame sequence to obtain the prediction frame sequence; perform video frame stitching processing on the initial frame, the prediction frame sequence, and the end frame to obtain the target video.

[0009] This application provides a video processing apparatus, comprising: an acquisition module for acquiring an initial frame and an end frame of a target video; a determination module for determining a forward prediction frame sequence corresponding to the initial frame and a backward prediction frame sequence corresponding to the end frame using an autoregressive prediction method, with the initial frame and the end frame as prediction start frames respectively; a prediction frame fusion module for performing prediction frame fusion processing on the forward prediction frame sequence and the backward prediction frame sequence according to the timestamp corresponding to each forward prediction frame in the forward prediction frame sequence and the timestamp corresponding to each backward prediction frame in the backward prediction frame sequence, to obtain a prediction frame sequence; and a stitching module for performing video frame stitching processing on the initial frame, the prediction frame sequence, and the end frame to obtain the target video.

[0010] In some embodiments, the determining module is further configured to: for each current frame, obtain the role state of the current frame, the role state of the initial frame, and the role state of the end frame; encode the role state of the current frame, the role state of the initial frame, and the role state of the end frame using an encoder to obtain an encoding vector; and decode the encoding vector using a decoder to obtain the next forward prediction frame and the next reverse prediction frame of the current frame.

[0011] In some embodiments, the determining module is further configured to: when predicting the next forward prediction frame of the current frame, determine the role state offset between the current frame and the end frame, and encode the role state of the current frame, the role state of the end frame, and the role state offset using an encoder to obtain a first encoding vector; when predicting the next reverse prediction frame of the current frame, determine the role state offset between the current frame and the initial frame, and encode the role state of the current frame, the role state of the initial frame, and the role state offset using an encoder to obtain a second encoding vector; extract data distribution space from the first encoding vector and the second encoding vector respectively to obtain a first extraction vector and a second extraction vector; and decode the first extraction vector and the second extraction vector respectively using the decoder to obtain the next forward prediction frame and the next reverse prediction frame of the current frame.

[0012] In some embodiments, the determining module is further configured to: extract the current frame data distribution space and the end frame data distribution space from the first encoding vector respectively, to obtain the current frame data distribution and the end frame data distribution accordingly; perform linear interpolation on the current frame data distribution and the end frame data distribution to obtain the first extracted vector; and perform the current frame data distribution space and the initial frame data distribution space from the second encoding vector respectively, to obtain the current frame data distribution and the initial frame data distribution accordingly; and perform linear interpolation on the current frame data distribution and the initial frame data distribution to obtain the second extracted vector.

[0013] In some embodiments, the apparatus further includes: a coefficient determination module, configured to determine linear interpolation coefficients based on the position of the current frame in the target video; the determination module is further configured to: perform linear interpolation processing on the current frame data distribution and the end frame data distribution based on the linear interpolation coefficients to obtain the first extraction vector; and perform linear interpolation processing on the current frame data distribution and the initial frame data distribution based on the linear interpolation coefficients to obtain the second extraction vector.

[0014] In some embodiments, the determining module is further configured to: determine each forward prediction frame in the forward prediction frame sequence sequentially using the initial frame as the prediction start frame via a forward generator; and determine each reverse prediction frame in the reverse prediction frame sequence sequentially using the end frame as the prediction start frame via a reverse generator; wherein the forward generator and the reverse generator operate alternately, and each time the reverse generator predicts a reverse prediction frame, it inputs the reverse prediction frame as condition information of the forward generator into the forward generator to predict the forward prediction frame; and each time the forward generator predicts a forward prediction frame, it inputs the forward prediction frame as condition information of the reverse generator into the reverse generator to predict the reverse prediction frame.

[0015] In some embodiments, the decoder includes multiple expert networks and a gating network; the apparatus further includes: a phase feature acquisition module for acquiring phase features of the target video; an input module for inputting the phase features into the gating network to obtain multiple mixing coefficients; and a linear mixing module for linearly mixing the multiple expert networks based on the multiple mixing coefficients to obtain the decoder.

[0016] In some embodiments, the prediction frame fusion module is further configured to: determine the forward prediction frames and reverse prediction frames with the same timestamp in the forward prediction frame sequence and the reverse prediction frame sequence as a pair of bidirectional prediction frames; and sequentially perform prediction frame fusion processing on each pair of bidirectional prediction frames corresponding to the forward prediction frame sequence and the reverse prediction frame sequence to obtain the prediction frame sequence.

[0017] In some embodiments, the prediction frame fusion module is further configured to: determine the forward remaining frames in the forward prediction frame sequence that have not undergone the prediction frame mixing process, and the reverse remaining frames in the reverse prediction frame sequence that have not undergone the prediction frame mixing process; determine the timestamps corresponding to the forward remaining frames and the reverse remaining frames respectively; sequentially perform prediction frame mixing process on each bidirectional prediction frame pair corresponding to the forward prediction frame sequence and the reverse prediction frame sequence to obtain a mixed processing frame; determine the timestamps of the forward prediction frames and the reverse prediction frames in the bidirectional prediction frame pair as the timestamps of the corresponding mixed processing frames; and perform splicing processing on the forward remaining frames, the mixed processing frames, and the reverse remaining frames according to the order of the timestamps to obtain the prediction frame sequence.

[0018] In some embodiments, the forward prediction frame includes the first position coordinates of multiple key points of the target object, and the reverse prediction frame includes the second position coordinates of multiple key points of the target object; the prediction frame fusion module is further configured to: determine the corresponding forward prediction frame fusion weight and the corresponding reverse prediction frame fusion weight according to the positions of the forward prediction frame and the reverse prediction frame in the target video; based on the fusion weight of the forward prediction frame and the fusion weight of the reverse prediction frame, perform prediction frame fusion processing on the first position coordinates and the second position coordinates of the same key points in each pair of bidirectional prediction frames in sequence to obtain the fused frame.

[0019] In some embodiments, the number of forward prediction frames in the forward prediction frame sequence is the same as the number of backward prediction frames in the backward prediction frame sequence; and the time interval between each two adjacent forward prediction frames in the target video in the forward prediction frame sequence is the same as the time interval between each two adjacent backward prediction frames in the target video in the backward prediction frame sequence.

[0020] In some embodiments, in the forward prediction frame sequence, the time interval between the first and last forward prediction frames in the target video is less than the time interval between the initial and final frames in the target video, and greater than half of the time interval between the initial and final frames in the target video; in the reverse prediction frame sequence, the time interval between the first and last reverse prediction frames in the target video is less than the time interval between the initial and final frames in the target video, and greater than half of the time interval between the initial and final frames in the target video.

[0021] This application provides a video processing device, including:

[0022] The memory is used to store executable instructions; the processor is used to implement the above-mentioned video processing method when executing the executable instructions stored in the memory.

[0023] This application provides a computer program product or computer program, which includes executable instructions stored in a computer-readable storage medium; wherein, when the processor of the video processing device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, it implements the above-described video processing method.

[0024] This application provides a computer-readable storage medium storing executable instructions, which, when executed by a processor, implement the aforementioned video processing method.

[0025] The embodiments of this application have the following beneficial effects: Using the initial frame and the end frame as prediction start frames respectively, an autoregressive prediction method is employed to determine the forward prediction frame sequence corresponding to the initial frame and the backward prediction frame sequence corresponding to the end frame. Then, prediction frame fusion processing is performed on the forward and backward prediction frame sequences to obtain the prediction frame sequence. Finally, video frame stitching processing is performed on the initial frame, the prediction frame sequence, and the end frame to obtain the target video. Thus, by using an autoregressive prediction method to determine the forward and backward prediction frame sequences from two directions respectively, and by fusing the forward and backward prediction frame sequences, the target frame deviation problem in keyframe interpolation tasks can be solved. Furthermore, the animation quality of the generated target video can be improved, problems such as target object slippage and drift in the generated target video can be alleviated, the workload of video post-processing can be reduced, and computational resources can be greatly saved. Attached Figure Description

[0026] Figures 1A to 1C This is a schematic diagram illustrating the use of the RMIB method to generate animated videos in related technologies;

[0027] Figure 2 It is the result of generating six samples at the same time under the same constraints using the RMIB method;

[0028] Figure 3 This is an optional architecture diagram of the video processing system provided in the embodiments of this application;

[0029] Figure 4 This is a schematic diagram of the structure of the video processing device provided in the embodiments of this application;

[0030] Figure 5 This is an optional flowchart illustrating the video processing method provided in an embodiment of this application;

[0031] Figure 6 This is another optional flowchart illustrating the video processing method provided in the embodiments of this application;

[0032] Figure 7 This is another optional flowchart illustrating the video processing method provided in the embodiments of this application;

[0033] Figure 8 This is a schematic diagram illustrating the effect of the video processing method according to an embodiment of this application;

[0034] Figure 9 This is a schematic diagram of the generation results obtained by generating the same data three times under the same conditions using the method described in the embodiments of this application;

[0035] Figure 10 This is a flowchart of the bidirectional generation mechanism provided in the embodiments of this application;

[0036] Figure 11 This is a schematic diagram of the structure of the CVAE network provided in the embodiments of this application;

[0037] Figure 12 This is a schematic diagram of the encoder structure of the S-CVAE provided in the embodiments of this application;

[0038] Figure 13 This is a schematic diagram of bidirectional alignment provided in an embodiment of this application;

[0039] Figures 14A to 14C This is a schematic diagram of the generation result of the transition animation provided in the embodiments of this application;

[0040] Figure 15 This is a schematic diagram of the interpolation results of multiple consecutive keyframes provided in an embodiment of this application;

[0041] Figure 16 This is a comparison chart of six predicted values ​​and their corresponding actual values ​​generated under the same conditions. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0043] In the following description, references to "some embodiments" refer to a subset of all possible embodiments. However, it is understood that "some embodiments" may be the same or different subsets of all possible embodiments and may be combined with each other without conflict. Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiments of this application pertain. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit the application.

[0044] Before explaining the video processing method of the embodiments of this application, the methods in related technologies will be described first.

[0045] In related technologies, keyframe interpolation is widely used in film production and video games. However, manual interpolation is extremely time-consuming, especially as the length of the transition animation sequence increases, exponentially increasing the time cost of producing animated videos (i.e., target videos). Some data-driven methods based on deep learning can effectively replace manual interpolation, significantly saving manpower and accelerating the animation video production process. Furthermore, the longer the transition animation sequence, the more pronounced the advantages of learning-based methods become. This is because the longer the sequence, the more time-consuming manual interpolation becomes. Longer sequences contain more information, placing higher demands on the diversity of interpolation results and significantly increasing the difficulty of manual interpolation.

[0046] Data-driven methods based on deep learning can effectively solve the above problems, but they also face new challenges. For example, prediction methods using RMIB in related technologies suffer from error accumulation, causing the generated segments to gradually drift away from the target frame. Figures 1A to 1C This is a schematic diagram illustrating the generation of animated video using the RMIB method in related technologies. The thin solid line represents the tail frame 101 generated by the RMIB method, and the thick solid line represents the given target frame 102. Figures 1A to 1C The alignment of the tail frame 101 generated by the RMIB method with the given target frame 102 shows that the generated video clip (i.e., the transition animation) is far from the target frame. Related techniques often employ post-processing to address this deviation. Specifically, the last frame or several frames of the generated transition animation are blended with the provided target frame or several frames following the target frame. This approach can lead to two problems: 1) After blending, if the specified target frame changes, the blended result will still deviate from the target frame; 2) The blended result cannot be guaranteed to be a natural animation, and may contain imperfections such as slippage and drift, requiring additional post-processing to resolve.

[0047] Furthermore, the results generated by RMIB and Transformer-based methods lack diversity, such as... Figure 2 As shown, where, Figure 2 These are the results of six samples generated at the same time under the same constraints using the RMIB method. It can be seen that the results of the six samples generated at the same time under the same constraints using the RMIB method are very similar, therefore, the generated results are very homogeneous and lack diversity. However, the transition animation results between the initial frame and the target frame usually have multiple possibilities, and a single generated result is unlikely to meet the needs of animators.

[0048] To address the problems existing in related technologies, this application provides a video processing method, which is a bidirectional keyframe interpolation scheme that can obtain diverse results. This application ensures that the generated transition animation perfectly matches the target frame, while also ensuring a certain degree of diversity in the generated results; that is, under the same conditions, the results generated each time are different. In other words, this application proposes a bidirectional keyframe interpolation scheme that can obtain diverse results, effectively solving the problems of diversity and target frame deviation in keyframe interpolation tasks. Furthermore, this application introduces phase features into the keyframe interpolation task for the first time. Phase can alleviate the slippage problem in the generated transition animation and improve animation quality.

[0049] In the video processing method provided in this application embodiment, firstly, the initial frame and the end frame of the target video are obtained; then, using the initial frame and the end frame as prediction start frames respectively, an autoregressive prediction method is adopted to determine the forward prediction frame sequence corresponding to the initial frame and the backward prediction frame sequence corresponding to the end frame; then, according to the timestamp corresponding to each forward prediction frame in the forward prediction frame sequence and the timestamp corresponding to each backward prediction frame in the backward prediction frame sequence, prediction frame fusion processing is performed on the forward prediction frame sequence and the backward prediction frame sequence to obtain the prediction frame sequence; finally, video frame stitching processing is performed on the initial frame, the prediction frame sequence, and the end frame to obtain the target video. Thus, by using an autoregressive prediction method to determine the forward prediction frame sequence and the backward prediction frame sequence from two directions respectively, and by fusing the forward prediction frame sequence and the backward prediction frame sequence, the target frame deviation problem in the keyframe interpolation task can be solved, and the animation quality of the generated target video can be improved, mitigating problems such as slippage and drift of the target object in the generated target video, reducing the workload of video post-processing, and greatly saving computing resources.

[0050] The following describes exemplary applications of the video processing device according to embodiments of this application. The video processing device provided in this application can be implemented as a terminal or a server. In one implementation, the video processing device provided in this application can be implemented as any terminal with video display capabilities, such as a laptop, tablet, desktop computer, mobile device (e.g., mobile phone, portable music player, personal digital assistant, dedicated messaging device, portable gaming device), smart robot, smart home appliance, and smart vehicle device, capable of processing given initial and ending frames to obtain a target video. In another implementation, the video processing device provided in this application can also be implemented as a server. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in this application embodiment. The following will describe an exemplary application when the video processing device is implemented as a server.

[0051] See Figure 3 , Figure 3 This is an optional architecture diagram of the video processing system provided in this application embodiment. This application embodiment uses the application of video processing methods to animation video production as an example for illustration. That is, the final generated target video can be an animation video. To support any animation video production application, the terminal in this application embodiment has at least one animation video production application installed. This animation video production application can automatically generate a transition animation between the initial frame and the end frame given the initial frame and the end frame of the animation video. This transition animation is the predicted frame sequence. Finally, the transition animation can be spliced ​​with the initial frame and the end frame respectively to generate a complete animation video.

[0052] In this embodiment, the video processing system 10 includes at least a terminal 100, a network 200, and a server 300, wherein the server 300 is a server for an animation video production application. The server 300 can constitute the video processing device of this embodiment. The terminal 100 connects to the server 300 through the network 200, which can be a wide area network (WAN), a local area network (LAN), or a combination of both. When running the animation video production application, the terminal 100 obtains the initial and final frames of the target video through the client of the animation video production application, generates a target video generation request with the initial and final frames, and sends the target video generation request to the server 300. Server 300 parses the target video generation request to obtain the initial frame and the end frame. Using the initial frame and the end frame as prediction start frames respectively, it employs an autoregressive prediction method to determine the forward prediction frame sequence corresponding to the initial frame and the backward prediction frame sequence corresponding to the end frame. Based on the timestamps corresponding to each forward prediction frame in the forward prediction frame sequence and each backward prediction frame in the backward prediction frame sequence, it performs prediction frame fusion processing on the forward prediction frame sequence and the backward prediction frame sequence to obtain the prediction frame sequence. Finally, it performs video frame stitching processing on the initial frame, the prediction frame sequence, and the end frame to obtain the target video. After obtaining the target video, server 300 sends it to terminal 100 via network 200, and terminal 100 displays the target video on the client.

[0053] In some embodiments, terminal 100 can obtain the initial and final frames of the target video through the client of an animation video production application, and send the initial and final frames to server 300. Server 300 uses the initial and final frames as prediction start frames, respectively, and employs an autoregressive prediction method to determine the forward prediction frame sequence corresponding to the initial frame and the backward prediction frame sequence corresponding to the final frame. Based on the timestamps corresponding to each forward prediction frame in the forward prediction frame sequence and each backward prediction frame in the backward prediction frame sequence, prediction frame fusion processing is performed on the forward and backward prediction frame sequences to obtain a prediction frame sequence. After obtaining the prediction frame sequence, server 300 sends the prediction frame sequence to terminal 100 via network 200. Terminal 100 performs video frame stitching processing on the initial frame, prediction frame sequence, and final frame to obtain and display the target video.

[0054] In some embodiments, the video processing method can also be implemented by the terminal 100. That is, the terminal, as the execution subject, uses the initial frame and the end frame as prediction start frames respectively, and adopts an autoregressive prediction method to determine the forward prediction frame sequence corresponding to the initial frame and the reverse prediction frame sequence corresponding to the end frame; and performs prediction frame fusion processing on the forward prediction frame sequence and the reverse prediction frame sequence according to the timestamp corresponding to each forward prediction frame in the forward prediction frame sequence and the timestamp corresponding to each reverse prediction frame in the reverse prediction frame sequence to obtain the prediction frame sequence; and performs video frame stitching processing on the initial frame, the prediction frame sequence and the end frame to obtain the target video.

[0055] The video processing method provided in this application embodiment can also be implemented based on a cloud platform and through cloud technology. For example, the server 300 mentioned above can be a cloud server. The cloud server uses an autoregressive prediction method to determine the forward prediction frame sequence corresponding to the initial frame and the backward prediction frame sequence corresponding to the end frame. Alternatively, the cloud server performs prediction frame fusion processing on the forward and backward prediction frame sequences to obtain a prediction frame sequence. Or, the cloud server performs video frame stitching processing on the initial frame, the prediction frame sequence, and the end frame to obtain the target video, etc.

[0056] In some embodiments, a cloud storage system may also be included, where the initial and final frames of the target video can be stored. Alternatively, the forward prediction frame sequence, the backward prediction frame sequence, and the prediction frame sequence may also be stored in the cloud storage, or the target video itself may be stored in the cloud storage. Thus, when the initial and final frames are input to request prediction of the target video, the forward and backward prediction frame sequences, or the prediction frame sequence, can be retrieved from the cloud storage to obtain the target video.

[0057] It's important to clarify that cloud technology refers to a hosting technology that unifies hardware, software, and network resources within a wide area network (WAN) or local area network (LAN) to achieve data computation, storage, processing, and sharing. Cloud technology is a collective term for network technologies, information technologies, integration technologies, management platform technologies, and application technologies applied in the cloud computing business model. It can form resource pools, providing flexible and convenient on-demand access. Cloud computing technology will become a crucial support. Backend services of technical network systems require substantial computing and storage resources, such as video websites, image websites, and many portal websites. With the rapid development and application of the internet industry, every item may have its own identification mark in the future, requiring transmission to backend systems for logical processing. Data at different levels will be processed separately, and various industry data will require robust system support, which can only be achieved through cloud computing.

[0058] Figure 4 This is a schematic diagram of the structure of the video processing device provided in the embodiments of this application. Figure 4 The video processing device shown includes at least one processor 310, a memory 350, at least one network interface 320, and a user interface 330. The various components in the video processing device are coupled together via a bus system 340. It is understood that the bus system 340 is used to implement communication between these components. In addition to a data bus, the bus system 340 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 4 The general labeled all buses as Bus System 340.

[0059] The processor 310 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0060] User interface 330 includes one or more output devices 331 that enable the presentation of media content, and one or more input devices 332.

[0061] Memory 350 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. Memory 350 may optionally include one or more storage devices physically located remote from processor 310. Memory 350 may include volatile memory or non-volatile memory, or both. Non-volatile memory may be read-only memory (ROM), and volatile memory may be random access memory (RAM). The memory 350 described in this application embodiment is intended to include any suitable type of memory. In some embodiments, memory 350 is capable of storing data to support various operations, examples of which include programs, modules, and data structures, or subsets or supersets thereof, as illustrated below.

[0062] Operating system 351 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0063] The network communication module 352 is used to reach other computing devices via one or more (wired or wireless) network interfaces 320, exemplary network interfaces 320 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0064] The input processing module 353 is used to detect and translate one or more user inputs or interactions from one or more input devices 332.

[0065] In some embodiments, the apparatus provided in this application may be implemented in software. Figure 4 A video processing apparatus 354 stored in memory 350 is shown. This video processing apparatus 354 can be a video processing device within a video processing equipment, and can be software in the form of programs and plugins. It includes the following software modules: an acquisition module 3541, a determination module 3542, a prediction frame fusion module 3543, and a stitching module 3544. These modules are logically linked and can therefore be arbitrarily combined or further separated according to their implemented functions. The functions of each module will be described below.

[0066] In other embodiments, the apparatus provided in this application can be implemented in hardware. As an example, the apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the video processing method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0067] The video processing methods provided in the embodiments of this application can be executed by a video processing device. The video processing device can be any terminal with video display function that can process a given initial frame and an end frame to obtain a target video, or it can be a server. That is, the video processing methods in the embodiments of this application can be executed by a terminal, by a server, or by an interaction between a terminal and a server.

[0068] See Figure 5 , Figure 5 This is an optional flowchart illustrating the video processing method provided in the embodiments of this application. The following will be combined with... Figure 5 The steps shown will be explained. It should be noted that... Figure 5 The video processing method in this example is illustrated by using a server as the execution entity.

[0069] Step S501: Obtain the initial frame and the end frame of the target video.

[0070] Here, the target video refers to the video to be generated. In this embodiment, the target video can be an animated video or a skeletal animation of the target object, such as a human skeletal animation, an animal skeletal animation, etc. In the skeletal animation, there may be no background image or background information, only the skeleton of the target object, and this skeleton is a three-dimensional skeleton.

[0071] The initial frame refers to the first frame of the target video, and the ending frame refers to the last frame of the target video. The solution in this application is to predict multiple consecutive video frames between the initial and ending frames.

[0072] This application can be applied to animation video production. To create an animation video clip, or to correct and update video frames within an animation video clip, an initial frame and an end frame can be provided, and a sequence of predicted frames can be predicted based on these given initial and end frames. When selecting the initial and end frames, for an animation video of a certain duration, multiple pairs of video frames corresponding to the animation video clips can be selected based on a certain time interval. Each pair of video frames includes an initial frame and an end frame. In two adjacent pairs of video frames, the end frame of the previous pair can be the initial frame of the next pair. After selecting multiple pairs of video frames based on the time interval, a sequence of predicted frames between the initial and end frames in each pair is predicted based on these multiple pairs of video frames.

[0073] Step S502: Using the initial frame and the end frame as prediction start frames respectively, an autoregressive prediction method is used to determine the forward prediction frame sequence corresponding to the initial frame and the reverse prediction frame sequence corresponding to the end frame.

[0074] Here, the initial frame can be used as the prediction start frame to predict the forward prediction frame sequence corresponding to the initial frame, and the end frame can be used as the prediction start frame to predict the backward prediction frame sequence corresponding to the end frame.

[0075] An autoregressive prediction method can be used to predict each forward prediction frame in the forward prediction frame sequence and each backward prediction frame in the backward prediction frame sequence. Here, the autoregressive prediction method means that when predicting each forward prediction frame in the forward prediction frame sequence, for the current prediction process, the input parameter is the previously predicted forward prediction frame that is adjacent to the currently predicted forward prediction frame. Similarly, when predicting each backward prediction frame in the backward prediction frame sequence, the input parameter is the previously predicted backward prediction frame that is adjacent to the currently predicted backward prediction frame. In other words, in the autoregressive prediction method, the data of the current frame is used to predict the data of the next frame, and then the data of the next frame is used as input to predict the data of the frame after that, and so on in a cyclical manner.

[0076] In some embodiments, an autoregressive model can be pre-built and integrated into the generator to predict each forward prediction frame and each backward prediction frame.

[0077] A forward prediction frame sequence includes multiple forward prediction frames, each corresponding to a timestamp. The initial and final frames correspond to the start and end times of the target video. In the forward video frame sequence, the initial frame of the target video is located before the initial position of the forward video frame sequence, and the final frame of the target video is located after the end position of the forward video frame sequence. The duration of the forward prediction frame sequence is less than or equal to the duration of the target video. For example, the duration of the target video can be 10 seconds (s), and the forward video frame sequence can include video frames at times 1s, 2s, 3s, 4s, 5s, 6s, and 7s, with the initial frame being the video frame at time 0 and the final frame being the video frame at time 10.

[0078] The reverse prediction frame sequence includes multiple reverse prediction frames, each corresponding to a timestamp. When predicting the reverse video frame sequence, the ending frame is used as the starting frame, resulting in multiple reverse video frames. At this point, the positions of the initial and ending frames of the target video in the reverse video frame sequence are exactly the opposite of their positions in the forward video frame sequence; that is, the initial frame of the target video is after the end position of the reverse video frame sequence, and the ending frame of the target video is before the initial position of the reverse video frame sequence. The duration of the reverse prediction frame sequence is also less than or equal to the duration of the target video. For example, if the duration of the target video is 10 seconds, the reverse video frame sequence could include video frames at times 9s, 8s, 7s, 6s, 5s, 4s, and 3s.

[0079] In some embodiments, since the forward and reverse video frame sequences are generated simultaneously, and the rates for predicting forward and reverse video frames are the same, the number of forward video frames in the forward video frame sequence is the same as the number of reverse video frames in the reverse video frame sequence. In the forward prediction frame sequence, the time interval between any two adjacent forward prediction frames in the target video is the same as the time interval between any two adjacent reverse prediction frames in the target video in the reverse prediction frame sequence. That is, the prediction frame rate of the forward video frame sequence is the same as the prediction frame rate of the reverse video frame sequence.

[0080] Step S503: According to the timestamp corresponding to each forward prediction frame in the forward prediction frame sequence and the timestamp corresponding to each reverse prediction frame in the reverse prediction frame sequence, perform prediction frame fusion processing on the forward prediction frame sequence and the reverse prediction frame sequence to obtain the prediction frame sequence.

[0081] Here, the prediction frame fusion process can involve mixing forward prediction frames and reverse video frames with the same timestamp to obtain a single, mixed prediction frame. This mixed prediction frame has the same timestamp as the corresponding forward and reverse prediction frames. After mixing all forward and reverse prediction frames with the same timestamp, all the mixed prediction frames are sorted and concatenated according to their timestamp order to obtain a prediction frame sequence.

[0082] In this embodiment, since the duration of the forward video frame sequence is shorter than that of the target video, and the duration of the reverse video frame sequence is also shorter than that of the target video, and the duration of the forward video frame sequence is the same as that of the reverse video frame sequence, it is not necessary to perform mixed processing on the video frames at every moment between the initial frame and the end frame when performing prediction frame fusion processing. This can greatly reduce the amount of data processed in the video and improve the generation efficiency of the prediction frame sequence.

[0083] In some embodiments, for forward and backward prediction frames that do not have the same timestamp in the forward and backward prediction frame sequences, they can be directly spliced ​​into the prediction frame sequence as prediction frames without mixing.

[0084] Step S504: Perform video frame stitching on the initial frame, the predicted frame sequence, and the end frame to obtain the target video.

[0085] In this embodiment, after obtaining the predicted frame sequence, the initial frame can be stitched to the beginning of the predicted frame sequence, and the end frame can be stitched to the end of the predicted frame sequence to form a continuous target video. Since the predicted frame sequence is based on the initial and end frames, the difference between the actions of the target object in the first few predicted frames and the initial frame is less than a difference threshold, and the difference between the actions of the target object in the last few predicted frames and the end frame is also less than a difference threshold. This makes the predicted frame sequence not only more seamless and continuous with the actions of the target object in the initial frame, but also more seamless and continuous with the actions of the target object in the end frame.

[0086] It should be noted that, in the embodiments of this application, since the prediction is performed on an animated video, and the animated video includes skeletal animation of a human or animal, when predicting the forward prediction frame and the backward prediction frame, the coordinates of the key points of each key position (e.g., joint position) of the human or animal in the forward prediction frame and the backward prediction frame can be predicted in the world coordinate system. Then, based on the predicted coordinates, the key points are connected in sequence to obtain the prediction frame corresponding to the skeletal animation.

[0087] The video processing method provided in this application uses an initial frame and an end frame as prediction start frames, respectively. An autoregressive prediction method is employed to determine a forward prediction frame sequence corresponding to the initial frame and a backward prediction frame sequence corresponding to the end frame. Then, prediction frame fusion processing is performed on the forward and backward prediction frame sequences to obtain a prediction frame sequence. Finally, video frame stitching processing is performed on the initial frame, the prediction frame sequence, and the end frame to obtain the target video. Thus, by using an autoregressive prediction method to determine the forward and backward prediction frame sequences from two directions, and fusing these sequences, the target frame deviation problem in keyframe interpolation tasks can be solved. Furthermore, the animation quality of the generated target video can be improved, problems such as target object slippage and drift in the generated target video can be alleviated, the workload of video post-processing can be reduced, and computational resources can be greatly saved.

[0088] In some embodiments, the video processing system includes at least a terminal and a server. The video processing method of this application embodiment can be applied to the generation of animated videos. An animated video creation application is installed on the terminal. By inputting the initial and ending frames of the animated video to be generated (i.e., the target video) through the client of the animated video creation application, a continuous animated video containing the initial and ending frames can be automatically generated.

[0089] Figure 6 This is another optional flowchart illustrating the video processing method provided in the embodiments of this application, such as... Figure 6 As shown, the method includes the following steps:

[0090] Step S601: The terminal acquires the initial frame and the end frame of the target video.

[0091] In some embodiments, users can input initial and final frames through the client of an animation video creation application. For example, they can draw initial and final frames on the client's input interface by using a pen, or they can click to select various key positions (e.g., joint positions of the human body) on the client's input interface, or they can input the coordinates of the key points corresponding to each key position in the world coordinate system.

[0092] In other embodiments, the initial and final frames of the target video can be video frames downloaded from the network or received by a client, with each key position already marked in the video frames. Alternatively, after downloading the video frames from the network or receiving them from the client, a user marking operation can be received to mark each key position in the initial and final frames. This ensures that subsequent predictions of the forward and backward prediction frame sequences can be accurately made based on the coordinates of the key points corresponding to the key positions.

[0093] In other embodiments, the video processing method can also be applied to update existing animated videos. Therefore, any two frames in the existing animated video can be extracted as the initial frame and the end frame, and the predicted frame sequence between the initial frame and the end frame can be re-predicted to generate a new animated video segment based on the new predicted frame sequence, thereby updating one or more animated video segments in the existing animated video.

[0094] In step S602, the terminal encapsulates the initial frame and the end frame into the target video generation request.

[0095] In step S603, the terminal sends the target video generation request to the server.

[0096] Step S604: For each current frame, the server obtains the character status of the current frame, the character status of the initial frame, and the character status of the end frame.

[0097] Here, the current frame refers to the video frame preceding the currently predicted video frame at any given moment during the prediction process of both the forward and backward prediction frame sequences. For example, when the N+1th forward video frame in the forward video frame sequence is being predicted, the Nth forward video frame is the current frame. If the currently predicted frame is the first forward video frame in the forward video frame sequence, then the current frame is the initial frame; if the currently predicted frame is the first backward video frame in the backward video frame sequence, then the current frame is the end frame.

[0098] In this embodiment, the character state includes, but is not limited to, at least one of the following: the target object's coordinates in the world coordinate system (e.g., x-coordinate, y-coordinate, z-coordinate), information about the target object's foot joints, information about whether the left and right feet are in contact with the ground, and the movement speed of the root node. The root node refers to the point corresponding to the center position of the target object; for example, when the target object is a human body, the root node could be a node at the human's hip position.

[0099] In step S605, the server calls the encoder to encode the character state of the current frame, the character state of the initial frame, and the character state of the end frame to obtain the encoded vector.

[0100] In some embodiments, encoding processing by an encoder can be implemented in the following way: when predicting the next forward prediction frame of the current frame, the role state offset between the current frame and the end frame can be determined first based on the role state of the current frame and the role state of the end frame. Then, the role state of the current frame, the role state of the end frame, and the role state offset are encoded by the encoder to obtain three encoded sub-vectors. Then, the three encoded sub-vectors are concatenated to obtain the first encoded vector, wherein the dimension of the first encoded vector is equal to the sum of the dimensions of the three encoded sub-vectors.

[0101] Here, the encoder can be the forward encoder in the forward generator. The forward encoder can be composed of three sub-encoders, which can be the current frame encoder, the target frame encoder, and the offset encoder, respectively. The current frame encoder is used to encode the character state of the current frame, the target frame encoder is used to encode the character state of the end frame, and the offset encoder is used to encode the character state offset.

[0102] When predicting the next reverse prediction frame of the current frame, the role state offset between the current frame and the initial frame can be determined first based on the role state of the current frame and the role state of the initial frame. Then, the role state of the current frame, the role state of the initial frame, and the role state offset are encoded by the encoder to obtain three encoded sub-vectors. Then, the three encoded sub-vectors are concatenated to obtain the second encoded vector, where the dimension of the second encoded vector is equal to the sum of the dimensions of the three encoded sub-vectors.

[0103] Here, the encoder can be the inverse encoder in the inverse generator, or the inverse encoder can be composed of three sub-encoders, which can be the current frame encoder, the target frame encoder, and the offset encoder, respectively. The current frame encoder is used to encode the character state of the current frame, the target frame encoder is used to encode the character state of the initial frame, and the offset encoder is used to encode the character state offset.

[0104] In step S606, the server calls the decoder to decode the encoded vector, thereby obtaining the next forward prediction frame and the next backward prediction frame of the current frame.

[0105] In some embodiments, decoding processing by a decoder can be achieved through the following steps S6061 and S6062 (not shown in the figure):

[0106] Step S6061: Before decoding, the data distribution space of the first encoding vector and the second encoding vector is extracted respectively to obtain the first extraction vector and the second extraction vector.

[0107] In this embodiment, the first encoding vector can be processed by extracting the data distribution space of the current frame and the data distribution space of the end frame, respectively, to obtain the data distribution of the current frame and the data distribution of the end frame. Then, linear interpolation is performed on the data distribution of the current frame and the data distribution of the end frame to obtain the first extracted vector. Similarly, the second encoding vector can be processed by extracting the data distribution space of the current frame and the data distribution space of the initial frame, respectively, to obtain the data distribution of the current frame and the data distribution of the initial frame. Then, linear interpolation is performed on the data distribution of the current frame and the data distribution of the initial frame to obtain the second extracted vector.

[0108] Here, extracting the current frame data distribution space from the first encoding vector means extracting the data distribution of the character's state in the current frame from the first encoding vector; extracting the end frame data distribution space from the first encoding vector means extracting the data distribution of the character's state in the end frame from the first encoding vector. Extracting the current frame data distribution space from the second encoding vector means extracting the data distribution of the character's state in the current frame from the second encoding vector; extracting the initial frame data distribution space from the second encoding vector means extracting the data distribution of the character's state in the initial frame from the second encoding vector.

[0109] Linear interpolation of the current frame data distribution and the end frame data distribution can be performed by weighted summation of the current frame data distribution and the end frame data distribution based on certain linear interpolation coefficients, resulting in a summation vector, which is the first extraction vector. Similarly, linear interpolation of the current frame data distribution and the initial frame data distribution can be performed by weighted summation of the current frame data distribution and the initial frame data distribution based on certain linear interpolation coefficients, resulting in a summation vector, which is the second extraction vector.

[0110] In some embodiments, linear interpolation coefficients can be determined based on the positions of the current frame and the end frame in the target video. That is, different linear interpolation systems exist for the current frame at different positions in the target video, and different linear interpolation coefficients can be used for the current frame and the end frame, with the sum of the linear interpolation coefficients for the current frame and the end frame being 1. Furthermore, linear interpolation processing can be performed on the data distribution of the current frame and the data distribution of the end frame based on the linear interpolation coefficients to obtain a first extraction vector. In implementation, this can be achieved by multiplying the data distribution of the current frame by its linear interpolation coefficient, multiplying the data distribution of the end frame by its linear interpolation coefficient, and then summing the two product vectors to obtain the first extraction vector.

[0111] In other embodiments, linear interpolation coefficients can be determined based on the positions of the current frame and the initial frame in the target video. That is, different linear interpolation systems exist for the current frame at different positions in the target video, and different linear interpolation coefficients can be used for the current frame and the initial frame, with the sum of the linear interpolation coefficients of the current frame and the initial frame being 1. Furthermore, linear interpolation processing can be performed on the data distribution of the current frame and the data distribution of the initial frame based on the linear interpolation coefficients to obtain a second extraction vector. In implementation, this can be achieved by multiplying the data distribution of the current frame by its linear interpolation coefficient, multiplying the data distribution of the initial frame by its linear interpolation coefficient, and then summing the two product vectors to obtain the second extraction vector.

[0112] Step S6062: The first extraction vector and the second extraction vector are decoded by the decoder to obtain the next forward prediction frame and the next reverse prediction frame of the current frame.

[0113] Here, when predicting the next forward prediction frame, the first extracted vector is decoded; when predicting the next backward prediction frame, the second extracted vector is decoded.

[0114] In step S607, the server performs prediction frame fusion processing on the forward prediction frame sequence and the reverse prediction frame sequence according to the timestamp corresponding to each forward prediction frame in the forward prediction frame sequence and the timestamp corresponding to each reverse prediction frame in the reverse prediction frame sequence to obtain the prediction frame sequence.

[0115] Here, the predicted frame fusion process can involve mixing forward predicted frames and reverse video frames with the same timestamp to obtain a single predicted frame, which has the same timestamp as the corresponding forward and reverse predicted frames. After mixing all forward and reverse predicted frames with the same timestamp, all the mixed predicted frames are sorted and concatenated according to their timestamps to obtain a predicted frame sequence. The implementation process of mixing forward and reverse predicted frames will be explained below.

[0116] In step S608, the server performs video frame stitching processing on the initial frame, the predicted frame sequence, and the end frame to obtain the target video.

[0117] Here, the initial frame can be stitched to the beginning of the predicted frame sequence, and the ending frame can be stitched to the end of the predicted frame sequence to form a continuous target video.

[0118] In step S609, the server sends the target video to the terminal.

[0119] In step S610, the terminal displays the target video on the current interface.

[0120] The video processing method provided in this application embodiment encodes the character state of the current frame, the character state of the initial frame, and the character state of the end frame by calling an encoder. During the encoding process, the encoding of each current frame is based on the result of the previously predicted frame. Thus, the prediction process takes into account the previous prediction results. This autoregressive approach makes the action between two adjacent prediction frames smoother, thereby greatly avoiding the difference between adjacent prediction frames in the forward and backward prediction frame sequences, improving the smoothness of the action in the forward and backward prediction frame sequences, and making the action of the prediction frame sequence generated after the subsequent prediction frame fusion process smoother as well. This avoids problems such as the sliding and drifting of the target object in the generated target video.

[0121] Figure 7 This is another optional flowchart illustrating the video processing method provided in the embodiments of this application, such as... Figure 7 As shown, the method includes the following steps:

[0122] In step S701, the terminal acquires the initial frame and the end frame of the target video.

[0123] In step S702, the terminal encapsulates the initial frame and the end frame into the target video generation request.

[0124] In step S703, the terminal sends the target video generation request to the server.

[0125] Step S704: Using the initial frame as the prediction start frame, the forward generator sequentially determines each forward prediction frame in the forward prediction frame sequence.

[0126] Here, the forward generator includes an encoder and a decoder. The encoder in the forward generator uses the initial frame as the prediction start frame and sequentially encodes the character state of each current frame and the character state of each end frame to obtain an encoded vector. In the implementation, the character state offset between the current and end frames can be determined first based on the character state of the current frame and the character state of the end frame. Then, the encoder encodes the character state of the current frame, the character state of the end frame, and the character state offset separately, obtaining three encoded sub-vectors. These three encoded sub-vectors are then concatenated to obtain the forward encoded vector. After obtaining the forward encoded vector, the decoder decodes the encoded vector to obtain the next forward prediction frame for the current frame.

[0127] Step S705: Using the end frame as the prediction start frame, the reverse generator sequentially determines each reverse prediction frame in the reverse prediction frame sequence.

[0128] Here, the inverse generator includes an encoder and a decoder. The encoder in the inverse generator uses the end frame as the prediction start frame and sequentially encodes the character state of each current frame and the character state of the initial frame to obtain an encoded vector. In the implementation, the character state offset between the current and initial frames can be determined first based on the character state of the current frame and the character state of the initial frame. Then, the encoder encodes the character state of the current frame, the character state of the initial frame, and the character state offset separately, obtaining three encoded sub-vectors. These three encoded sub-vectors are then concatenated to obtain the inverse encoded vector. After obtaining the inverse encoded vector, the decoder decodes the encoded vector to obtain the next inverse prediction frame for the current frame.

[0129] In this embodiment of the application, the forward generator and the backward generator can run alternately. When the backward generator predicts a backward prediction frame each time, it can use the predicted backward prediction frame as the condition information of the forward generator and input it into the forward generator to predict the next forward prediction frame. Similarly, when the forward generator predicts a forward prediction frame each time, it can also use the forward prediction frame as the condition information of the backward generator and input it into the backward generator to predict the next backward prediction frame.

[0130] In this embodiment, the forward generator and the reverse generator have the same structure. In some embodiments, the decoder includes multiple expert networks and a gating network. The decoder generation process is described below, and the decoder can be generated through the following steps S11 to S13 (not shown in the figure):

[0131] Step S11: Obtain the phase features of the target video.

[0132] Here, the phase feature can be the local motion phase, a concept in the field of motion generation, which is a periodic signal.

[0133] For example, the phase feature can be obtained as follows: For an image sequence, if the foot of a person in any image in the sequence touches the ground, the number is set to 1; if the foot does not touch the ground, the number is set to 0. This generates a number sequence consisting of 0s and 1s, which corresponds to the image sequence. This number sequence is first approximated as a trigonometric function curve using a trigonometric function approximation method, thus forming a continuous signal. Then, this continuous signal is filtered to make the trigonometric function curve smoother. Next, the periodic signal obtained after the filtering operation is multiplied by the foot movement speed in the image sequence, and the product is used as the amplitude of the trigonometric function curve. After such a series of processing, the resulting curve is the phase feature of this embodiment.

[0134] Step S12: Input the phase features into the gated network to obtain multiple mixing coefficients.

[0135] In this embodiment, the gating network is used to coordinate multiple expert networks. Phase features serve as the input to the gating network, and the output is a set of mixing coefficients.

[0136] Step S13: Linearly mix multiple expert networks based on multiple mixing coefficients to obtain the decoder.

[0137] Here, each expert network corresponds to a mixing coefficient. When performing linear mixing, the mixing coefficient can be multiplied by the network parameters of the corresponding expert network, and the product can be used as the network parameters in the decoder.

[0138] Step S706: Forward prediction frames and reverse prediction frames with the same timestamp in the forward prediction frame sequence and the reverse prediction frame sequence are identified as a pair of bidirectional prediction frames.

[0139] In some embodiments, the number of forward prediction frames in the forward prediction frame sequence is the same as the number of backward prediction frames in the backward prediction frame sequence; and the time interval between each two adjacent forward prediction frames in the target video in the forward prediction frame sequence is the same as the time interval between each two adjacent backward prediction frames in the target video in the backward prediction frame sequence.

[0140] In some embodiments, in the forward prediction frame sequence, the time interval between the first and last forward prediction frames in the target video is less than the time interval between the initial and final frames in the target video, and greater than half of the time interval between the initial and final frames in the target video; in the backward prediction frame sequence, the time interval between the first and last backward prediction frames in the target video is less than the time interval between the initial and final frames in the target video, and greater than half of the time interval between the initial and final frames in the target video.

[0141] Step S707: Perform prediction frame fusion processing on each bidirectional prediction frame pair corresponding to the forward prediction frame sequence and the reverse prediction frame sequence in sequence to obtain the prediction frame sequence.

[0142] In some embodiments, the prediction frame fusion process can be implemented through the following steps S7071 to S7075 (not shown in the figure):

[0143] Step S7071: Determine the forward remaining frames in the forward prediction frame sequence that have not undergone prediction frame mixing processing, and the reverse remaining frames in the reverse prediction frame sequence that have not undergone prediction frame mixing processing.

[0144] Step S7072: Determine the timestamps corresponding to the forward and reverse remaining frames.

[0145] Step S7073: Perform prediction frame mixing processing on each pair of bidirectional prediction frames corresponding to the forward prediction frame sequence and the reverse prediction frame sequence in sequence to obtain a mixed processing frame.

[0146] In some embodiments, the forward prediction frame includes the first position coordinates of multiple key points of the target object, and the backward prediction frame also includes the second position coordinates of multiple key points of the target object.

[0147] Here, the target object can be a human or an animal, and the key points of the target object can be the joint positions of the human or animal. Each forward prediction frame and backward prediction frame can include the position coordinates of multiple joint positions.

[0148] The prediction frame mixing process is performed sequentially on each pair of bidirectional prediction frames corresponding to the forward prediction frame sequence and the reverse prediction frame sequence. This can be achieved as follows: Based on the positions of the forward prediction frame and the reverse prediction frame in the target video, the corresponding forward prediction frame mixing weight and the corresponding reverse prediction frame mixing weight are determined sequentially. Then, based on the mixing weights of the forward prediction frame and the reverse prediction frame, the first position coordinates and the second position coordinates of the same key point in each pair of bidirectional prediction frames are sequentially subjected to prediction frame mixing to obtain the mixed frame.

[0149] In this embodiment, the forward prediction frame mixing weight can be determined based on the position of the forward prediction frame in the target video. The closer the forward video frame is to the end frame of the target video, the smaller the forward prediction frame mixing weight; conversely, the farther the forward video frame is from the end frame, the larger the forward prediction frame mixing weight. That is, the closer the forward video frame is to the beginning frame of the target video, the larger the forward prediction frame mixing weight; and the farther the forward video frame is from the beginning frame, the smaller the forward prediction frame mixing weight. Similarly, the backward prediction frame mixing weight can be determined based on the position of the backward prediction frame in the target video. The closer the backward prediction frame is to the end frame of the target video, the larger the backward prediction frame mixing weight; and the farther the backward prediction frame is from the end frame, the smaller the backward prediction frame mixing weight. That is, the closer the backward prediction frame is to the beginning frame of the target video, the smaller the backward prediction frame mixing weight; and the farther the backward prediction frame is from the beginning frame, the larger the backward prediction frame mixing weight.

[0150] Here, based on the mixing weights of the forward prediction frame and the reverse prediction frame, the first and second position coordinates of the same key points in each pair of bidirectional prediction frames are sequentially subjected to prediction frame mixing processing. This can be achieved by multiplying the mixing weight of the forward prediction frame with the first position coordinate and multiplying the mixing weight of the reverse prediction frame with the second position coordinate, then summing the two products to obtain the mixing coordinates of each same key point in each pair of bidirectional prediction frames. Finally, the same key points are connected based on the mixing coordinates of all the same key points to obtain the motion image of the target object in the mixed processing frame, which is the mixed processing frame.

[0151] For example, the target video has a total of 11 frames (the initial frame is frame 0, and the ending frame is frame 10). The forward prediction frame sequence generates the 5+2 forward frames, totaling 7 frames, and the backward prediction frame sequence generates the 5+2 backward frames, also totaling 7 frames. This results in a 5-frame overlap: frames 3, 4, 5, 6, and 7. Therefore, the prediction frame mixing process in this embodiment can be performed using linear interpolation, mixing the forward frames 3, 4, 5, 6, and 7 with the backward frames 7, 6, 5, 4, and 3. During linear interpolation, both the forward and backward sides have a coefficient, which is the mixing weight. The mixing weight is largest near the leftmost position (the starting position of the forward direction) and smallest near the rightmost position (the ending position of the forward direction). When the ratio is largest, the mixing weight is 1; when the ratio is smallest, the mixing weight is 0. In other words, from left to right, the positive coefficient gradually decreases and the negative coefficient gradually increases. At the far right, the positive coefficient is 0 and the negative coefficient is 1.

[0152] Step S7074: Determine the timestamps of the forward prediction frame and the reverse prediction frame in the bidirectional prediction frame pair as the timestamps of the corresponding hybrid processing frame.

[0153] Step S7075: According to the timestamp sequence, the forward remaining frames, the mixed processing frames, and the reverse remaining frames are spliced ​​together to obtain the predicted frame sequence.

[0154] In step S708, the server performs video frame stitching processing on the initial frame, the predicted frame sequence, and the end frame to obtain the target video.

[0155] In step S709, the server sends the target video to the terminal.

[0156] In step S710, the terminal displays the target video on the current interface.

[0157] The video processing method provided in this application, when performing prediction frame fusion processing on a forward prediction frame sequence and a backward prediction frame sequence, determines the mixing weights of the forward prediction frames and the backward prediction frames based on the positions of the forward prediction frames and the backward prediction frames in the target video. Then, based on the mixing weights of the forward and backward prediction frames, a weighted sum of the first and second position coordinates of the same key points in the bidirectional prediction frame pair is performed to achieve prediction frame fusion processing. Specifically, the mixing weight of the forward prediction frame closer to the initial frame is larger, and the mixing weight of the backward prediction frame closer to the end frame is also larger. This ensures that the actions of the target object between the fusion processing frames closer to the initial frame and the initial frame are closer, resulting in smoother actions; similarly, the actions of the target object between the fusion processing frames closer to the end frame and the end frame are closer, resulting in smoother actions. This further avoids problems such as slippage and drift of the target object in the generated target video.

[0158] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0159] This application provides a video processing method applicable to keyframe interpolation of variable-length sequences. Given keyframes, the method described in this application can complete transition animations. Practical applications include creating animation assets with only keyframes provided, or secondary completion of partially omitted transition animations during motion capture. For example, the animation can be game animation, and the video processing method described in this application can be applied to the creation of game animations or to complete partially omitted transition animations in game animations.

[0160] In other embodiments, the video processing method can also be applied to updating other animation frames between any two animation frames in a game animation. For example, in a turn-based game application, once the player's operation command is determined, the final result of the current turn is determined, that is, the initial frame and the final end frame are determined. However, for the game animation segment between the initial frame and the end frame, due to the diversity of the game process in multiple player matches, there can be multiple match process animations. Therefore, in this turn-based game scenario, the method of the embodiments of this application can be used to first predict the game animation segment A1 of any player A based on the determined initial frame and end frame. At the same time, the method of the embodiments of this application can be used to update the game animation segment A1 to obtain a new game animation segment A2, and the game animation segment A2 is used as the game animation of player B and displayed on player B's terminal.

[0161] Furthermore, since the result generated by the embodiment of this application can perfectly match the target frame (i.e., the end frame), it is possible to generate multi-segment transition animations without the aid of post-processing. Figure 8 This is a schematic diagram illustrating the effect of the video processing method according to an embodiment of this application, such as... Figure 8 As shown, video frame 801 of the person is a given keyframe (it can be either the initial frame or the final frame). Figure 8 The diagram shows the interpolation results of multiple consecutive keyframes. Frame 801 of the video of the person is a keyframe, and frame 802 of the video of the person is the generated frame after prediction (i.e., the prediction frame). Multiple prediction frames between two adjacent keyframes constitute a prediction frame sequence.

[0162] The solution in this application embodiment can also meet diverse needs, and under the same constraints, differentiated results can be generated each time. Figure 9 This is a schematic diagram showing the generation results of three separate extractions at the same moment under the same conditions, using the method described in this application. Figure 9 As shown, the generation results obtained by generating the same data three times under the same conditions and capturing the same moment are all different. Therefore, the method of this application embodiment can generate differentiated results.

[0163] The video processing method provided in this application is based on a bidirectional generation mechanism. The bidirectional generation mechanism of this application will be described below.

[0164] The bidirectional generation mechanism is designed primarily to address the issue of the generated transition animation (i.e., the predicted frame sequence) deviating from the target frame due to error accumulation. Figure 10 This is a flowchart of the bidirectional generation mechanism provided in the embodiments of this application, such as... Figure 10As shown, motion sequences are generated starting from the initial frame and the end frame respectively, and then the two sequences are stitched together in the middle region to form a complete transition animation. To achieve a bidirectional generation mechanism, this embodiment requires two generators: a forward generator 1001 and a backward generator 1002. The forward generator generates a forward motion sequence (i.e., a forward predicted frame sequence) starting from the initial frame, while the backward generator generates a backward motion sequence (i.e., a backward predicted frame sequence) starting from the end frame. In the figure, L represents the length of the entire animation segment, i.e., the number of animation frames included in the entire animation segment, and K represents several additional animation frames used to mix with the other sequence to obtain a smooth stitching result. Therefore, the sequence length in each direction is L / 2 + K. During stitching, the parts of the two sequences that overlap in time are mixed, and then the mixed result is stitched together with the remaining parts of the two sequences to form a complete transition animation. To improve the naturalness of the generated results, the generated results will be processed by a pair of discriminators: a long discriminator (1003) and a short discriminator (1004).

[0165] Unlike unidirectional generation methods, the bidirectional generation mechanism in this application shifts the post-processing blending operation from the end of the transition animation to the middle. This perfectly solves the problem of the result deviating from the target frame due to blending at the end. In keyframe interpolation tasks, the boundaries should be as consistent as possible with the given initial and ending frames, while the middle region has no original data and should ideally have as much diversity as possible. This characteristic is very compatible with the bidirectional generation mechanism. At the same time, this also places higher demands on the implementation of the bidirectional generation mechanism, requiring a sufficiently large and diverse space in the middle region of the splicing. To cooperate with the bidirectional generation mechanism, this application also designs the following motion generation model, in which the generators in both directions adopt this design.

[0166] The action generation model in this application is based on a Conditional Variational Auto Encoder (CVAE) network. Here is a description of the CVAE network: CVAE is a deep generative network that adds conditional signal input to VAE, thereby enabling the generation of specified data types. Figure 11 This is a schematic diagram of the CVAE network structure, as shown below. Figure 11As shown, the CVAE network contains an encoder (E) and a decoder (G), where the encoder is used to extract the distribution space of the data x, and the decoder, guided by the conditional signal c, can map the latent variable z sampled from the distribution space to the desired output X'.

[0167] To better achieve stitching in the bidirectional generation mechanism, this application embodiment incorporates some special designs; therefore, the motion generation model of this application embodiment is referred to as S-CVAE (Stitching-CVAE). S-CVAE includes an encoder and a decoder. The encoder maps the initial and final frames of the animation data to a latent variable space, where latent variables z can be sampled. Under the adjustment of conditional signals and the control of phase characteristics, the decoder can restore the sampled latent variables z to the character's state in the next frame, thereby achieving motion generation.

[0168] Figure 12 This is a schematic diagram of the encoder structure of the S-CVAE provided in the embodiments of this application, as shown below. Figure 12 As shown, the encoder in this embodiment uses three sub-encoders: the current frame encoder 121 (State Encoder), the target frame encoder 122 (Target Encoder), and the offset encoder 123 (Offset Encoder) to encode the character state in the current frame, the character state in the target frame, and the offset of the character state between the current frame and the target frame, respectively. After concatenation, the data distribution space is extracted again through a Long Short-Term Memory (LSTM) network 124 and a fully connected layer (FC) 125, and finally random sampling processing 126 is performed. To make S-CVAE more suitable for concatenation tasks in bidirectional mechanisms, the following aspects are added to this embodiment:

[0169] (1) Latent interpolation: such as Figure 12 As shown, the character state of both the current frame and the target frame will be extracted, and then linear interpolation will be performed on the two spaces using the following formula (1):

[0170] (1);

[0171] in, This indicates the data distribution of the current frame; This represents the data distribution of the target frame. The coefficients of the target frame. It gradually increases from 0 to 1 as it approaches the target frame. This will cause the generated pose to be closer to the target frame as it gets closer; μ represents the mean; θ represents the standard deviation. This is the linear interpolation result of linear interpolation of the data distribution of the current frame and the target frame.

[0172] (2) Two-way alignment: Figure 13 This is a schematic diagram of bidirectional alignment provided in an embodiment of this application, as shown below. Figure 13 As shown, the two generators operate alternately. First, the result of the reverse generator 131 serves as the conditional signal for the forward generator 132. Then, the latest result generated by the forward generator 132 serves as the control signal for the reverse generator 131, guiding the generation of the reverse action sequence. This alternation ensures that the two generators generate sequences closer to the target frame for easier splicing. Otherwise, if the sequences generated by the two generators differ too much, splicing will fail.

[0173] (3) Stitching Loss Function: The stitching loss is calculated as the L1 distance between the global joint positions of the overlapping parts of the forward and reverse animation sequences. The loss corresponding to this L1 distance can constrain the results of the sequences generated in the two directions to be close to each other in the middle region, so as to facilitate stitching. The definition of the stitching loss function provided in the embodiments of this application is as follows: (2)

[0174] (2);

[0175] Where P represents the global position information of the characters in the generated sequence. The values ​​of the forward-generated sequence, The value of the reverse-generated sequence.

[0176] (4) Phase Generator: The decoder part of S-CVAE introduces phase features as control signals. Phase is the "phase" of the animation, which can indicate which stage of the motion cycle is currently in. Phase features can eliminate ambiguity in long sequence motion, alleviate the sliding steps in the generation of transition animation, and improve the animation quality. In this embodiment, the phase features of both feet are obtained by using the ground contact information of the feet through low-pass filtering, trigonometric function approximation and other operations.

[0177] In this embodiment, the phase features used by the decoder are derived from the phase generator. The calculated phase features serve as training data for the phase generator. The phase generator predicts phase features using the current frame's character state as input. The phase generator is a multilayer perceptron (MLP) network trained using a phase feature reconstruction loss.

[0178] The S-CVAE encoder and decoder employ an asymmetric design. The encoder structure is as follows: Figure 12 As shown, the decoder is designed based on a Mixture of Experts Networks (MoEN). The decoder comprises multiple expert networks and a gating network responsible for coordinating these networks. Compared to a pure MLP, the Mixture of Experts Networks has stronger learning and generation capabilities. Phase features serve as input to the gating network, and the output is a set of mixing coefficients. These coefficients can be used to linearly mix multiple expert networks to obtain the final generator. In this embodiment, the generator for each direction is constructed using the above model, ultimately forming a complete bidirectional generation mechanism.

[0179] The bidirectional keyframe interpolation technique proposed in this application can perfectly solve the target frame deviation problem caused by error accumulation in autoregressive methods. For example... Figures 14A to 14C The schematic diagram of the transition animation generation result shown shows that, in different examples, the tail frame 141 (thin solid line) of the transition animation generated by the method of this application embodiment can perfectly match the target frame 142 (thick solid line).

[0180] Since post-processing is not required to correct target frame deviations, multi-segment keyframe interpolation can be performed without post-processing. For example... Figure 15 The diagram shown is a schematic representation of the interpolation results of multiple consecutive keyframes provided in an embodiment of this application. The person video frame 151 is a given keyframe (which can be either the initial frame or the end frame). The person video frame 152 is the generated frame after prediction (i.e., the predicted frame). Multiple predicted frames between two adjacent keyframes constitute a prediction frame sequence.

[0181] Due to the introduction of S-CVAE, the results generated by the method in this application embodiment are diverse; that is, even under the same constraints, multiple generation processes result in differences between the samples generated each time. For example... Figure 16 The chart shown compares the six predicted values ​​generated under the same conditions with their corresponding actual values.

[0182] In addition to the above advantages, the application of this embodiment will greatly save the manpower consumption of manual keyframe interpolation and increase the speed of animation asset generation.

[0183] It is understood that in the embodiments of this application, if the content involves user information, such as the initial and final frames of the target video, the predicted frame sequence, and the target video itself, and if it involves data related to user information or enterprise information, user permission or consent is required when the embodiments of this application are applied to specific products or technologies, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0184] The following continues to describe the exemplary structure of the video processing apparatus 354 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 4 As shown, the video processing apparatus 354 includes:

[0185] The system comprises: an acquisition module for acquiring the initial frame and the end frame of the target video; a determination module for determining, using the initial frame and the end frame as prediction start frames respectively, an autoregressive prediction method to determine the forward prediction frame sequence corresponding to the initial frame and the backward prediction frame sequence corresponding to the end frame; a prediction frame fusion module for performing prediction frame fusion processing on the forward prediction frame sequence and the backward prediction frame sequence according to the timestamps corresponding to each forward prediction frame in the forward prediction frame sequence and each backward prediction frame sequence in the backward prediction frame sequence, to obtain a prediction frame sequence; and a stitching module for performing video frame stitching processing on the initial frame, the prediction frame sequence, and the end frame to obtain the target video.

[0186] In some embodiments, the determining module is further configured to: for each current frame, obtain the role state of the current frame, the role state of the initial frame, and the role state of the end frame; encode the role state of the current frame, the role state of the initial frame, and the role state of the end frame using an encoder to obtain an encoding vector; and decode the encoding vector using a decoder to obtain the next forward prediction frame and the next reverse prediction frame of the current frame.

[0187] In some embodiments, the determining module is further configured to: when predicting the next forward prediction frame of the current frame, determine the role state offset between the current frame and the end frame, and encode the role state of the current frame, the role state of the end frame, and the role state offset using an encoder to obtain a first encoding vector; when predicting the next reverse prediction frame of the current frame, determine the role state offset between the current frame and the initial frame, and encode the role state of the current frame, the role state of the initial frame, and the role state offset using an encoder to obtain a second encoding vector; extract data distribution space from the first encoding vector and the second encoding vector respectively to obtain a first extraction vector and a second extraction vector; and decode the first extraction vector and the second extraction vector respectively using the decoder to obtain the next forward prediction frame and the next reverse prediction frame of the current frame.

[0188] In some embodiments, the determining module is further configured to: extract the current frame data distribution space and the end frame data distribution space from the first encoding vector respectively, to obtain the current frame data distribution and the end frame data distribution accordingly; perform linear interpolation on the current frame data distribution and the end frame data distribution to obtain the first extracted vector; and perform the current frame data distribution space and the initial frame data distribution space from the second encoding vector respectively, to obtain the current frame data distribution and the initial frame data distribution accordingly; and perform linear interpolation on the current frame data distribution and the initial frame data distribution to obtain the second extracted vector.

[0189] In some embodiments, the apparatus further includes: a coefficient determination module, configured to determine linear interpolation coefficients based on the position of the current frame in the target video; the determination module is further configured to: perform linear interpolation processing on the current frame data distribution and the end frame data distribution based on the linear interpolation coefficients to obtain the first extraction vector; and perform linear interpolation processing on the current frame data distribution and the initial frame data distribution based on the linear interpolation coefficients to obtain the second extraction vector.

[0190] In some embodiments, the determining module is further configured to: determine each forward prediction frame in the forward prediction frame sequence sequentially using the initial frame as the prediction start frame via a forward generator; and determine each reverse prediction frame in the reverse prediction frame sequence sequentially using the end frame as the prediction start frame via a reverse generator; wherein the forward generator and the reverse generator operate alternately, and each time the reverse generator predicts a reverse prediction frame, it inputs the reverse prediction frame as condition information of the forward generator into the forward generator to predict the forward prediction frame; and each time the forward generator predicts a forward prediction frame, it inputs the forward prediction frame as condition information of the reverse generator into the reverse generator to predict the reverse prediction frame.

[0191] In some embodiments, the decoder includes multiple expert networks and a gating network; the apparatus further includes: a phase feature acquisition module for acquiring phase features of the target video; an input module for inputting the phase features into the gating network to obtain multiple mixing coefficients; and a linear mixing module for linearly mixing the multiple expert networks based on the multiple mixing coefficients to obtain the decoder.

[0192] In some embodiments, the prediction frame fusion module is further configured to: determine the forward prediction frames and reverse prediction frames with the same timestamp in the forward prediction frame sequence and the reverse prediction frame sequence as a pair of bidirectional prediction frames; and sequentially perform prediction frame fusion processing on each pair of bidirectional prediction frames corresponding to the forward prediction frame sequence and the reverse prediction frame sequence to obtain the prediction frame sequence.

[0193] In some embodiments, the prediction frame fusion module is further configured to: determine the forward remaining frames in the forward prediction frame sequence that have not undergone the prediction frame mixing process, and the reverse remaining frames in the reverse prediction frame sequence that have not undergone the prediction frame mixing process; determine the timestamps corresponding to the forward remaining frames and the reverse remaining frames respectively; sequentially perform prediction frame mixing process on each bidirectional prediction frame pair corresponding to the forward prediction frame sequence and the reverse prediction frame sequence to obtain a mixed processing frame; determine the timestamps of the forward prediction frames and the reverse prediction frames in the bidirectional prediction frame pair as the timestamps of the corresponding mixed processing frames; and perform splicing processing on the forward remaining frames, the mixed processing frames, and the reverse remaining frames according to the order of the timestamps to obtain the prediction frame sequence.

[0194] In some embodiments, the forward prediction frame includes the first position coordinates of multiple key points of the target object, and the reverse prediction frame includes the second position coordinates of multiple key points of the target object; the prediction frame fusion module is further configured to: determine the corresponding forward prediction frame fusion weight and the corresponding reverse prediction frame fusion weight according to the positions of the forward prediction frame and the reverse prediction frame in the target video; based on the fusion weight of the forward prediction frame and the fusion weight of the reverse prediction frame, perform prediction frame fusion processing on the first position coordinates and the second position coordinates of the same key points in each pair of bidirectional prediction frames in sequence to obtain the fused frame.

[0195] In some embodiments, the number of forward prediction frames in the forward prediction frame sequence is the same as the number of backward prediction frames in the backward prediction frame sequence; and the time interval between each two adjacent forward prediction frames in the target video in the forward prediction frame sequence is the same as the time interval between each two adjacent backward prediction frames in the target video in the backward prediction frame sequence.

[0196] In some embodiments, in the forward prediction frame sequence, the time interval between the first and last forward prediction frames in the target video is less than the time interval between the initial and final frames in the target video, and greater than half of the time interval between the initial and final frames in the target video; in the reverse prediction frame sequence, the time interval between the first and last reverse prediction frames in the target video is less than the time interval between the initial and final frames in the target video, and greater than half of the time interval between the initial and final frames in the target video.

[0197] It should be noted that the description of the apparatus in this application embodiment is similar to the description of the method embodiment described above, and has similar beneficial effects as the method embodiment; therefore, it will not be repeated. For technical details not disclosed in this apparatus embodiment, please refer to the description of the method embodiment of this application for understanding.

[0198] This application provides a computer program product or computer program that includes executable instructions, which are computer instructions; the executable instructions are stored in a computer-readable storage medium. When the processor of a video processing device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, the video processing device performs the method described in this application embodiment.

[0199] This application provides a storage medium storing executable instructions. When these executable instructions are executed by a processor, they cause the processor to perform the method provided in this application, for example... Figure 5 The method shown.

[0200] In some embodiments, the storage medium may be a computer-readable storage medium, such as a ferromagnetic random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic surface memory, optical disc, or a compact disk-read-only memory (CD-ROM); or it may be a device that includes one or any combination of the above-mentioned memories.

[0201] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0202] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file containing other programs or data, for example, in one or more scripts within a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., files storing one or more modules, subroutines, or code sections). As an example, executable instructions may be deployed to execute on a single electronic device, or on multiple electronic devices located in one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0203] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A method of video processing, the method comprising: The method includes: Obtain the initial and final frames of the target video; Using the initial frame and the end frame as prediction start frames, respectively, an autoregressive prediction method is employed to determine a forward prediction frame sequence corresponding to the initial frame and a backward prediction frame sequence corresponding to the end frame. Specifically, a forward generator, using the initial frame as the prediction start frame, sequentially determines each forward prediction frame in the forward prediction frame sequence; a backward generator, using the end frame as the prediction start frame, sequentially determines each backward prediction frame in the backward prediction frame sequence. The forward generator and the backward generator operate alternately. Each time a backward prediction frame is predicted, the backward generator uses it as condition information for the forward generator and inputs it into the forward generator to predict the forward prediction frame. Similarly, each time a forward prediction frame is predicted, the forward generator uses it as condition information for the backward generator and inputs it into the backward generator to predict the backward prediction frame. Based on the timestamps corresponding to each forward prediction frame in the forward prediction frame sequence and each reverse prediction frame in the reverse prediction frame sequence, prediction frame fusion processing is performed on the forward prediction frame sequence and the reverse prediction frame sequence to obtain a prediction frame sequence. Specifically, the forward prediction frame closer to the end frame of the target video has a smaller mixing weight, and the forward prediction frame farther from the end frame of the target video has a larger mixing weight; similarly, the reverse prediction frame closer to the end frame of the target video has a larger mixing weight, and the reverse prediction frame farther from the end frame of the target video has a smaller mixing weight. The initial frame, the predicted frame sequence, and the ending frame are spliced ​​together to obtain the target video.

2. The method of claim 1, wherein, The step of determining the forward prediction frame sequence corresponding to the initial frame and the backward prediction frame sequence corresponding to the end frame using an autoregressive prediction method, with the initial frame and the end frame as the prediction start frames respectively, includes: For each current frame, obtain the character state of the current frame, the character state of the initial frame, and the character state of the end frame; The character state of the current frame, the character state of the initial frame, and the character state of the end frame are encoded by an encoder to obtain an encoding vector. The decoder decodes the encoded vector to obtain the next forward prediction frame and the next backward prediction frame of the current frame.

3. The method of claim 2, wherein, The process of encoding the character state of the current frame, the character state of the initial frame, and the character state of the end frame using an encoder to obtain an encoding vector includes: When predicting the next positive prediction frame of the current frame, the role state offset between the current frame and the end frame is determined, and the role state of the current frame, the role state of the end frame and the role state offset are encoded by an encoder to obtain a first encoding vector. When predicting the next reverse prediction frame of the current frame, the role state offset between the current frame and the initial frame is determined, and the role state of the current frame, the role state of the initial frame, and the role state offset are encoded by an encoder to obtain a second encoding vector. Correspondingly, the step of decoding the encoded vector using a decoder to obtain the next forward prediction frame and the next backward prediction frame of the current frame includes: Data distribution space extraction is performed on the first encoding vector and the second encoding vector respectively to obtain the first extraction vector and the second extraction vector; The decoder decodes the first extraction vector and the second extraction vector respectively to obtain the next forward prediction frame and the next reverse prediction frame of the current frame.

4. The method of claim 3, wherein, The step of extracting the data distribution space from the first encoding vector and the second encoding vector respectively to obtain the first extraction vector and the second extraction vector includes: The data distribution space of the current frame and the data distribution space of the end frame are extracted from the first encoding vector respectively to obtain the data distribution of the current frame and the data distribution of the end frame. The first extraction vector is obtained by performing linear interpolation on the current frame data distribution and the end frame data distribution. as well as, The data distribution space of the current frame and the data distribution space of the initial frame are extracted from the second encoding vector respectively to obtain the data distribution of the current frame and the data distribution of the initial frame. The current frame data distribution and the initial frame data distribution are subjected to linear interpolation to obtain the second extraction vector.

5. The method of claim 4, wherein, The method further includes: The linear interpolation coefficients are determined based on the position of the current frame in the target video; The step of performing linear interpolation on the current frame data distribution and the end frame data distribution to obtain the first extracted vector includes: Based on the linear interpolation coefficients, linear interpolation is performed on the current frame data distribution and the end frame data distribution to obtain the first extraction vector; The step of performing linear interpolation on the current frame data distribution and the initial frame data distribution to obtain the second extracted vector includes: Based on the linear interpolation coefficients, linear interpolation is performed on the current frame data distribution and the initial frame data distribution to obtain the second extraction vector.

6. The method of claim 2, wherein, The decoder includes multiple expert networks and a gating network; the method further includes: Obtain the phase features of the target video; The phase features are input into the gated network to obtain multiple mixing coefficients; The decoder is obtained by linearly mixing the multiple expert networks based on the multiple mixing coefficients.

7. The method of claim 1, wherein, The step of performing prediction frame fusion processing on the forward prediction frame sequence and the reverse prediction frame sequence according to the timestamp corresponding to each forward prediction frame in the forward prediction frame sequence and the timestamp corresponding to each reverse prediction frame in the reverse prediction frame sequence to obtain a prediction frame sequence includes: The forward prediction frames and the reverse prediction frames with the same timestamp in the forward prediction frame sequence and the reverse prediction frame sequence are identified as a pair of bidirectional prediction frames. The prediction frame sequence is obtained by sequentially performing prediction frame fusion processing on each bidirectional prediction frame pair corresponding to the forward prediction frame sequence and the reverse prediction frame sequence.

8. The method of claim 7, wherein, The step of sequentially performing prediction frame fusion processing on each bidirectional prediction frame pair corresponding to the forward prediction frame sequence and the reverse prediction frame sequence to obtain the prediction frame sequence includes: Determine the forward remaining frames in the forward prediction frame sequence that have not undergone the prediction frame mixing process, and the reverse remaining frames in the reverse prediction frame sequence that have not undergone the prediction frame mixing process; Determine the timestamps corresponding to the forward remaining frame and the reverse remaining frame; Each pair of bidirectional prediction frames corresponding to the forward prediction frame sequence and the reverse prediction frame sequence is sequentially subjected to prediction frame mixing processing to obtain a mixed processing frame. The timestamps of the forward prediction frame and the reverse prediction frame in the bidirectional prediction frame pair are determined as the timestamps of the corresponding hybrid processing frame. The forward remaining frames, the mixed processing frames, and the reverse remaining frames are concatenated according to the timestamp sequence to obtain the predicted frame sequence.

9. The method of claim 8, wherein, The forward prediction frame includes the first position coordinates of multiple key points of the target object, and the reverse prediction frame includes the second position coordinates of multiple key points of the target object; The step of sequentially performing prediction frame mixing processing on each pair of bidirectional prediction frames corresponding to the forward prediction frame sequence and the reverse prediction frame sequence to obtain a mixed processing frame includes: Based on the positions of the forward prediction frame and the backward prediction frame in the target video, the corresponding forward prediction frame mixing weight and the corresponding backward prediction frame mixing weight are determined sequentially. Based on the forward prediction frame mixing weight and the reverse prediction frame mixing weight, the first and second position coordinates of the same key points in each bidirectional prediction frame pair are sequentially subjected to prediction frame mixing processing to obtain the mixed processing frame.

10. The method of claim 7, wherein, The number of forward prediction frames in the forward prediction frame sequence is the same as the number of reverse prediction frames in the reverse prediction frame sequence; In the forward prediction frame sequence, the time interval between each two adjacent forward prediction frames in the target video is the same as the time interval between each two adjacent reverse prediction frames in the target video in the reverse prediction frame sequence.

11. The method according to claim 7, characterized in that, In the forward prediction frame sequence, the time interval between the first and last forward prediction frames in the target video is less than the time interval between the initial and final frames in the target video, and greater than half of the time interval between the initial and final frames in the target video. In the reverse prediction frame sequence, the time interval between the first and last reverse prediction frames in the target video is less than the time interval between the initial and final frames in the target video, but greater than half of the time interval between the initial and final frames in the target video.

12. A video processing apparatus, comprising: The device includes: The acquisition module is used to acquire the initial and final frames of the target video. A determination module is used to determine a forward prediction frame sequence corresponding to the initial frame and a backward prediction frame sequence corresponding to the end frame, respectively, using an autoregressive prediction method, with the initial frame and the end frame as prediction start frames. Specifically, a forward generator determines each forward prediction frame in the forward prediction frame sequence sequentially with the initial frame as the prediction start frame; a backward generator determines each backward prediction frame in the backward prediction frame sequence sequentially with the end frame as the prediction start frame. The forward generator and the backward generator operate alternately, and each time the backward generator predicts a backward prediction frame, it inputs the backward prediction frame as condition information into the forward generator to predict the forward prediction frame; similarly, each time the forward generator predicts a forward prediction frame, it inputs the forward prediction frame as condition information into the backward generator to predict the backward prediction frame. The prediction frame fusion module is used to perform prediction frame fusion processing on the forward prediction frame sequence and the reverse prediction frame sequence according to the timestamp corresponding to each forward prediction frame in the forward prediction frame sequence and the timestamp corresponding to each reverse prediction frame in the reverse prediction frame sequence to obtain a prediction frame sequence. Specifically, the forward prediction frame closer to the end frame of the target video has a smaller mixing weight, and the forward prediction frame farther from the end frame of the target video has a larger mixing weight; similarly, the reverse prediction frame closer to the end frame of the target video has a larger mixing weight, and the reverse prediction frame farther from the end frame of the target video has a smaller mixing weight. The splicing module is used to perform video frame splicing processing on the initial frame, the predicted frame sequence, and the end frame to obtain the target video.

13. The apparatus of claim 12, wherein, The determining module is further configured to: For each current frame, obtain the character state of the current frame, the character state of the initial frame, and the character state of the end frame; The character state of the current frame, the character state of the initial frame, and the character state of the end frame are encoded by an encoder to obtain an encoding vector. The decoder decodes the encoded vector to obtain the next forward prediction frame and the next backward prediction frame of the current frame.

14. The apparatus of claim 13, wherein, The determining module is further configured to: When predicting the next positive prediction frame of the current frame, the role state offset between the current frame and the end frame is determined, and the role state of the current frame, the role state of the end frame and the role state offset are encoded by an encoder to obtain a first encoding vector. When predicting the next reverse prediction frame of the current frame, the role state offset between the current frame and the initial frame is determined, and the role state of the current frame, the role state of the initial frame, and the role state offset are encoded by an encoder to obtain a second encoding vector. Correspondingly, the step of decoding the encoded vector using a decoder to obtain the next forward prediction frame and the next backward prediction frame of the current frame includes: Data distribution space extraction is performed on the first encoding vector and the second encoding vector respectively to obtain the first extraction vector and the second extraction vector; The decoder decodes the first extraction vector and the second extraction vector respectively to obtain the next forward prediction frame and the next reverse prediction frame of the current frame.

15. The apparatus according to claim 14, characterized in that, The determining module is further configured to: The data distribution space of the current frame and the data distribution space of the end frame are extracted from the first encoding vector respectively to obtain the data distribution of the current frame and the data distribution of the end frame. The first extraction vector is obtained by performing linear interpolation on the current frame data distribution and the end frame data distribution. as well as, The data distribution space of the current frame and the data distribution space of the initial frame are extracted from the second encoding vector respectively to obtain the data distribution of the current frame and the data distribution of the initial frame. The current frame data distribution and the initial frame data distribution are subjected to linear interpolation to obtain the second extraction vector.

16. The apparatus of claim 15, wherein, The device further includes: The coefficient determination module is used to determine the linear interpolation coefficients based on the position of the current frame in the target video; The step of performing linear interpolation on the current frame data distribution and the end frame data distribution to obtain the first extracted vector includes: Based on the linear interpolation coefficients, linear interpolation is performed on the current frame data distribution and the end frame data distribution to obtain the first extraction vector; The step of performing linear interpolation on the current frame data distribution and the initial frame data distribution to obtain the second extracted vector includes: Based on the linear interpolation coefficients, linear interpolation is performed on the current frame data distribution and the initial frame data distribution to obtain the second extraction vector.

17. The apparatus of claim 13, wherein, The decoder includes multiple expert networks and a gating network; the device also includes: A phase feature acquisition module is used to acquire the phase features of the target video; The input module is used to input the phase features into the gated network to obtain multiple mixing coefficients; A linear mixing module is used to linearly mix the multiple expert networks based on the multiple mixing coefficients to obtain the decoder.

18. A video processing device, comprising: include: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the video processing method according to any one of claims 1 to 11.

19. A computer-readable storage medium, characterized in that, The device stores executable instructions for causing a processor to execute the executable instructions to implement the video processing method according to any one of claims 1 to 11.

20. A computer program product, comprising computer-executable instructions, characterized in that, When the computer-executable instructions are executed by the processor, they implement the video processing method according to any one of claims 1 to 11.

Images